How to Audit Software Vendors: Part 3 - Validation Center

Today is Part 3, the final post of our series – How to Audit Software Vendors. This post will review three examples of how you can leverage the audit results in a risk-based validation approach. If you haven’t already, be sure to read Part 1 on Audit Preparation and Part 2 on Audit Execution.

FDA Guidance on computer system validation promotes the use of vendor documentation in your validation.

“If the vendor can provide information about their system requirements, software requirements, validation process, and the results of their validation, the … manufacturer can use that information as a beginning point for their required validation documentation. The vendor’s life cycle documentation, such as testing protocols and results, source code, design specification, and requirements specification, can be useful in establishing that the software has been validated.” — FDA Guidance: General Principles of Software Validation

But it doesn’t end with taking the vendor’s documentation at face value – they may or may not be compliant with regulations. There are several examples of organizations who received Warning Letters for poor use of vendor documentation. Therefore, we recommend a risk based approach to using vendor documentation for your validation projects.

For more on risk based validation approaches, attend our free 90-minute webinar.

Let’s look at some scenarios.

With a risk based approach, the ability to use vendor documentation depends on the rating you gave the vendor during the audit. For the examples below, we used the following rating scale:

Depending on the scope of the audit, the vendor may have a rating for different areas.

You can see in the examples below how the ratings affect your use of vendor documentation or services.

Scenario 1 – Requirements and Test Protocols

Your policy requires Requirements and Test Protocols for validated systems. Your software vendor provides you with their requirements and protocols. During the audit, you evaluate the vendor’s quality and procedures for requirements documentation and protocol development.

Based on the vendor’s rating, usage of their documentation would be:

• Full Approval – Use vendor’s requirements documentation and protocols. Review and approve in-house
• Conditional Approval – Depends upon the condition of approval
• Not Approved – Do not use vendor’s documentation. Write in-house, as normal.

Because you audited the vendor’s requirements documentation and protocol development practices and quality, you have determined the likelihood of error with the documentation.

Scenario 2 – Development Testing Practices and Quality

Your policy requires comprehensive testing with 100% path coverage using multiple data scenarios for validated systems. Your vendor does not provide protocols or results. During the audit you evaluated their software quality and SDLC procedures.

Based on the vendor’s rating, usage of vendor documentation would be:

• Full Approval – Reduced path coverage and data scenarios (especially for less critical features)
• Conditional Approval – Depends upon the condition of approval
• Not Approved – Thorough testing with 100% path coverage using multiple data scenarios

As a result of auditing your vendor’s development testing practices and quality, you have determined the likelihood of a software error getting through testing.

Scenario 3 – Infrastructure Qualification Practices and Support

Your policy states that validated systems must be hosted on qualified infrastructure. Your vendor offers to host the system on their servers. During the audit, you evaluated their infrastructure qualification practices and support.

Based on their rating, usage of vendor documentation would be:

• Full Approval – Allow vendor to host the system. Periodically audit their hosting service
• Conditional Approval – Depends upon the condition of approval
• Not Approved – Do not allow vendor to host the validated system. Host in-house or on other qualified infrastructure.

Because you audited the vendor’s infrastructure qualification practices and support procedures, you have determined the likelihood of errors with hosting the system on the vendor’s servers.

Audit results are going to affect how you make decisions for your entire project.

By performing an audit earlier in your system’s life cycle, you get a clearer understanding of your future efforts over the lifetime of the system. Ideally, it’s best to audit vendors prior to purchasing their product, giving you the most leverage in negotiations as the vendor is more willing to make changes to win your business.

During the audit, use a risk-based approach to categorize the findings based on the impact to the product (software) and the impact to regulatory compliance (documentation). Remember that auditing your software vendor is supposed to be a win-win situation for everyone. You are there because you and the vendor would like to do business together.

And finally, based on the vendor’s rating, take a risk-based approach to using vendor documentation for your validation project. In favorable cases, vendor documentation can reduce your need for rework and additional validation efforts.