The Key To Successful Software Vendor Assessments (Hint: It's A Checklist!)

How do you choose a software vendor? And once you do, will they be able to keep you compliant? Which vendor will provide a high quality software product and protect your data? Are they going to be there when you need support? These are just a few of the questions you’re looking to answer when selecting a software vendor.

But how do you make the best recommendation for your company? There is one item that we use for every supplier audit – a software vendor assessment checklist. It is the key to successful vendor evaluations. Not only does it keep each audit on track with a purpose and scope, but it makes giving a yes or no recommendation so much easier.

What should be included in your checklist? Let’s dive in.

Getting Started

When To Perform Software Vendor Assessments

Ideally, you should be assessing your vendors during the vendor selection process – before you buy their product. That way, you can make an informed decision about which vendor can keep you compliant and will be able to support you throughout the lifetime of your relationship.

If you’ve already purchased the product, using a checklist is still valuable. You will identify areas where your vendor is doing well and where there are compliance gaps.

And when it’s time to re-evaluate your supplier, pull out your previous assessment and focus on the areas that needed improvement.

If you want a deeper dive into the vendor assessment process, attend our next free webinar.

Inspection Categories

To make your software vendor assessment checklist, first you need to identify the various inspection categories needed to evaluate your suppliers. We have audited numerous software vendors over the past few decades, so our checklist contains over 250 questions from 83 inspection categories. I’ll walk you through some of them today, but your checklist should contain categories that make sense to your organization.

Also, don’t worry about getting all of the categories right the first time you create your checklist. As you get a few vendor audits under your belt you will be able to improve your categories and questions.

Once you have your categories in order, start compiling questions for each.

Rating Scale

Before we get into the questions, we need to talk about the type of results you’re looking to get out of a vendor assessment. A rating scale will guide you in the evaluation of your software supplier and summarize the assessment recommendation. Essentially, it is an easy way to get a go/no-go decision.

When you’re evaluating a vendor on hundreds of questions, it’s easier to keep things simple with a Pass/Fail rating. But often times there are some grey areas, so our certified auditors use a three-point scale below.

1. Acceptable
2. Partially Acceptable
3. Not Acceptable

A more complex scale may make sense for your organization, but don’t go overboard.

Tip: Do not use more than a 5-point scale or it will be hard to synthesize the results for your audit report.

Get Our 250-Question Vendor Assessment Template Now

Questions To Ask During Software Vendor Assessments

Finally – the questions! I’m going to walk you through several inspection categories and the types of questions to ask for each. This is not a complete list for each category, but it should help you get some momentum going as you develop your checklist.

About The Vendor

To start, you want to know about the vendor’s commitment to your industry and the likelihood that they will be around when you need support. Questions can include:

1. How long has the vendor been in business?
2. In what industry(ies) does the vendor focus its resources?
3. How long has the product of interest been on the market?
4. What companies are using the product of interest?
5. Does the vendor have any certifications (such as ISO, Malcom Baldridge)?
6. Does the vendor’s strategy, business outlook and organizational structure suggest a long-term commitment to the product and industry?

If your company’s Purchasing or Legal department has already screened the vendor, you don’t need to cover these items again during your inspection.

General Quality Topics

These general questions will help you determine whether or not the vendor has the framework in place to support product quality.

1. Is there an overall program to ensure software quality? Does the program include a software life cycle, testing practices, documentation requirements and responsibilities?
2. Is the quality organization independent from the IT organization?
3. Does Quality Assurance have a role in the system life cycle – especially in testing and system release?
4. Does the Quality group perform periodic audits of system quality, documentation and procedural compliance?

When the vendor answers “Yes” to a question, ask for some form of evidence to review. For example, evidence for these questions could be in the form of a Quality Plan, organization charts, job descriptions and responsibilities, etc.

System Life Cycle and Requirements

Here you want to get a more detailed look at the vendor’s quality and compliance framework for software development.

1. Is there a formal system/software life cycle?
2. Are detailed functional requirements available to clearly define what the system needs to do? Are requirements approved? Are requirements changes approved?
3. Are there adequate, independent quality assurance assessments throughout the software development process?
4. Do requirements include regulatory requirements (i.e, 21 CFR Part 11)?

The vendor might include evidence in the form of requirements documents, SOPs and deliverables throughout the SDLC.

Tip: If you plan to use the vendor’s requirements documentation, check that they meet your internal standards.

Design and Coding

Compare the vendor’s practices with best practices for coding and design.

1. Do formal coding standards and/or guidelines exist? If yes, do they contain requirements for naming conventions, revision level notations, program descriptions, data flow diagrams, structured coding?
2. Do formal user interface standards and/or guidelines exist?
3. Do formal design documentation standards and/or guidelines exist? If yes, review examples.
4. Are all standards under change control?

Typically you will be reviewing your vendor’s SOPs and Design Documents. Solid practices here increase the chance that the vendor will deliver a high-quality software product.

Testing

Address the vendor’s testing practices. Good vendor testing practices should give you confidence to reduce the redundancy of your internal testing.

1. Is there a testing procedure? Does the testing process include requirements for testing types (e.g. unit, system, integration), documenting test results, comparison of results to acceptance criteria, formal approvals, and ongoing evaluation?
2. Are Test Plans used? Do Test Plans defines the approaches, data, scenarios, conditions, responsibilities, and documentation needed to establish the adequate performance of the system?
3. Are Test Summaries available? Do Test Summaries summarize all the system’s test deliverables and activities, and provide evidence that the system is fully tested?
4. Are there test protocols and results? Were the tests protocols approved prior to execution? Do tests include sufficient details, such as data checks, calculations, security? Were the test results reviewed and approved?
5. Are there traceability matrices between specifications and programs? Between specifications and tests?
6. Are system changes tested? Does testing include regression testing of unmodified functionality?

Your software vendor will likely provide you with test plans, test summary, approved test protocols with documented results, etc. If you plan to purchase or use testing documentation from the vendor in your validation, be sure to confirm that their testing documentation meets your company’s standards.

System Documentation

Good documentation practices by the vendor will increase the likelihood that the vendor will be able to support the version of their software that you implement – even after the staff that created the version has moved on to other projects or other companies.

1. Does the system have up to date documentation, including data structures and flows, interactions with other systems, design & architecture?
2. Is the documentation updated each time a change is made?
3. Is documentation managed using change control?
4. What is the retention time period for all system related documentation?
5. How are paper records protected from loss? How are electronic records protected from loss?
6. How are source code, programs, and configuration settings managed and protected? Who has access? Describe how the source code for a given release is controlled.
7. Are back-ups retained in a separated, secure location? Are back-ups retained for the duration required?
8. Is there a Disaster Recovery Plan (DRP)? Has the plan been executed (tested)?

You want to ensure your vendor has thorough documentation on how the system works. Documentation should be updated as changes are made to the system. Additionally, review that your vendor is protecting their documentation from being lost or destroyed. They shouldn’t leave documentation in unlocked cabinets, on personal computers or disorganized network drives.

Security

Find out what the vendor’s practices are for securing physical and electronic assets. Are they protecting their system? Are they protecting their hardware?

1. Is there an adequate security system to prevent unauthorized modification of source code, builds, and distribution copies of software? What are the security measures?
2. Are the development and manufacturing facilities adequately secured against unauthorized entry? Is there a documented authorization list? How is access authorized?

Review your vendor’s SOPs, source code check-in/out records as well as observe their security practices.

Procedures

Check to see if the vendor has adequate procedures in place. Focus on the procedures that are most important to your company.

1. Is the system supported by approved procedures? Do procedures include disaster recovery, back up, maintenance (if hosted), information security, incident management, system change control, and configuration management?
2. Are procedures under change control?
3. Are the procedures periodically reviewed?

You want to see that there are procedures to support the system, especially if it is hosted, as well as procedures for source code, configuration, etc. By having documented change control, your vendor has changes that go through an approval process. Prior versions of the software or system would be available. Also, the typical periodic review period is somewhere between 18 months and 3 years.

Change Management

If you plant to implement future versions or patches to the vendor’s software, explore how the vendor manages changes to the system.

1. Are system changes documented? Are system changes approved?
2. Are changes evaluated for the degree of testing needed?
3. Are configuration changes documented?
4. Is system documentation updated when changes are made?
5. Are users and support personnel retrained when changes are made?

You want to ensure that your vendor is documenting changes to their system and that changes are authorized, documented and tested. Additionally, are the vendor’s support personnel retrained after changes to the system are made? What about retraining the users?

Support

If you plan to rely on the vendor’s support of the purchased software, spend some time asking questions here. If not, you can cover this section lightly.

1. Does the vendor utilize a maintenance/support agreement? If yes, what software product elements are maintained and supported?
2. Are prior releases adequately supported? How long are prior releases supported?
3. Is the vendor’s help desk support function for software and hardware adequate? What is the availability of the help desk?
4. Does the documentation for new releases of the product provide enough information to allow the customer to determine the impact of every change in the release?

It may be important to know how long the vendor will support the release you’re implementing. What are their support hours? Will they give you thorough documentation for new releases? It’s good to see evidence that they will be there when you need support.

Incident Management

Incidents will occur. They can include software defects, data integrity issues, user errors and more. Determine whether or not the vendor is going to be there for you when you have a problem.

1. Are system incidents documented? Are errors, bugs, and defects categorized by severity, urgency, and priority?
2. Are records maintained of all known problems for each revision?
3. Are system incidents evaluated to determine correction and prevention activities?
4. Are system users made aware of critical system defects? How quickly?
5. Does the vendor have an effective program for resolving documented defects?

The vendor may provide you with incident records, CAPA system and notifications for these types of questions. Verify that issues are documented and prioritized. Does the vendor strive to proactively prevent future issues? As the user of the software, will you be informed if another customer detects a critical defect? Are issues resolved? Also check on whether fixes are available as patches on older releases or only included in the next release.

Data Integrity & Protection

Gauge the software’s ability to comply with specific GxP record requirements. Ask questions that apply to your industry.

1. Is the system secured by unique user-ids and passwords?
2. Are there controls to ensure that data can only be entered and changed by authorized personnel?
3. Is access to high levels of access (e.g., Super User) restricted?
4. Is critical data verified by a second person, or by a validated electronic method?

Here you are looking for the vendor to provide requirements, test protocols and results.

Audit Trails

Review the vendor’s audit trail. The first two questions are most relevant for hosted applications. And the last question is for any application, where needed by regulation.

1. Is there an audit trail for critical data and activities?
2. Can the audit trail be reviewed for irregularities?
3. Does the audit trail include user, date, time?

Again, you are looking for the vendor to provide requirements, test protocols and results. Irregularities in audit trails are periodically cited in FDA Warning Letters. Typically, this occurs when management does not review the audit trail and therefore fails to notice that lab tests are being rerun to obtain new (aka better) results.

Training & Personnel

You want to know if the vendor will be able to support the system a few years from now.

1. Does the vendor have its own development staff? If so, how many? What is the percentage of contract developers vs. Vendor employed developers? What is the turnover rate?
2. Is there documentation on the qualifications and training background of personnel engaged in design, coding, testing, validation, installation, and operation of the systems? Does this include consultants and sub-contractors?

21 CFR Part 11 requires that personnel who work on the electronic records/signature system have the training and experience to do the job. So take a look at your vendor’s training requirements, records and personnel resumes.

Vendors & Subcontractors

This might come as a shock, but your vendor might be outsourcing all or part of their system.

1. Are there policies and procedures for assessing potential suppliers of hardware and software?
2. Is documentation provided by the vendor reviewed and approved internally?

Also, if your vendor has vendors, check on their vendor management practices. Look at their Vendor Assessment SOP, Vendor Assessment records and approvals.

Wrap Up

As you can see, there are many inspection categories and questions to ask your software vendors. For your checklist, remember to think about your company’s internal policies and data.

If you don’t have time to develop your own checklist, get ours – it has over 250 questions from 83 inspection categories. It’s an editable Word doc so that you can personalize it for your organization.

Get Our 250-question Software Vendor Assessment Template Now

And if you need more documentation, we have a Software Vendor Assessment Package. It includes both the Software Vendor Assessment template and an SOP that outlines the quality assurance approach to audits and assessments of software suppliers.