This post was recently updated on July 6, 2020

Welcome to Part 2 of our series on how to audit software vendors. We will focus on the main event – doing the inspection! Before you go any further, make sure you have read How to Audit Software Vendors: Part 1 – Audit Preparation. It goes into detail about what you should be doing to get ready for the audit itself.

Audit Execution

Based on the audit method, perform the activities listed below. Notice that the Questionnaire and Basic Assessment methods only include the discovery phase.

Opening Meeting

The Lead Auditor will kick off the audit by facilitating the Opening Meeting (if your audit method requires one). Your vendor’s management, audit team and possible key department heads will be in attendance. This is a chance for everyone to review the audit schedule, logistics and the vendor to give a company or product overview.

Facility Tour

Only onsite audits are going to include a facility tour. Even though this is a software vendor you’re evaluating, you want to ensure that everything is neat and in good repair. You don’t want leaky ceiling tiles falling on documentation or computer equipment.

You will likely see what kind of physical and data security procedures exist. Are there doors that are propped open or is everyone wearing ID badge to get through locked doors? If the vendor is going to be hosting your system or testing with live data, do you see other companies’ real data lying around on reports or screen prints? Are passwords posted at desks? Also, ensure that documentation is secured in locked rooms or cabinets.

Discovery

Finally, we can get to the main event – Discovery! This is where you determine whether or not the vendor’s system and documentation are suitable for your company.

Discovery will typically take place in a separate audit room, away from the production area. In larger audits, there is often a backroom coordinator who’s making arrangements for interviews and document reviews. You should be aware that sometimes documentation is maintained at another site. Be sure to allow a reasonable amount of time for retrieval.

Don’t be surprised if you’re escorted everywhere, even to the cafeteria for lunch. When helping our clients prepare for an audit, you can bet we direct them to escort the auditors everywhere – you shouldn’t be free to roam anywhere you want!

For each topic, auditors will ‘interview and observe’ the auditees like this:

1. Ask whether the vendor has documented “fill-in-the-blank” procedure (e.g., Testing)
2. Find out if the vendor follows its procedure (e.g., Test Plan, Test Results, etc.)
3. Find out if the procedure is working or effective (e.g., defects are caught before product release to customers)

Auditor Approaches, Techniques & Tips

Auditors use several approaches, typically a combination of these when inspecting a vendor.

• Sampling – Review samples of documents
• Trace Forward – Review a process from start to finish
• Trace Backwards – Review a process from finish to start
• Checklists

As the lead auditor, you will spend your time interviewing folks and asking a lot of questions, hence the phrase ‘interview and observe’. Additionally, you will review documentation and sometimes you may need to observe people at work.

Tips:

• Ask open ended questions rather than Yes/No questions
• Paraphrasing can help you confirm that you’re understanding the information correctly
• Be professional – you’re not going to get anywhere being aggressive or trying to play good cop – bad cop

‘What if … ?’ Situations

What if the auditee doesn’t know the answer to the question?

Accept the answer “I’ll get back to you”. Make a note to follow-up later in the audit.

What if the vendor refuses to answer an audit question and states that it’s ‘confidential information’?

Be willing to sign a non-disclosure agreement to see the requested information. If you don’t utilize an NDA, note in the audit report that the material was not reviewed.

What if you find a deficiency?

Alert the vendor’s audit coordinator, explaining the deficiency vs. the audit standard. Note the deficiency in your audit report. If they immediately correct the deficiency, verify the correction and note the correction in the audit report.

It’s important to remember that auditing your software vendor is hopefully a win-win situation for everyone. You are there because your company wants to do business with the vendor and the vendor agreed to the audit because they want your business. Be courteous and friendly, keep information confidential and keep good records. Don’t argue, debate or act judgmental. Don’t waste the vendor’s time with long lunches or phone calls and never accept gifts or bribes.

After the Audit

Categorize the Findings

Use a risk-based approach to categorize the audit findings.

Approach 1: Rate findings based on impact to the product (works well for software)

Approach 2: Rate findings based on impact to regulatory compliance (works well for vendor supplied documentation)

We typically rate findings on a 3-point scale, but you can develop your own scale and definitions. Here is an example of what each rating means:

1. Critical – has compromised the quality of the product or deviates from regulation
2. Major – could potentially compromise the quality of the product or deviates from regulatory guidance or industry standard
3. Minor – compromises non-essential attributes of the product or deviates from auditing company’s view of best practices

Hold a Closing Meeting

For onsite audits, you will lead the closing meeting upon completion of the audit. Your audit team (if you have one), the vendor’s management and audit team will attend. The purpose of this meeting is to ensure that both teams are on the same page, understanding the findings and observations as well as any Corrective Action Requests and timeline for the final audit report.

We like to give everyone a draft of the audit report so they can follow along as we review the findings and observations – going in order from Critical to Major. Review your Corrective Action Requests and include a timeline for completion. Also, give the vendor your timing for the final Audit Report.

Tip: Get the final Audit Report to the vendor before they lose focus – which could be as quick as the next day, but not longer than 2 weeks.

Post-Audit Activities

Give your vendor a rating

First, review your audit findings and determine a rating for your vendor. Depending on the scope of the audit, you could rate the vendor in multiple areas, such as software vs. system documentation vs. testing documentation.

Write the Final Audit Report

When writing the final audit report, start by listing the findings and observations, grouped by criticality. Include your vendor rating and any corrective action requests. Generally, the lead auditor and the organization who authorized the audit give signature approval on the final report.

Do not include subjective information, recommendations or names of employees associated with findings. The Audit Report is to be concise and objective. Also, it is not appropriate to include topics in the final report that were not discussed in the Closing Meeting.

For more details on how to organize an audit report, attend our next Auditing Software Vendors free webinar.

Corrective Action Requests

We issue Correct Action Requests (CARs) when correction is essential to future use of the vendor or vendor documentation. They can be separate from, or appendices to, the audit report. CARs contain the following information:

• Auditor lists the discrepant condition – point out the deficiency, but do not tell the vendor HOW to fix it
• Auditee determines the root cause, action to correct the issue and a scheduled completion date
• Auditee signature/date

Generally, we request vendors return CAR responses within 2 weeks. But keep in mind that completion of action could take longer. Another reason why it’s better to perform the audit prior to purchase:

Unless there’s a contractual requirement in place, the vendor has no obligation to execute CARs.

Follow-up with the Vendor

Deliver the final audit report and CARs (if there are any), giving the vendor time to perform corrective actions. You may choose to modify the vendor rating after corrective actions have been verified as implemented or effective. After that, you can close the audit and retain the records.

What do you do if CAR follow-up shows that the vendor did not resolve the problem?

First, you can always consider alternate vendors. Or, you can put risk mitigation activities in place on your end such as extra testing, supplemental documentation, etc.

What’s next?

The audit is complete. You’ve given your vendor a rating and may or may not move forward with their product or documentation, but your work is not yet finished. By auditing the vendor’s requirements documentation and protocol development practices and quality, you’ve determined the likelihood of error with their product or documentation.

Next, Part 3 of of this software vendor audit series discusses how you can leverage your vendor audit results in a risk-based validation approach.

  Read More Posts About: Compliance Audits