Writing NPS Protocols and Testing for Medical Device Manufacturers

There’s a lot of preparation involved in validating non-product software, and the actual testing is just one of many activities. Much like other technical testing activities, well-executed testing is completely dependent on well-crafted protocols that are written prior to testing and approved. You can think of them as guides that define quantitative or qualitative acceptance criteria in advance of testing. They:

• Provide complete test instructions for software and specify instructions or guidelines to achieve a result
• Define relationship to any hardware testing needed
• Help determine if rules or procedures are done correctly
• Confirm linkage to requirements and specifications and prevent crucial steps from being overlooked
• Follow standard methodology
• Contain references to SOPs (standard operating procedures)

A protocol will include your data collection methods and tools plus analysis and any deviations made. You’ll want to be sure that your testing activities align with any related process or equipment (IQ, OQ, and PQ) validation. There is no problem using typical equipment and process validation approach for NPS systems, but you need to understand how these are linked. We won’t dive into that in this article, but read this excellent overview article on IQ, OQ and PQ.

You should aim to create detailed enough protocols that non-software people could run automated testing and receive results for acceptance. This testing can be conducted onsite or remotely. In writing your protocol, you will define:

• Test procedure – How to do the test (protocol), who, what, needs, etc.
• Test case – Testing a specific requirement or specification
• Test requirement – Test case + test procedure = how the requirement is met

Because testing requires deep understanding of how the software application is used, you’ll need to work with Subject Matter Experts (SME) to develop test cases and test procedures. The project team must accept the test protocols before starting the formal testing.

Configuration, Functional, and Requirements Testing on NPS Software

With protocols written and approved, it’s time to begin testing. Never conduct the formal testing without doing a dry run first! There are three types of testing.

1 – Configuration Testing (CT)

This type of testing verifies or checks the configuration, coding of the program, and that low-level requirements are met. It is typically performed on high-risk NPS software systems and includes code testing and software unit testing. If the software was developed outside of your organization and is out of your control, you may not be able to perform this type of testing, because suppliers are not going to provide the source code or access to code. Some example issues to address include:

• Which type of development process was followed?
• Which operating system is being used, and which software is required?
• What is the location and path for any source code software?
• Which testing protocols have been generated?
• Which training records are needed for qualification of testers or those conducting training?

2 – Functional Testing (FT)

Functional Testing confirms proper operation of NPS software according to its specifications. Typically includes specific operations, such as connection to equipment, sensors, and monitors. Related to high- and moderate-risk NPS software systems.

• Do key functions of the NPS system operate as specified in the design?
• Can the system operate with software installed, or was it added during the installation procedure (equipment)?
• Are the process functions operating properly without errors or missing functionalities?
• Can the process connect to all its units, be networked, be directly connected, and use the connectivity to the various functions involved?
• Are data transferring accurately and on time to the functions that need data, including databases and reporting functions?

3 – Requirements Testing (RT)

Requirements Testing ensures that the NPS system meets user requirements, needs, and system design. Typically done on low- to moderate-risk software, including most off-the-shelf (OTS) software.

• Is the software system performing per requirements?
• How do we know the software is doing what it is supposed to do?
• Does our black box software produce expected results?
• Where does data get transmitted and stored? What about data integrity?
• What happens when the software does not perform as expected?

It is not uncommon for functional and requirements testing to be done together for typical non-product software, but this is not an ideal practice for high-risk validation projects. Bespoke software will typically require CT, RT, and FT.

In general, verification should be performed by an expert knowledgeable in software coding, while validation should be performed by an individual knowledgeable about how the NPS system operates. Use caution, however. The people who developed the application may introduce “nothing wrong with our code” bias. Ideally, validation should be done by an independent person / entity.

Your testing will generate testing records and a validation summary report, which may include a statement and reference to the protocol, linkages to process and equipment validation (MVP), reference to all raw data, and more.

Creating an NPS Traceability Matrix

The traceability matrix is a core document in NPS validation because it links together product design requirements, design specifications, and testing requirements in one place. It also provides a means of tying together identified software hazards with control implementation and testing. The traceability analysis commonly consists of a matrix with line items for requirements, specifications and tests, and pointers to hazards and related risk controls. It can be used to:

• Map individual requirements to validation testing (requirements testing)
• Map individual specifications to verification testing (functional and configuration testing)
• Create a one-to-many relationship depending on complexity
• Create a tabular structure with detail contained in testing reports

Although it dates to 2005, the FDA “Guidance for the Content of Premarket Submissions for Software Contained in Medical Devices” has a good example of traceability matrix content. Don’t be surprised if an auditor starts with this document to see if NPS software validation was done correctly and the amount of test coverage.

Validation Summary Report (VSR)

There’s light at the end of the tunnel! Your validation summary report summarizes all validation activities, outcomes, documents and any follow-up actions. This is the final document in the software validation process. As best practice you should link NPS software validation to equipment and process validation, either in the project or through references. You should assign identification reports from validation activities as part of the quality system documentation. It’s also important that you make a very clear statement about the results, so there is no ambiguity about the outcome. You want to make sure that someone else reviews the report and approves it.

NPS Software Validation Is Not Once and Done

OTS software is updated. Proprietary code is “tweaked.” Equipment is replaced. New processes are implemented. Device designs change. New security risks pop up. These are all changes that can trigger a reevaluation of NPS software. Your plan should include a process for evaluating the type of change and level of validation effort required. This could be as simple as documenting what has occurred along with a rationale for what you did or did not do. Or it could be as extensive as a full revalidation. It’s really critical that you maintain effective supplier relationships to ensure that you are getting all communications of new versions and changes.

There will also surely come a time when NPS software needs to be retired, and when this occurs you’ll want to consider things such as whether the source code needs to be maintained, whether there is any raw data from the process that needs to be stored, whether migration of data needs to occur, and many other factors. Again, it’s good to maintain a checklist with questions to consider when this occurs.

Take Your Knowledge of NPS Software to the Next Level

You’ve made it this far. Want to expand your skills to ninja level?  Consider our intensive NPS software validation training class, with instructor-led live training that can be taken from the comfort of your own desk. If you have a specific need, our V&V consulting team is also available to assist with any medical device software validation projects.

Get answers right now. Call