Risk-based approaches to commissioning and qualification (C&Q) is a common standard for site and project plans. What does that mean and why is there so much implementation variability across the industry? Our CQV benchmark initiative is focused on:
- What is the industry’s level of sophistication for quality risk management (QRM) driven risk-based C&Q?
- What is the value of moving to a true QRM risk-based approach?
Benchmarking provides an initial baseline to track industry adoption and focus areas for individual company gap assessment and improvement.
A Brief History
The ISPE GAMP guide introduced the V model in 1994. ISPE Baseline Guide Volume 5, 1st Edition (2001) introduced commissioning (Cx) and use of Impact Assessments as a formal means to identify the impact of a system on product quality. This C&Q process became the industry standard practice but was/is rarely fully implemented by the industry. ASTM E2500 introduced concepts found in ICHQ8, Q9, Q10 into a science and risk-based approach to C&Q. There are high expectations for the upcoming release of the Second Edition of Baseline Guide Volume 5, which will establish QRM and risk-based verification as the industry standard using common terminology and methodology.
Benchmarking
CAI is partnering with owner companies and mining data from over 1,000 projects that span our 23-year history to answer two questions:
1. What is the industry’s adoption of a risk-based approach to C&Q?
2. What is the per-deliverable savings with various C&Q approaches?
Herein describes the effort to date including the framework and initial data set for the two questions posed above.
Benchmark – Level of Sophistication
A comprehensive list of typical project deliverables was reviewed for several owner companies, sites within each company, and various projects at each site in order to answer the question on where we stand as an industry on C&Q adoption of a risk-based approach.
Four levels of adoption were identified:
| Level of Sophistication | Description | Attributes |
| 1 | Pre-Baseline Guide 5 1st Edition | No formal commissioning, IOQ of all attributes without discrimination of criticality to product |
| 2 | V-Model & Partial Leveraging | Formal Commissioning, followed by IOQ with significant, if not complete, repeat of testing that occurred in Cx |
| 3 |
SLIA & CCA = CAs Or SLIA & RA = CAs
Leveraging Utilized |
Formal Commissioning, with streamlined IOQ, only repeating tests deemed critical to product quality (via some form of risk assessment) |
| 4 |
SLIA & RA = CAs |
Integrated Verification with comprehensive QRM integrated with approach |
Table 1
Note: Levels 1-4 represent increasing adoption of QRM and the risk-based approach. There is an evolution of determining critical aspects (CAs) and critical design elements (CDEs) via risk assessment as compared to less effective methods. CA = the controls identified to reduce the risk of process control failure. CDE = Engineering and automation design elements that implement the controls.
Level 3 includes two levels of sophistication:
1. Use of system impact assessments (SLIA), component criticality assessments (CCA), and SME general knowledge to determine critical aspects (CAs) and critical design elements (CDEs).
2. Use of SLIA and formal risk assessment (RA) to assess risk mitigation for impact on product and patient safety. The risk mitigations form the basis for the CAs and CDEs, and therefore, a more science and process-based rationale for CA and CDE determination.
Twenty-three deliverables/tasks were assessed across the levels of adoption described above.
Deliverable/task areas include:
- Requirements
- Specification and design
- Verification
- Acceptance and release
- Post-release
- Change/Issue management
- Quality unit role
Preliminary data revealed a broad spectrum of adoption of various approaches but a continuing overall trend toward efforts to advance to a risk-based approach. For instance, some firms were found not using URSs, risk assessment, or commissioning and using qualification for all activities with full Quality review and approval. Such an approach represents a level 1 adoption. Meaning that such an approach is pre-1987 guidance and is not in line with current regulatory expectations to define intended use or to understand risks to product quality and patient safety. In contrast, several companies are using a fully risk-based approach, foregoing traditional IQ and OQ protocols in favor of Engineering-driven installation and operational verification as well as the use of a Quality approved acceptability report to validate systems for intended use. Such an approach represents a level 4 adoption.
The Goal:
Our goal for level-of-adoption benchmarking is to provide data that allows companies to see tangible opportunities to save time and money if they implement risk-based practices. Owner companies can utilize our benchmarking methodology to assess their own state on the risk-based approach journey. Owners can then act to address gaps by finding the most impactful areas for initial focus.
The Findings:
Our initial benchmarking identified five key areas for initial focus.
- Critical Aspect/Critical Design Element definition
- Efficient User Requirement Specifications
- Effective use of GEPs and design reviews
- Process validation and product and process qualification
- Cleaning validation
Benchmark – Savings by Approach
The previous benchmark reviewed the current approach used for each deliverable. The second benchmark focused on specific types of equipment and the associated hours for generation and execution for Cx, IQ, and OQ based on the employed approach.
Three of the four approaches were assessed:
| Approach Used for C&Q (from Table 1) | Average Gen Hours | Average Exe Hours | ||||
| Cx | IQ | OQ | Cx | IQ | OQ | |
| Level 2* | 44 | 49 | 54 | 65 | 80 | 99 |
| Level 3 | 36 | 23 | 30 | 49 | 23 | 35 |
| Level 4 | 26 | 10 | 13 | 46 | 11 | 14 |
Table 2
Note: The data represents actual results from projects utilizing SLIA, CCA, and leveraging. Most projects repeat testing in IOQ based on legacy practices and a perception that all URS items designated as “quality” require a clean run test within IOQ, therefore mitigating the ability to truly leverage, driving up IOQ hours. A small subset of clients have successfully limited IOQ to a summary report or gap assessment of commissioning with minimal repeat testing through leveraging. Before BG5 (Level 1), there was no formal commissioning adopted by the industry. As such, systems would be “qualified” but not truly ready for operations without additional startup, debug, and engineering runs required for a fully functioning process due to no or lacking URS, no design review, and no or limited understanding of critical aspects.
The initial data is encouraging and shows that risk-based approaches provide significant savings over traditional approaches.
We note that commissioning execution is relatively uniform as commissioning is a fundamental GEP required activity for all systems. A future reduction in commissioning execution duration is expected as Quality’s role evolves from quality control of commissioning deliverables to quality assurance oversight of a robust Eng/Cx quality system. There is considerable variability in the level of Quality oversight of commissioning; however, the current state is that Quality, more often than not, pre- and post-reviews commissioning that is to be leveraged. A significant reduction is seen in execution even when a rerun of IOQ is performed for CAs/CDEs. Our direct project experience confirms significant savings in execution effort and schedule when risk assessments are used to define critical aspects and IOQ is limited to that which is critical.
The Impact
Implementation of a risk-based approach is not a silver bullet. Product and process knowledge is required as are robust good engineering practices (GEPs). Early activities (URS development, risk assessment, design review, etc.) are required and not shown in the data above. The schedule and personnel savings are clear. Ask yourself: what is the value in delivering a major project two to four months early? A focus on product and process risks to product quality yields long-term operational and lifecycle efficiencies, providing benefits beyond the initial capital expenditure. Too often capital projects focus on initial delivery and the long-term operational benefits are not considered. Product and process knowledge significantly impacts the operation and maintenance phases allowing for optimization of spare parts, maintenance, SOPs, training, calibration periodicity, calibration level, CAPA, change control, etc.
Future state
Additional data will be collected and trended over time. Interested in participating? Firms who participate are provided access to the full data set. CAI will work with your team to baseline your company, site, or project and determine improvement areas for focus. These early wins can help your team rapidly advance and optimize with improved project outcomes as a result. CAI can assist with site VMP and GEP planning and generation.
Contact Nathan Temple, Global Director of CQV, to participate or with questions by emailing nathan.temple@cagents.com
Links:
FDA 21 CFR 211.42: https://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?fr=211.42
FDA Pharmaceutical cGMPs for the 21st Century: https://www.fda.gov/media/77391/download
ICH Q9: http://www.ich.org/products/guidelines/quality/quality-single/article/quality-risk-management.html
FDA Guidance for Industry – Process Validation (2011): https://www.fda.gov/media/71021/download
Eudralex Volume 4: https://ec.europa.eu/health/documents/eudralex/vol-4_en
EU Annex 15: https://ec.europa.eu/health/sites/health/files/files/eudralex/vol-4/2015-10_annex15.pdf
ASTM E2500: https://www.astm.org/Standards/E2500.htm
ISPE Baseline Guide Volume 5: https://ispe.org/publications/guidance-documents/baseline-guide-vol-5-commissioning-qualification-2nd-edition