// February 22, 2023

Innovative Matrix Framework Assesses Risk for Substandard Antibiotics

Person pouring medicine onto a spoon

Substandard medicines are a huge public health threat. Proactively evaluating drug products using a relative risk assessment is important to reduce the prevalence of substandard antibiotics. A proof-of-concept relative risk assessment of quality issues among different antibiotics was developed using risk ranking and filtering principles combined with criteria from the World Health Organization (WHO).

USP Quality Institute fellow Carly Ching recently published a study, Relative Risk Assessment for Substandard Antibiotics Along the Manufacturing and Supply Chain: A Proof‑of‑Concept Study, which appeared in the journal Therapeutic Innovation & Regulatory Science. The study extracted a standard set of empirical data about different antibiotic products and generated a relative score for each parameter and an objective score for each antibiotic within that range. This comprehensive analysis helped to determine the relative risk of an antibiotic to be substandard. This evaluation provided a landscape of factors to estimate the risks and enable the prioritization of surveillance and mitigation efforts for the product and bacterial resistant organisms.

I recently spoke with Carly, who provided insights on her research.

USP: What are risk assessments?

A: Risk assessments or risk-based approaches are proactive processes to identify adverse events before they happen. One important area to better understand risks and root causes for medicine quality issues is along manufacturing and supply chain routes.

USP: Why are risk assessments for quality of medicines important?

A: Ensuring high-quality medicines is essential for desired health outcomes, however in low- and middle-income countries, about 1 in 10 medical product samples fail to meet quality standards. Demonstrated and validated risk assessments are currently limited, and it is unclear exactly where medicine quality issues arise, as it could happen during manufacturing, along the distribution chain or post-purchase with the consumer. Risk assessment of products for quality issues arising along the manufacturing and supply chain can thus inform surveillance and management of interventions designed to reduce the burden of substandard antibiotics, which are authorized products that fail to meet quality limits or specifications. Our assessment of risk for substandard antibiotics along the supply chain is rooted in a common risk management tool called risk ranking and filtering. This method compares and ranks risks involving evaluation of quantitative and qualitative factors. Notably, USP has also been developing resources for risk-based post-marketing surveillance of medicines.

USP: What were the objectives of the study?

A: The goal of this study was to investigate whether a comparative risk assessment framework, which adapts World Health Organization (WHO) criteria for estimating risks for quality issues posed by individual medicines, is applicable and can identify antibiotics with a higher relative risk of substandard prevalence. The main objectives were to (1) identify data on factors affecting antibiotic product quality across the manufacturing and supply chain, (2) develop and apply a risk evaluation framework to a subset of antibiotics and perform relative risk scoring to estimate whether certain antibiotics have a higher risk of quality issues (i.e., greater risk of not meeting quality limits), and (3) summarize context-specific factors impacting medicine quality.

USP: Why did you choose to look at substandard antibiotics?

A: We decided to focus on substandard antibiotics because of the risk of bacterial antibiotic resistance development due to the use of substandard antibiotics. Understanding the relevant risk to antibiotic quality may inform predictions for antibiotic resistance development. Our work focusing on substandard antibiotics allowed us to compare specific attributes of one medicine class, a strategy we believe can be used in the future for other medicine classes.

USP: What approach did you use? How was it novel?

A: As a proof-of-concept for a relative risk assessment for quality issues on the product level for antibiotics, we combined risk ranking and filtering principles and the WHO criteria for estimating risks for quality issues posed by individual medicines. The risks were divided into “affecting probability of incidence of quality issue” or “increasing severity of exposure and harm to health due to the antibiotic being substandard,” This same criterion is currently being used within a recently developed USP risk-based post-marketing surveillance tool to assign a medicine risk score based on available pertaining data and expertise team decisions. Our assessment is novel as it used a standard and comparable set of empirical data which served as a proxy for each risk criterion. For example – for the criterion of “Complexity of manufacture,” we noted the average number of process impurities listed in the monographs of the USP and European Pharmacopoeia compendia. Any impurity above the pharmacopeial limit stated in the monograph would render the antibiotic substandard. The more impurities possible because of the fabrication complexity, the higher the risk that the medicines may fail to meet quality limits. A strength of this approach is that scores were relative to standardized quantitative values and the data was location-independent, which reduces subjectivity of assessments that utilize expert knowledge or team evaluation. A final risk matrix in which total severity score was plotted against total probability score was then compared to field data for validation. The higher the score the higher the risk for incidence of quality issue. However, one thing to be stressed is that a higher risk for quality issues does not comment on absolute or overall incidence of a substandard antibiotic, so it should not be interpreted by a patient to avoid these medicines, but rather that these are products for which additional quality checks or surveillance may be useful.

USP: What types of challenges did you face in doing this study?

A: Data quality and availability was the biggest challenge. Furthermore, many of the data sources used within our framework are subscription-based or use proprietary analytics which can limit accessibility. One type of data that I tried to access but could not find easily was quality control batch data from manufacturers and wholesalers on antibiotics, both from in-process and after-process quality control. Increased availability of these types of data would help to better determine where quality issues arise.

USP: What were the key outcomes from the study? What were some limitations?

A: Of the 28 antibiotic products assessed, 32% were categorized as highest risk, 46% as high risk, 18% as medium risk, and 4% as lowest risk. Importantly, the comparison of the risk scores and incidence of quality failure from the USP Medicines Quality Database showed significant correlation, demonstrating that the framework and extracted data sets were applicable to determine relative risk for substandard antibiotics. Further validation and triangulation to other datasets would help develop and refine the analysis.

One limitation is that our risk assessment did not consider differences between toxicity of impurities, starting raw materials and solvents, or quality of excipients, as this was beyond the scope of our proof-of-concept study. It will be important to include these factors in future refinements with increased data availability. Additionally, a similar assessment to that presented above for falsified medicines, which are fraudulent, is likely to differ due to differing context-specific factors, such as medicine shortages, inappropriate demand, and high prices.

USP: What do you hope your findings will lead to?

A: We hope that our study is an important step in risk prioritization and brings attention to proactive risk assessments for medicine quality, as reports of substandard medicine, and specifically substandard antibiotic circulation continue to be published. This could be better achieved, however, with increased collaboration between agencies to increase data sharing and access to data that researchers can utilize. We also hope that this framework and assessment can be expanded and adapted to other medicines, outside of antibiotics.

USP: Are there any potential policy implications?

A: Overall, resources for surveillance, interventions, and research for medicine quality are limited, making risk evaluation and prioritization tools important for resource allocation. Results of our risk assessment are valuable for guiding policy development and risk priorities for quality surveillance by both manufacturers and suppliers (i.e., products that need strengthened or additional quality checks) and stewardship for patients (i.e., ensuring that products, especially those that are high risk, are not expired and stored correctly).