Terminology and Guidelines for Diagnostic Assay Development and Validation: Best Practices for Molecular Tests

INTRODUCTION TO TEST VALIDATION

Plant pathogen diagnostics is one of the pillars of agricultural biosecurity and the protection of natural resources (Harmon et al. 2022; Schaad et al. 2003). Every year, agricultural and natural systems are threatened by plant pathogens reaching new areas or new strains overcoming resistance in agricultural crops (Boluk et al. 2020; Ceresini et al. 2018; Keith et al. 2022). Accurate and reliable diagnostic protocols allow diagnosticians to screen for plant pathogens and national plant protection organizations (NPPOs) to deploy necessary measures for eradication or management. Validated protocols ensure that diagnosticians from different laboratories (national and international) can utilize the same test and produce similar results, thus strengthening the country's ability to respond to plant pathogen emergencies (Arif et al. 2021). Validation, at its core, is the process that determines the fitness of an assay by a thorough analysis of different parameters to ensure its performance for a specific use. The validation process relies on prior method or assay optimization and standardization to ensure readiness for implementation. Regardless of the system in question, the process of validation follows similar standards and requirements.

Although the term “validation” has been used loosely in the past, a validated protocol holds incredible value to the user. Several entities have defined and differentiated between both method or assay validation and process validation, the former focused on a specific method and the latter focused on the entirety of the diagnostic system, which involves several tests and the final diagnosis based on the interpretation of all tests (van der Vlugt et al. 2007). The U.S. Food and Drug Administration, in harmony with the International Conference on Harmonization (ICH), defines analytical method validation as the process of demonstrating that an analytical procedure is suitable for its intended purpose (ICH 2005). The International Organization for Standardization (ISO) defines validation as “confirmation, through the provision of objective evidence, that the requirements for a specific intended use or application have been fulfilled” (ISO 9000 2015). It is the intended purpose and the data needs associated with it that must be considered in all steps of the validation process.

All validation guidelines start with defining the intended purpose of the test, method, or assay. This information will define the performance characteristics that must be assessed during the validation process and identify the end user, which in turn can become part of the process. Guidance on the level of validation required for a test has been established by the Integrated Consortium of Laboratory Networks and more specific to plant pathogens by NPPOs (EPPO 2021; IPPC 2016). A framework for diagnostic assay development and validation was presented by Cardwell et al. in 2018 in which tiers were identified according to the intended use. These tiers define the data that must be included in the validation process, from Tier 1 to Tier 4 (Cardwell et al. 2018b).

The most recent pathogen outbreaks experienced in numerous countries have raised, again, the need for robust diagnostic protocols that can be deployed for early detection during emergency responses (Cardwell and Bailey 2022; Ceresini et al. 2018). Method validation and international harmonization have been a topic of discussion at international forums for the last few years, giving rise to new action. VALITEST is a project funded by the European Union to improve the validation process. In this framework, test performance studies were organized and validation data produced for a number of different plant pathogens (Trontin et al. 2021). NAPPO has also launched a harmonization project focused on diagnostic methods for Tomato brown rugose fruit virus (ToBRFV). The IPPC also recognizes the value of diagnostic protocol harmonization and the use of validated protocols by the work of the Technical Panel on Diagnostic Protocols, as well as the new focus on an international diagnostic network of laboratories.

The concept of validation has been widely discussed, and although the core concepts remain unchanged, new diagnostic technologies have challenged how we see and apply these validation concepts to take scientific advances from a research tool to a diagnostic one (Gaafar et al. 2021; Maree et al. 2018). In 2018, Cardwell et al. introduced the validation process and discussed the basic concepts (Cardwell et al. 2018b). In this publication, we expand on the validation terminology, present general guidelines for conducting validation, and discuss the most common pitfalls, with the goal of facilitating the transfer of diagnostic tools from the developers’ bench to diagnosticians.

KEY CONCEPTS AND TERMINOLOGY

Designs of validation experiments are dictated by how the test will be used, and, therefore, the end use must be considered throughout the validation process. This use is in fact incorporated into the definition of a diagnostic test as the “determination according to requirements for a specific intended use or application” (ISO 9000 2015). A diagnostic test, or assay, is a component of a more general method (ISO/IEC Guide 99 2007). In a test validation, consideration is given to the biology and diversity of the pathosystem (pathogen, host), the environments in which the test might be used, and how the test data will be applied in a broader diagnostic context. Given the diversity of test targets (e.g., taxonomic groups), technologies (e.g., PCR, next generation sequencing, isothermal assays, etc.), and applications (laboratory use/field-deployable, routine/certification/regulatory purposes, etc.), the details of a validation scheme can vary considerably. However, there are key attributes of a validation that are consistently assessed despite these differences, including specificity (in silico and in vitro), sensitivity, precision, and robustness. Taken together, characterization of these attributes in a test provides evidence that the positives are positive (sensitivity, inclusive specificity) and negatives are negative (selectivity, exclusive specificity) for every end user (robustness) every time (precision). For each of these attributes, there are specific performance characteristics that can be systematically evaluated. The extent to which all these performance characteristics need to be addressed in a validation is based on who will be using the test, described by the validation tier (Table 1).

TABLE 1 Definitions of key validation terms and associated validation tier assignments

Validation attribute	Performance characteristic	Tier	Definition
Sensitivity	Analytical sensitivity	1	Limit of detection that is consistently observed
	Diagnostic sensitivity	2	Percentage of positive samples correctly identified in an experiment with known positive controls
Specificity	Analytical specificity	1	Ability to detect all members of the intended target (inclusivity) and not detect other pathogens (exclusivity) or host material (selectivity)
	Diagnostic specificity	2	Percentage of negative samples correctly identified in an experiment with known negative controls
Precision	Repeatability	1	Consistency in results with a single operator
	Intermediate precision	2	Consistency in results with multiple operators or instruments
	Reproducibility	3	Consistency in results with multiple operators and instruments
Robustness	Planned deviation analysis	3	Performance of test using materials and techniques that differ from the validated test to accommodate the resources of an end user
	Stability	4	Performance of test reagents or controls over time

View as image HTML

In Tier 1, an assay developer is creating a test to be used in their laboratory or by similar experts within their field of study (Fig. 1). The data produced from their tests might require further confirmation prior to making a judgment on how to manage a disease. At this tier, the sensitivity of the test is measured by determining analytical sensitivity. This measurement provides information on the lowest concentration of the target that can be reliably detected under ideal test conditions. This reliability is determined through the use of replicates in multiple experiments with differing concentrations of the target. These data can be expressed as a percentage representing the repeatability of the test. Specificity determinations in Tier 1 include careful considerations of test reactivity with the diversity of target and nontarget samples. Inclusive specificity test panels are designed to ensure the full range of target pathogens are detected, representing differences in genetics, phenotype, geographic origin, and hosts. Exclusive specificity testing ensures that the test method does not detect taxonomically closely related pathogens or, in some cases, other microorganisms commonly found in the primary hosts. Lastly, tests in a Tier 1 validation scheme are challenged with extracts of “healthy” plant tissues and substrates that might be used as input material by the end user. This selectivity testing is important in assessing any potential negative impacts on test performance conferred by the presence of inhibitors.

Download as PowerPoint

In Tier 2, a researcher or diagnostician is seeking to demonstrate that a test previously validated at the Tier 1 level can be used for pathogen surveillance. The analytical performance characteristics have previously been described using the limited sample sets available during test design. In Tier 2, analysts might broaden these inclusivity and exclusivity panels to account for additional pathosystems beyond the initial test design. This expanded testing increases the utility of the assay for a greater diversity of end users; however, the scope of data collection is flexible and might not be required in all circumstances. At this tier, the error rate becomes the focus of the validation, and the test fitness is challenged. The ability of the test method to detect only true positives (no false negatives) is calculated as diagnostic sensitivity. Conversely, the rate at which a test does not detect true negative samples (no false positives) is calculated as diagnostic specificity. At this stage, several analysts might perform the test to account for typical variances in technique, or the validator will perform the test using a range of typical instrumentation. This intermediate precision testing builds confidence that the test will perform consistently in Tier 3. Taken together, these performance characteristics are used to set a threshold for positive versus negative diagnoses for the test. This diagnostic threshold is the limiting value where true positives can reliably be distinguished from true negatives.

Tier 3 validation is performed by diagnosticians for a test that is introduced for broad use in the plant pathogen testing community. The test precision is once again challenged through reproducibility testing whereby the diagnostic sensitivity and specificity of the test are compared among laboratories, each with different personnel and equipment. The environment in which these tests are now performed often includes variables outside of the conditions assayed in previous validation tiers. For field-deployable assays, the test must be validated in field conditions. A particular user might deviate from the standard method to ensure that the test is useful in their laboratory, and therefore, the test must demonstrate sufficient robustness to tolerate small changes. In the framework of validation, any changes to the test method must be planned and tested in a verification study that includes variables and controls that account for risks introduced by the deviation.

Tier 4 validation assesses a test that is in active use by specialists and nonspecialists alike. This tier evaluates a test's fitness for use in coordinated international diagnostics and, as a result, is only relevant for test targets that pose significant risk to plant trade. Reproducibility and robustness are evaluated in a test performance study managed by a neutral administrative agency. Here, the stability, homogeneity, and availability of test reagents and controls are essential to test fitness as the materials are expected to be distributed globally as identical replicates.

Consistent use of validation terminology not only gives developers a clear understanding of what expectations need to be met for publication but also helps stakeholders understand whether an assay is fit for a specific purpose. There are a number of other descriptors of test performance that are useful for certain technologies and tiers, many of which can be found in the online Diagnostic Assay Terminology glossary on the American Phytopathological Society website (Cardwell et al. 2018a). It is highly recommended that authors mention the validation tier level in the abstract or title of the manuscript. This will facilitate the adaptation of the assay by appropriate end users.

GUIDELINES ON APPLYING CONCEPTS

Designing assays for validation success

When developing a diagnostic assay, one of the most important considerations is understanding the circumstances in which the assay will be utilized. Tests that have a lower analytical specificity and higher throughput can be used for testing large populations (screening tests), provided there is a secondary test with higher analytical specificity that can be used for confirmations. For assays used at point-of-care, convenience, speed, and robustness are key criteria that determine test viability. These same characteristics in a laboratory setting might be much less critical as compared with analytical sensitivity and specificity. This understanding of intended use defines the assay development needs for specificity (screening vs. confirmatory), the chosen platform (portable vs. non-portable), sample throughput (low to high throughput), and results turnaround time (minutes vs. days). Although the final use of an assay might sometimes be impossible to predict, assay developers must gather as much information as possible to ensure that all aspects of the design (such as assay technology, targeted locus, and internal control) and validation parameters are adequate for the intended purpose. Here, we discuss some of these considerations for assay design and validation, as well as common pitfalls that can be encountered throughout the process.

Understanding of the pathogen taxonomic status and phylogenetics.

The pathogen's taxonomic status is one of the most important aspects to understand when designing an assay. Without knowing the current taxonomic classification and phylogenetics of the pathogen and related genera or species, it is difficult to complete an accurate validation of the assay's analytical specificity. This encompasses not just making sure closely related taxa are included in the evaluation but also understanding intraspecific variation so the assay will detect all isolates of the targeted pathogen collected from different geographic regions and hosts (Barba et al. 2010; Cardwell et al. 2018b). Although culture collections are available for some plant pathogenic genera, for many, they are not or are in research laboratories and not publicly available (Harmon et al. 2023; McCluskey et al. 2016). Nucleic acid or infected tissue is often shared among various international scientists’ collections, and in the example of infected tissue, pathogen movement permits are required. These collaborations ultimately benefit the diagnostic community. More challenging, though, is the changing taxonomic classification of certain groups of pathogens or those that are newly discovered.

Undoubtedly, exotic plant pathogens pose a challenge, as a comprehensive understanding of their taxonomic status might not be known. Limited access to a diverse collection of isolates or importation and quarantine constraints to work with these pathogens can limit the development and validation of a diagnostic assay. This type of research often requires international collaborations, often under Material Transfer Agreements, which can delay the validation of an assay. Nevertheless, validation must always strive to include a wide diversity of the targeted pathogen, closely related species, and coexisting organisms.

Common pitfall: Assays based on outdated taxonomic information.

Taxonomic classification or systematics of many plant pathogens is an evolving process, so it is important to keep current with the literature. Although public DNA sequence databases can assist with confirming classification, it is essential to make sure species classification is consistent with current taxonomy and that sequences used are from correctly identified isolates. For example, the genus Verticillium was reevaluated in 2011 with the description of five additional species (Inderbitzin et al. 2011); confirming that species classification of accessions in GenBank reflects this taxonomic revision would be essential before using these sequences for designing a diagnostic assay. Taxonomic changes could also occur after an assay has been validated, which can trigger a reevaluation of the assay's specificity. For example, the description of a new Phyllosticta species, P. paracitricarpa (Guarnaccia et al. 2017), posed a new challenge for validated molecular diagnostic assays targeting the quarantine citrus black spot pathogen P. citricarpa. The widely used and validated PCR-based diagnostic assays for P. citricarpa will also cross-react with this newly described species. The analytical specificity of the existing assays has changed because of the description of a new species in this genus; therefore, the interpretation of final results must be reassessed.

Selecting the best target sequence for the diagnostic assay.

The performance of an assay will depend largely on the target chosen. The analytical sensitivity of a nucleic acid-based diagnostic assay is directly correlated with the copy number of the targeted sequence: the higher the copy number, the greater the sensitivity. For example, the multi-copy nrRNA ITS-based TaqMan PCR assays for specific detection of Phymatotrichopsis omnivora showed the highest analytical sensitivity compared with the beta-tubulin and RPB2-based assays (Arif et al. 2013). This is one reason most assays target repetitive sequences, such as the ribosomal DNA, multiple copy genes, or mitochondrial sequences. Choosing primer sequences that consist of multiple nucleotides unique to the target also improves the chances for consistent and robust results among laboratories, in contrast to an assay design with a single-nucleotide polymorphism (SNP) in the primer annealing site. Tests with SNP-targeted primers rely on precise annealing temperature for maintaining analytical specificity (Crandall et al. 2021; Kunjeti et al. 2016). A high level of specificity can also be attained by selecting a gene order that is not present in other taxa, thereby preventing amplification with the short cycling parameters used in TaqMan assays, even if the primers anneal to nontarget DNA (Bilodeau et al. 2014; LeBlanc et al. 2021; Miles et al. 2015, 2017). With increased access to whole genomes for a large number of plant pathogens and cheaper sequencing technologies, the potential to identify new targets has increased dramatically (Feau et al. 2018). To help facilitate future optimization of the assay in the event and possible use of the locus for detection of other taxa, it is desirable to have the DNA sequence data used to develop the assay deposited in a public database such as GenBank and alignments provided as supplementary files in publications.

Common pitfall: Inadequate diagnostic target.

Diagnostic targets that rely on the increased annealing temperature for adequate specificity are commonly used, particularly when no other targets are available. However, it is not uncommon for thermal cyclers to have a slightly different calibration, resulting in different block temperatures at the same setting. Cases where there is variation in temperature uniformity across the block have also been observed. These temperature variations can be detrimental to the performance of the assay when deployed to other diagnostic laboratories. Ideally, a laboratory would perform regular calibration of thermal cycling equipment and other gauges to minimize this variation; however, some end users of an assay might have limited funding to expend on quality assurance. Designing an assay that reduces the impact this might have on assay performance improves the chances of accurate and consistent performance among laboratories with varied resource provisions.

Optimization of analytical sensitivity.

Although the ideal target for a diagnostic assay has multiple copies to increase the analytical sensitivity, this is not always possible. However, selecting a unique target gene provides the possibility to increase sensitivity and robustness by running the assay at a lower annealing temperature. In cases where low or single-copy sequences are targeted, sensitivity can sometimes be improved by using a nested amplification with a second primer pair located within the region of the initial amplification. To avoid the extra laboratory work associated with setting up another amplification and the potential for cross contamination, it is advisable that this is done as a single-tube nested amplification (Burkhardt et al. 2018). The initial amplification primers have a higher annealing temperature than the nested primers and are present in a concentration where they are used up after 20 to 25 cycles of amplification (empirically determined with a dilution series of primers). The cycling parameters are adjusted such that after the first-round amplification with a higher annealing temperature, 40 cycles are run at the lower annealing temperature necessary for running the detection assay. Other factors to consider when optimizing analytical sensitivity include using nucleic acid extraction procedures that reduce the amount of PCR inhibitors present in the final nucleic acid extract, confirming the optimum plant part to sample for detecting the pathogen, and selection of test reagents that reliably amplify the target nucleic acid from the substrates under study. In some cases, adding AT-rich sequences at the 5′ position of the primers optimized the thermodynamics of the reaction and increased the yield and analytical sensitivity (Afonina et al. 2007; Larrea-Sarmiento et al. 2019).

Choosing the best technology.

There are several technologies available for conducting diagnostic assays. Although conventional PCR and agarose gel electrophoresis can be cost effective, it is not ideal due to increased time commitment and reduced throughput. Several real-time PCR assays that measure fluorescence as an indicator of target genomic region amplification are available to assay developers, each with their advantages and disadvantages. SYBR Green assays only need a specific primer pair and therefore have a low time commitment for assay development but lack the increased specificity and require highly specific primers with minimal background amplification and melt curve analysis to confirm amplification of the targeted template. Fluorescent probe technologies (TaqMan, molecular beacon probes, Scorpion probes, etc.) offer enhanced specificity by incorporating both primers and a probe that anneals to the amplicon to release the fluorophore that signals the amount of amplification that has occurred. The most common technology used in molecular diagnostics is TaqMan due to its generally superior analytical sensitivity and specificity performance characteristics during test design. Although TaqMan assays are often considered the gold standard, they do require purified DNA/RNA, and detection can be affected by the presence of contaminating compounds that make it through the nucleic acid extraction process and impact amplification efficiency (Nakhla et al. 2010; Stulberg et al. 2016; Tang et al. 2014).

Development of isothermal diagnostic assays that do not require DNA or RNA extraction are gaining in importance in both the medical and plant disease detection arenas. In these assays, the presence of PCR inhibitors has a limited impact on results, results can be obtained in a short period of time, and results are field-deployable so practitioners can complete diagnosis directly in the field (Larrea-Sarmiento et al. 2021). Loop-mediated isothermal amplification (LAMP) uses several primer pairs to run the assay and generates a high amount of amplified template that can be detected either colorimetrically or fluorometrically, and results can be obtained in less than 20 min (Domingo et al. 2021). In the case of recombinant polymerase amplification (RPA), incubation temperatures range from 37 to 42°C (reaction can be incubated in the palm of a closed hand), and several portable fluorometers are available for recording results directly in the field. Developer kits from TwistDx (www.twistdx.co.uk) and Agdia's AmplifyRP can be used for designing diagnostic assays, and Agdia's preformulated pathogen-specific AmplifyRP kits provide plant diagnosticians with an easy-to-use, portable tool for an increasing selection of plant pathogens (www.agdia.com). The entirety of the testing protocol, from plant tissue to result interpretation, takes 30 min or less. This detection technology was found to provide an accurate and sensitive means for detection of the quarantine plant pathogens Phytophthora ramorum, P. kernoviae, and Dickeya species (Boluk et al. 2020; Mccoy et al. 2020; Miles et al. 2015), as well as other fungal pathogens (Burkhardt et al. 2018, 2019). Kits are available for these assays that include an exonuclease that digests the targeted amplicon, thereby reducing the potential for cross contamination when working with a large number of samples.

Recently, the clustered regularly interspaced short palindromic repeats (CRISPR) and CRISPR-associated enzymes (Cas) have emerged as a new technology for ultrasensitive diagnostics. CRISPR sequences occur naturally in bacterial genomes as remnant bacteriophage DNA that can be transcribed and used as guide molecules in a bacterium's immune response to phage infection. This same mechanism can be used as a gene editing tool in a wide variety of organisms and applications (Jinek et al. 2012). Current CRISPR-Cas-based diagnostic technologies are based on the endonuclease activity of Cas enzymes after recognition and binding of guide RNA to a complementary sequence. Once activated, the Cas enzyme proceeds with the collateral cleavage of reporter molecules that can be detected by a spectrophotometer or lateral flow devices. The increased analytical sensitivity and specificity could make portable assays more desirable. CRISPR-Cas-based diagnostic assays have been developed for various plant pathogens such as ‘Candidatus Liberibacter asiaticus’ (Wheatley et al. 2021) and ToBRFV (Alon et al. 2021), with many more under development.

It is often useful to have an assay designed that facilitates confirmation of a pathogen diagnosis by DNA sequence analysis of a template. For the sequence to be informative, a target region should be selected that is of sufficient length and sequence diversity to allow for specific discrimination between taxa. The test can also be strategically designed to incorporate secondary primer-binding sites with greater specificity. For example, in the RPA assay for P. ramorum and P. kernoviae, the amplification primers were designed to enable amplification of the diagnostic target with an RPA kit that does not have an exonuclease, the product of which was amplified to generate a sequencing template by PCR with primers nested within the RPA primers (Miles et al. 2015). Secondary confirmation testing of isothermal amplification products can offer an advantage; however, this is not always desirable or recommended for general diagnostic purposes. Here, the test design must also consider the environment in which the testing occurs and the risk associated with handling of amplified products. Contamination can significantly impact the operations of a laboratory and observed specificity of a test. This is also true of isothermal assays, where the amplification occurs in a separate reaction vessel from the detection reaction, necessitating the transfer of amplicons in a multistep process (Aman et al. 2020; Babu et al. 2018; Kellner et al. 2019).

Common pitfall: Chosen technology does not meet user needs.

The technology selected for a given diagnostic assay must meet the needs of the customer, user, and stakeholders. Routine diagnostics have traditionally been conducted at more specialized laboratories; however, in the last decade, technologies have emerged to support molecular detection at point-of-care (i.e., in the field or greenhouse). Before the assay is designed, a thorough understanding of the final cost per sample and customer needs for screening in the field or confirmatory diagnostics needs to be defined. Portable technologies such as isothermal amplification or serology are excellent for quick diagnostics at the field level when decisions need to be made quickly. Unless a field test has been validated with crude plant extracts, a rapid assay might require purification of nucleic acids prior to testing, which can reduce the speed of the entire testing process and limit practical use in the field. Meanwhile, established diagnostics laboratories with high-throughput capabilities can be more cost effective for large-scale surveys when screening for specific pathogens. Untargeted technologies, such as high-throughput sequencing, are also available to support biosurveillance efforts but generally require specialized equipment and training (Liefting et al. 2021). Although most assays can be adapted to work with other instruments, a planned deviation and sometimes reoptimization might need to take place for them to be recommended for deployment (Table 2). Developing assays without an understanding of the end-user needs will decrease their chances of successful deployment and implementation.

TABLE 2 Common gaps that affect test fitness over time and their validation solutions: Test component now unavailable

1. Original design	Test target	‘Candidatus Liberibacter asiaticus’ (HLB)
	Test technology	Real time PCR
	Host tissue	Citrus leaf
	Application	Regulatory survey
2. Change in requirements	Event	Instrument change (qPCR)
	Effect	Fluorophores no longer measurable
	Solution	Substitute fluorophores
3. Risk identification	Sensitivity	Change in limit of detection
	Specificity	Change in background
4. Validation need		Equivalency testing (analytical sensitivity and specificity) comparing test on old instrument/fluorophores with new instrument/fluorophores
5. End result		Redeveloped test with broader accessibility

View as image HTML

Use of internal controls.

It is not uncommon for substrates used for nucleic acid extraction from environmental samples to have compounds that either inhibit PCR amplification or reduce amplification efficiency, especially when sampling necrotic tissue, seeds, roots, or soil samples. Although techniques used for nucleic acid extraction are often successful in preventing these compounds from making it into the final purified extract, they are not always 100% effective. To reduce the chances of a false negative result, it is important to include an amplification control in the assay to evaluate if PCR inhibitors are present. Several different approaches for using internal controls (ICs) have been used. When sampling plant tissue, an endogenous IC targeting the plant genome can be used (Arif et al. 2021; Mittelberger et al. 2020; Tooley et al. 2006). Exogenous controls can also be added prior to extraction to ensure adequate extraction performance. In the development and validation of a real-time RT-PCR test for Pospiviroids, a Dahlia latent viroid was included as an internal isolation control (Botermans et al. 2020). An exogenous IC added to the amplification can also be used with assays testing water or soil samples that lack plant DNA. These are synthetic sequences not found in nature (Bilodeau et al. 2012; Haudenshield and Hartman 2011). Although it is possible to run the control amplifications in a separate amplification from the pathogen, multiplexing them with the pathogen assay reduces work and provides a more accurate evaluation of amplification efficiency in the same reaction tube as the pathogen assay (Arif et al. 2021).

When using an IC, it is important to make sure its amplification does not compete with amplification of the pathogen target as this can cause a false negative or, in cases where pathogen quantification is desired, not accurately reflect the amount of the pathogen present. With an endogenous plant IC, because the plant DNA is at a much higher concentration than the pathogen, the best approach is to limit the amount of primers and probe (when applicable) so that IC amplification stops shortly after a positive result is obtained. In real-time PCR, this can be determined empirically by running a dilution series or orthogonal array of primer/probe concentrations with plant DNA in the same amounts that are used when assaying environmental samples and evaluating at what point the signal from logarithmic amplification levels off shortly after passing a positive C_t. For a Phytophthora ramorum TaqMan assay, this was obtained with a concentration of 100 nM for the endogenous control primers and 80 nM for the probe, compared with 1,000 nM and 400 nM, respectively, for the pathogen assay (Tooley et al. 2006). With an exogenous IC, an effective approach has been to assay a dilution series of the IC template and select a concentration that has a C_t approaching the analytical sensitivity of the pathogen assay to minimize the competition for amplification when the pathogen is detected with a lower C_t (Bilodeau et al. 2012). Differences in sensitivity can occur between different fluorometric channels in laboratory instrumentation. Where this is known, it is recommended that the IC is labeled with a fluorophore to be read on the weaker channel, as the interpretation of IC results can be strictly qualitative.

Validation of the diagnostic assay

At the very basic level, most assay developers reach Tier 1 validation when evaluating their designed assay (Fig. 1). This validation tier is meant to capture the basic performance parameters for a diagnostic assay. Analytical sensitivity and specificity are parameters evaluated early in the assay development to ensure that the assay design is optimal and worth pursuing further. Once optimized, the assay is tested against a comprehensive panel to determine inclusivity and exclusivity. The Tier 1 validation also includes an assessment of the assay repeatability, that is, the level of agreement of the assay results under the same conditions (same operator, instrumentation, etc.). At the conclusion of a Tier 1 validation, its fitness for the intended purpose must be evaluated, and a decision must be made to publish the assay for specialist use or recommend the assay for redesign and optimization. The assay can then continue to higher levels of validation depending on the need for deployment.

Design approaches for validating an assay.

The successful validation of an assay starts with a validation plan that clearly defines the purpose and objectives, reference material needed or available, risks associated, and training needs (Saunders et al. 2013). The validation will follow the optimized standard operating procedure or protocols for the assay as intended to be used at deployment. The validation plan lays out the number and type of samples, replicates, operators, and other variables that are essential to produce suitable data for the proposed validation tier and justify the recommendation for a particular use. In most cases, multiple parameters can be calculated from the same set of data (e.g., repeatability and analytical sensitivity). It is generally recommended to run a minimum of three replicates per sample for parameters such as repeatability, reproducibility, and analytical sensitivity to ensure that variability is accounted for at different sample concentrations and between/within operators; however, this number could increase if high variability is expected or observed (EPPO 2021). It is important to note the inclusion of technical replicates in methods such as real-time PCR or ELISA, but these must not be considered replicates in a validation study. Assay variability could increase as the pathogen concentration in the sample decreases; as such, it is important to quantify repeatability and reproducibility at multiple concentrations of the target. This can be easily attained by testing the assay at “high,” “medium,” and “low” dilutions of pathogen samples.

A validation plan for higher tiers (Tiers 2 to 4) must account for coordination with other operators and laboratories. These collaborations might require the use of permits to ship samples or agreements between countries or institutions for the work that will be conducted and the responsibilities of each institution. As these processes can take time, proper planning will ensure that the validation of the assay is not delayed. Interlaboratory validations are extremely beneficial to the assay validation and the successful implementation of the assay. These collaborations confirm that the assay is reproducible when used in different scenarios and that the protocol, as written, is clear to other users.

Limit of detection.

All diagnostic assays have their limitations, and it is important to understand these when interpreting test results. The limit of detection (LOD) is the quantified expression of a test's analytical sensitivity and is defined as the lowest quantity of an analyte that can be detected with a defined certainty. This can be determined by testing a dilution series of purified pathogen nucleic acid or synthetic targets that can cover the dynamic range of detection for the assay. When testing real-time PCR assays, this dilution series is essential for demonstrating the linearity of amplification and determining assay sensitivity under optimal conditions. However, to accurately determine the LOD under real conditions, it is important to also test a serial dilution in the presence of environmental DNA to mimic, as best as possible, real testing conditions. The LOD is determined by evaluating at what point amplification is not consistent among replicates or when the variation among replicates is such that it reduces the R² of the regression analysis. This knowledge will facilitate design of an efficient sampling strategy; for example, whether combining multiple samples into a single assay compromises the accuracy of detection.

Analytical specificity (inclusive and exclusive specificity).

Diagnostic assays are designed to detect a specific organism or group of organisms. One of the main parameters to consider when validating an assay is how specific it is against the target and nontarget organisms. This includes the assay's inclusivity, defined as the performance of the assay to detect the targeted organisms that include genetic variation, geographical distribution, and hosts, as well as the assay's exclusivity, defined as the performance of an assay against nontarget organisms that are either closely related to the target or that can be found coexisting in the same sample. To ensure a comprehensive understanding of the assay specificity, a large panel of samples of the target organism must be evaluated. Similarly, closely related species (within the same clades) and nontarget organisms, such as coexisting organisms or contaminants, must be tested. Although it might not be realistic to test every possible organism that can potentially be co-extracted in a sample, the range of organisms tested must reflect those present in field samples.

Common pitfall: Adjusting for increased specificity or sensitivity.

Several parameters of an assay can be adjusted that can result in increased analytical specificity against closely related species or increased sensitivity. However, there is a sensitivity-specificity trade-off in which assays designed for high analytical sensitivity might tend to have higher cross reactions with closely related or nontarget organisms, and those designed for high analytical specificity might be compromised with an undesirable sensitivity. Common factors such as adjusting annealing temperatures are often used to achieve the best outcome on this trade-off. Assays requiring narrowly defined annealing temperatures can fail when transferred to other platforms or laboratories and fail validation at higher tiers (low reproducibility). In contrast, highly specific assays might rely on single-copy genes that can result in undesirably lower analytical sensitivity.

Validating for test fitness

At the conclusion of a Tier 1 validation, the assay can be recommended for (i) use by a specialist, (ii) further validation (Tiers 2 to 4), or (iii) redesign and optimization (Fig. 1). It is important that at this stage, the needs for the assay and the state of the taxonomy and pathogen distribution are reevaluated as they could change quickly. For example, a pathogen might be considered exotic to a country at the assay design stage but might have become established in the country by the conclusion of a Tier 1 validation. These changes in distribution can impact how the assay will be utilized.

The validation of an assay continues at higher tiers to ensure adequate assay performance that can support its use by diagnosticians within a network or a wide variety of users nationally or internationally. Parameters addressing the assay's precision and robustness are evaluated and must be considered before moving to the next validation tier.

Robustness.

To evaluate the robustness of an assay and its ability to be successfully used by others, it would be helpful to evaluate performance in multiple laboratories using different equipment prior to publication/implementation. One approach that is often used is to send out a series of proficiency panels to ensure the cooperating laboratory is running the assay properly. Historically, these test performance studies have been coordinated using the framework of projects such as the Euphresco research coordination network or EU-funded projects (e.g., VALITEST) and USDA APHIS PPQ National Plant Protection Laboratory Accreditation Program; however, they can also be organized by assay developers (Mccoy et al. 2020). The first-round panel could include several positive and negative samples with a known C_t value that can be used to gauge how well the assay was run. This would be followed by a second round of samples that include a dilution series of DNA so that linearity and LOD could be evaluated, as well as samples with a low or high C_t when run in the developer's laboratory. Lastly, infected plant material can be provided to confirm the cooperating laboratory is successful with the nucleic acid extraction process. An additional benefit of having assays run in other laboratories would be to ensure the procedures used were clearly presented to eliminate possible problems with running them properly.

Validation of the diagnostic process.

Different from the validation of a diagnostic assay, the validation of a diagnostic process ensures that all components, from sampling to result interpretation, are considered when developing and validating a test. With the end use in mind, the developer will identify suitable extraction protocols (either crude or purified nucleic acid) and develop tests according to the output from these. Some diagnostic workflows also rely on multiple tests that could vary in sensitivity or specificity. A screening assay can target multiple species, whereas a confirmatory assay might be more specific to the targeted pathogens (Table 3). Understanding how the results from individual assays of a larger diagnostic protocol will be collectively interpreted to make a diagnosis is important to incorporate into a validation design. Guidance on how to interpret results is important not only to assure fitness of the test in a particular diagnostic setting but also in communicating diagnoses between agencies in larger programs. Recommending a diagnostic threshold, or the test values that mark the border between a positive and negative result, helps to standardize interpretation. This threshold value range is determined by comparing the LOD of a test (analytical sensitivity) with the maximum values observed with nonspecific background (selectivity) and cross-reactions with nontarget organisms (exclusive specificity) (Hedman et al. 2018). It is not uncommon for a laboratory to encounter samples that produce inconclusive results at or near this threshold value, in which case a developer can help analysts by recommending a confirmatory test that has an equivalent or greater analytical sensitivity (ISO DIS 13484 2017).

TABLE 3 Common gaps that affect test fitness over time and their validation solutions: Higher analytical specificity desired

1. Original design	Test target	Clavibacter michiganensis (subsp. michiganensis and nebraskensis)
	Test technology	Lateral flow
	Host tissue	Tomato or corn leaf and stem
	Application	Field testing
2. Change in requirements	Event	Cross-reaction with soil bacterium
	Effect	False positives requiring secondary test
	Solution	Design molecular assay
3. Risk identification	Sensitivity	Must be lower limit of detection
	Specificity	Must not cross-react
4. Validation need		New test development and validation for both C. m. michiganensis and C. m. nebraskensis that are cross-compatible with lateral flow assay
5. End result		New tests with higher analytical specificity

View as image HTML

Common pitfall: Inadequate sampling recommendations.

One of the first questions a diagnostician might encounter when looking to apply a test to a particular disease concern is when and where to take a sample. Unequal pathogen distribution in planta can lead to false negative reporting if these issues are not factored into the test design. Bacteria, phytoplasmas, and certain viruses such as Potato leafroll virus can be restricted to plant phloem (Bendix and Lewis 2016), requiring the preferential inclusion of vascular tissue in a test sample. Other viruses, such as Iris yellow spot virus, are unevenly distributed within a single leaf blade, and sampling occurs near the lesion site (Smith et al. 2006). Pathogen titers in woody hosts can vary by season to the extent that poor timing might lead to inconsistent results, as observed with Xylella fastidiosa (Hopkins 1981), Phytophthora ramorum (Vettraino et al. 2010), Phytoplasmas (Terlizzi and Credi 2007; Wright et al. 2022), and grapevine red blotch virus (Kahl et al. 2022; Setiono et al. 2018). This seasonal variability of pathogen distribution in perennial hosts could require year-over-year sampling in zero-tolerance programs (Villamor et al. 2022). Sampling in seed testing presents yet another challenge as the number of seeds in each composite subsample, and the number of subsamples per lot, has a direct effect on the confidence of a test result. Limited quantities of available seed in a lot can restrict the scope of a validation in determining, with statistical significance, an LOD. These complications in validation design have been addressed by testing organizations that support the seed trade industry, such as the International Seed Trade Association (ISTA). Test developers validating for seed testing are recommended to follow standard recommendations for composite subsample size and number (ISTA 2022; Morrison 1999) (Table 4). When possible, test designers should describe in detail how and when samples were taken, as well as any probable risks associated with deviations in sampling.

TABLE 4 Common gaps that affect test fitness over time and their validation solutions: Change in application

1. Original design	Test target	Tomato mottle mosaic virus (ToMMV)
	Test technology	Reverse transcription PCR
	Host tissue	Pepper and tomato leaf
	Application	Laboratory diagnostics
2. Change in requirements	Event	Seeds confirmed as propagules
	Effect	Need an international seed testing standard
	Solution	Develop assay and validate to standard
3. Risk identification	Sensitivity	Detect as few as 1 in 1,000 seeds
	Specificity	No false negatives in seed matrix
4. Validation need		Test development and validation that meets international seed testing standards, such as Internal Seed Testing Association
5. End result		Tier 4 assay validated for use in seed

View as image HTML

Assay verification (incorporating assays developed in other laboratories).

Different from assay validation, assay verification involves testing an already validated assay (by a different laboratory) and ensuring that it works in your laboratory before implementing it for diagnostic use. The assay verification relies on the previous data and protocol and ensures that the assay, when conducted as written, has performance characteristics that match those described in the initial validation. Although it might be tempting to take an assay developed in another laboratory and run it in the same way as the other assays in your laboratory, it is important to take a step back and run the assay exactly as it was reported first to make sure it is working as intended; this includes techniques for DNA extraction and selection of master mix. When this is not possible due to reagent availability or resource constraints, a verification experiment and/or risk analysis should be conducted to ensure an assay's performance characteristics are not substantially affected by the change to protocol. Although commercial extraction kits generally provide nucleic acid suitable for many downstream purposes, the final quantity of DNA/RNA and contaminants present will vary. The soil DNA extraction procedure reported by Bilodeau et al. (2012) for their soil quantification assay of Verticillium dahliae was selected after evaluating multiple extraction kits for the greatest sensitivity of detection. Other extraction kits provided a greater concentration of DNA from the sample but had more inhibitors that impacted amplification efficiency; conversely, other kits provided cleaner DNA but had much lower final concentrations of DNA that impacted sensitivity of detection. Real-time PCR master mixes have different components that can influence the analytical specificity and sensitivity of an assay. For example, when the Martin laboratory first ran the P. ramorum assay developed by Tooley et al. (2006) using a master mix from another company, there were seven false positives with other species; switching to the same ABI master mix used in the developing laboratory eliminated this problem, and the assay worked as reported (unpublished data). Sometimes a master mix was chosen by the developing laboratory because it provided the assay with a greater analytical sensitivity; switching to another one could compromise the LOD and accuracy of quantification. For example, Bilodeau et al. (2012) found that in side-by-side comparisons, the Real Master Mix without Rox (currently sold as Perfecta Multiplex qPCR ToughMix) reduced the C_t of detection between 3 and 4 compared with another commonly used master mix.

In view of the possibility of differences in calibration among thermal cyclers, it can be useful to initially run the assay under identical conditions as reported by the assay developer, with the exception of reducing the annealing temperature by a couple of degrees. If a successful amplification occurs, the annealing temperature can be incrementally increased to ensure specificity of detection. Thermal cyclers can also vary in ramp rates (degrees Celsius per second), which can impact performance of a PCR. For this reason, it is recommended to report the ramping interval used when possible.

Practical aspects of post-validation monitoring

Monitoring of assay performance.

Proper validation of an assay to ensure it works as intended is essential prior to its diagnostic use, but it should not be considered the endpoint. It is also essential to monitor assay performance over time and in other laboratories to ensure it continues to work properly and identify failure points that need to be corrected. This could represent everything from easy-to-solve issues such as refining procedures to reduce the chances of errors to having to reoptimize the assay to account for inconsistent results.

Once an assay is incorporated into routine use, the number of data points on sample types, pathogen titer, and other previously unknown variables will increase. These challenge the assay and could require aspects of the diagnostic process to be further adjusted. This often occurs when plant pathogens are identified on new hosts. Likely, these hosts were not tested during validation and might pose a challenge to the nucleic acid extraction technique and cause inhibition of target amplification. Developers can help prevent future end users from having to perform verification testing with newly identified hosts by testing plant species beyond reported hosts to include possible hosts. These can include plant species belonging to the same family or economically significant crops that are not related by taxonomic status but are commonly co-cultivated with the primary host (Morris and Moury 2019). Monitoring the assay performance when new hosts are encountered, new environmental conditions, or sample conditions is important to adjust and reoptimize as needed.

Adjusting for obsolete instrumentation or assay reagents.

When designing and optimizing an assay, it is advantageous to use well-established equipment/technology to reduce the chance that it becomes obsolete and is no longer available (Table 2). This is particularly important with consumable supplies (DNA extraction kits, master mix) as having to change these can have a meaningful impact on the sensitivity and specificity of an assay. Evaluating different brands or models of thermal cyclers in the validation phase of assay development can reduce the impact of specific equipment on the ability of laboratories to properly run the assays; however, having to change DNA extraction kits or master mixes could require a revalidation of assay sensitivity/specificity.

CONCLUSION

Discovery of an emergent threat often spurs new test development research, resulting in an assortment of options for the end user. The finding of ToBRFV in 2017 spurred the creation of various PCR, qPCR, ddPCR, LAMP, RPA, serological, and electrochemical assays (Agdia 2021; Alkowni et al. 2019; Alon et al. 2021; Bernabé-Orts et al. 2021, 2022; Fidan et al. 2021; ISHI 2019; Ling et al. 2019; Magaña-Álvarez 2021; Menzel and Winter 2021; Panno et al. 2019; Reddy et al. 2022; Rizzo et al. 2021; Rodriguez-Mendoza et al. 2019; Sarkes et al. 2020; Vargas-Hernández 2022; Yan et al. 2021). This flood of new tests necessitated an effort to evaluate and compare test performance to help users determine which test was best fit for their purposes (Giesbers et al. 2021; Luigi et al. 2022). Publication of new and improved tests has continued after the completion of these test performance studies. A similar effort was undertaken to evaluate specificity of various diagnostic assays for Phytophthora ramorum using a “ring trial” approach (Martin et al. 2009). Screening of diagnostic samples has also occurred throughout this period, even in the absence of basic test validation data. Decisions made regarding the presence or absence of a pathogen are highly consequential to the profitability and marketability of crops, as well as the control and exclusion of diseases that impact ecosystems. It is therefore essential to systematically evaluate the performance of a test before its use to ensure that diagnosticians, producers, and regulators have confidence in the results.

The extent to which a test is validated prior to publication is dependent on the purpose for which it is developed. There might be a need to identify a pest within the scope of a larger research program within a laboratory, to share a useful method with others that might encounter this pest, or to coordinate a large survey among a geographically or resource-diverse collection of facilities. These varying test applications can be related to tiered requirements for assay development. Communicating the tier at, or extent to which, a test has been assessed aids other interested parties in understanding risks associated with result interpretation.

At its core, a well-designed test gives its users confidence that the positive samples are positive and negative samples are negative every time the test is run. Validation elements such as analytical sensitivity and inclusive specificity help to assure that multiple isolates of a positive are detected at concentrations found in nature. Exclusive specificity and selectivity assessments inform users about how well the test can discriminate related species and the environment from the target. Diagnostic sensitivity and specificity testing informs on the performance of the test as compared with other methods. Collecting and sharing data on repeatability, reproducibility, and robustness are key to understanding the extent to which a test can be relied on for disease management decisions.

Despite these terms being present in the vernacular of clinical pathology for decades, they have been applied disparately in agricultural diagnostics (Sharma and Luster 2023). Measurements of LOD under idealized laboratory conditions are common, but selectivity and reproducibility assessments are less so. Test conditions selected during design need to mirror those used in the field if the analyst is to expect similar performance. These conditions incorporate all elements of a pathosystem, from localization of the pathogen with the host anatomy to the presence of inhibitors and the phytobiome likely to be encountered by a surveyor. The guidelines presented in this paper will help to inform reviewers of test development publications of minimum standards and best practices of assay validation.

There is an increasing wealth of genomic data that can be used in test design, as well as increasing access to technologies for sequencing interesting samples. This can help fuel accelerated test development but can also lead to misdiagnosis from an ill-informed choice of a sequence target. Selecting for gene target stability and specificity is key to ensuring a test has a life beyond its initial publication. Thoughtful aggregation of accessions within databases used to design the test is key to ensuring a test is broadly reactive to one taxon and no others.

Apart from the resource challenges faced by test developers, there are other unmet needs in the plant pathogen community that complicate validation. Many of these require translational research into how to resolve interferents in a sample or how to account for the developmental or seasonal differences in pathogen titer when sampling. There are many options for incorporating ICs into an assay, and not all might be suitable for a particular host or technology. There is also a fundamental need for vouchered specimens and collections, both in vitro and in silico, that can be used for modeling sensors in their various forms.

It would be unreasonable to assume that any one test developer can effectively create an assay while developing the basic informational tools and biomaterials needed during the course of a validation. This likely explains in part the limited nature of validations in publications released to date. It would also be unreasonable to expect a diagnostician to be responsible for conducting the robustness testing needed for estimating error in various test environments and repeating validations when modifications are necessary. Pathosystems are themselves unstable entities that respond to changes in their environment from the generation of divergent isolates or expanded host ranges. The scope of a test validation must also expand accordingly or the validation data risks becoming increasingly irrelevant to the modern application of the test.

For many years, EPPO has advanced the state of knowledge for test validation in plant pathology, developing standards of terms, quality management, and test validation. Efforts by the research community and commercial manufacturers to organize and publish the results of test performance comparisons have further developed the state of the field. Terminology and requirements described within this paper mirror and complement these existing standards, with the goal of increasing the uniformity and quality of validation data within the broader phytopathology research community.

Sometimes it takes a village, or a network, to make a test. Ideally, all stakeholders (reference collection providers, designers, users, and regulators) can contribute to effective test development by communicating their respective resources and needs. As new information is discovered in the field, validation could become a more continuous process if analysts are able to readily share their interesting results (i.e., new isolates, hosts) with researchers. This type of test performance feedback loop within an accessible infrastructure (such as an online network) would greatly benefit the international community and enable more efficient test validation that is more capable of responding to the increasing demands of pathogen identification.

Recommendations

Following are recommendations for enhancing the quality of test validation for the plant pathology research community and their stakeholders:

• For test developers: State the tier to which a test was validated and the intended use of the assay at the time of publication.

• For journal editors and reviewers: Require that test validation data be reported as described in this paper, to the extent that is required by the tier claimed by the author(s); request that sequence alignments used to design the assay be deposited in public databases or supplied as supplementary material in the publication.

• For diagnosticians and other test validators: Publish validation data for existing tests, especially when a higher tier of validation has been met.

• For the community: Develop a communication network of reference collection managers, test developers, and diagnosticians for improved efficiency of validation and dynamic monitoring of test performance.