Background: High-throughput assays that can infer neutralizing activity against
SARS-CoV-2 are of great importance for assessing the immunity induced by natural
infection and COVID-19 vaccines. We aimed to evaluate the performance and degree
of correlation of three fully automated anti-SARS-CoV-2 immunoassays with
neutralization activity using a surrogate virus-neutralizing test (sVNT) from
GenScript, targeting the receptor-binding domain. Methods: 110 sera collected
from PCR-confirmed asymptomatic COVID-19 individuals were tested for neutralizing
antibodies (nAbs) using the sVNT. Positive samples were tested on three automated
immunoassays targeting different viral antigens: Mindray
CL-900i®, Abbott Architect, and Ortho VITROS®.
The diagnostic sensitivity, specificity, agreement, and correlation with the sVNT
were assessed. Receiver operating characteristic (ROC) curve analysis was
performed to determine optimal thresholds for predicting the presence of
neutralizing activity by each assay. Results: All three assays
showed 100% specificities. The highest sensitivity was 99.0%, demonstrated by
VITROS®, followed by 94.3%, for CL-900i®, and
81.0%, for Architect. Both VITROS® and CL-900i®
had the strongest correlation with the sVNT (
Following the emergence of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) in China, the infectious agent of Coronavirus Disease 2019 (COVID-19), the world faced a global health crisis with over 121 million confirmed cases and 2.68 million deaths worldwide [1]. Since the release of the SARS-CoV-2 genome sequence in January 2020, all efforts were directed towards the development of effective anti-viral drugs and vaccines [2]. Up to date, more than 180 vaccines have been developed and many of which have moved into phase III of clinical trials and received the emergency use authorization (EUA) by the U.S. Food and Drug Administration (FDA) [3]. Consequently, mass immunization campaigns were launched worldwide to ease some of the restrictions that were enforced by COVID-19. Even so, a substantial ongoing surge in cases and fatalities was still seen across many countries, indicating that the rollout of vaccination probably came out too late in these countries [4].
Neutralizing antibodies (nAbs) against SARS-CoV-2 play a key role in the quest for protective immunity and have been the reference method for assessing protection against several viruses after vaccination, including smallpox, polio, and influenza viruses [5, 6, 7, 8]. The antibody immune responses against SARS-CoV-2 have been extensively described in literature, where the limited pre-existing immunity was suggested to account for the remarkable rise in cases worldwide [4]. It was shown in several studies that SARS-CoV-2 infected individuals develop protective humoral immunity that persists for at least six months following infection [9, 10, 11, 12]. Although not suited for early diagnosis, serological assays have been widely used to detect antibody responses against SARS-CoV-2 and identify the disease prevalence in populations [1, 13]. Also, serological assays are important for measuring the efficacy of containment measures and screening convalescent sera for therapeutic and prophylactic purposes [12].
The clinical performance of the commercially available serology assays detecting Abs against SARS-CoV-2 has been extensively evaluated by numerous studies [12, 14, 15, 16]. In these studies, immunoassays that target the spike (S) protein of SARS-CoV-2 or the receptor-binding domain (RBD) have shown the best performance in terms of sensitivity and specificity [10]. However, measuring the antibody response against the SARS-CoV-2 is not sufficient for assessing protective immunity since most of these Abs are binding Abs that do not neutralize the virus [17]. On the other hand, nAbs are more indicative of protective immunity as they block the interaction between the virus and the host cells [18]. Methods for detecting nAbs include virus neutralization assays (VNAs) which are the gold standard for measuring protective immunity following infection or vaccination [17, 18, 19, 20]. However, VNAs are cumbersome, time-consuming, require biosafety level 3 (BSL-3) facilities, and cannot be implemented in routine practice [21]. Thus, correlation studies between high-throughput commercial SARS-CoV-2 serology platforms with nAbs are needed. In this regard, few studies have investigated the correlation between VNAs and SARS-CoV-2 immunoassays [9, 16, 22, 23, 24]. A recent study comparing six SARS-CoV-2 immunoassays with microneutralization assay showed variable performances depending on the target antigen with a sensitivity and specificity ranging between 43.8–87.8% and 68.3–97.5%, respectively. Thus, due to variability of reported performances among the assays, further investigation is needed to accurately identify reliable assays that provide the best correlation with neutralizing activity against SARS-CoV-2. Such assays are crucial to evaluate the effectiveness of the currently approved vaccines, determine the durability of the produced nAbs, and identify whether the vaccines are effective against the recently emerged SARS-CoV-2 variants [25, 26].
In this study, we aimed to evaluate the performance characteristics of three fully automated immunoassays (CL-900i®, Architect, and VITROS®) targeting different SARS-CoV-2 antigens in comparison with a surrogate virus-neutralizing test (sVNT) that detects nAbs against SARS-CoV-2 RBD.
The performance of the following three CE-marked SARS-CoV-2 chemiluminescent immunoassays was evaluated: CL-900i® which detects IgG Abs against viral S and nucleocapsid (N) proteins; Architect i4000SR which detects IgG Abs against the N protein; and VITROS® which detects total Abs (IgG, IgA, and IgM) against S1 antigen. The sera used were collected from volunteer individuals between July 26 and September 9, 2020, as a nationwide survey sub-study [27]. The study was approved by the Institutional Review Boards at Qatar University (QU-IRB 1492-E/21).
A total of 110 sera samples from RT-PCR-confirmed asymptomatic individuals were used to determine the sensitivity of each immunoassay was assessed in comparison to the sVNT. Details regarding the demographic and clinical characteristics of the participants are shown in Supplementary Table 1. Nasopharyngeal swab specimens from all participants were tested for SARS-CoV-2 using the Superscript III One-Step RT-PCR reaction mix with PlatinumR Taq DNA polymerase (Cat No. 12594100, ThermoFisher, Waltham, MA, USA) after RNA extraction using Qiagen RNA extraction kit. Each sample was tested using three sets of primers: one set targeting the E gene for screening and the other two sets targeting the RdRp gene for confirmation as described in [28]. Quant Studio 6 Flex real-time PCR System was used, and cycle threshold (CT) values below 30 were considered positive, while CT values between 30–33 were considered reactive.
To determine the specificity of the immunoassays and investigate cross-reactivity, we used a group of 70 pre-pandemic plasma samples collected from blood donors before 2019 and used in previous studies we conducted [12, 14, 29, 30, 31, 32]. The panel comprised of plasma samples seropositive for (a) dengue virus (n = 13), (b) influenza (n = 15), (c) West Nile Virus and Parvovirus B19 (n = 9), (d) non-respiratory viruses (n = 29) and (e) antinuclear antibodies (ANAs) (n = 4). More details about the characteristics of the pre-pandemic control samples are shown in Supplementary Table 2.
Three commercial automated serology platforms from different companies were used to detect anti-SARS-CoV-2 Abs in COVID-19 patients and pre-pandemic sera samples. These assays are: (i) CL-900i® SARS-CoV-2 IgG (Mindray, Shenzhen, China), (ii) Architect SARS-CoV-2 IgG (Abbott Laboratories, USA), and (iii) VITROS® Anti-SARS-CoV-2 Total Ab (Ortho Clinical Diagnostics, USA). All tests were carried out according to the manufacturers’ instructions. The characteristics of the immunoassays, including detection method, targeted antigens, detected antibody, and result interpretation, are summarized in Table 1 (Ref. [33, 34, 35, 36, 37, 38]).
Manufacturer | Immunoassay name | Automated system | Detection method/assay type | Detected antibody | Targeted SARS-CoV-2 antigen (s) | Result interpretation | Reference |
Mindray Bio-Medical Electronics Co., Ltd. | CL-900i® SARS-CoV-2 IgG | CL-900i® system | CLIA* | IgG | S and N* | [33, 34] | |
Abbott Laboratories | Architect SARS-CoV-2 IgG | ARCHITECT® i4000SR | CMIA* | IgG | N | [35, 36, 37] | |
Ortho Clinical Diagnostics | VITROS® Anti-SARS-CoV-2 Total Ab | VITROS® ECiQ | CLIA | IgG, IgM, and IgA | S (S1 subunit) * | [38] | |
*CLIA, chemiluminescence immunoassay; CMIA, chemiluminescent microparticle immunoassay; S: spike protein; N: nucleocapsid protein; S1: subunit of the spike protein. |
A SARS-CoV-2 surrogate virus neutralization test (sVNT) was used as a reference
test in this study (Cat. No. L00847, GenScript Biotech, NJ, USA) for detecting
nAbs against SARS-CoV-2. This assay utilizes the same format as enzyme-linked
immunosorbent assay (ELISA) by detecting Abs that competitively inhibit the
interaction between recombinant SARS-CoV-2 RBD-HRP fusion protein and recombinant
ACE2 protein coated on a 96-well plate [39]. The assay was shown to have a high
degree of correlation with the conventional pseudovirus neutralization test
(pVNT, R2 = 0.84) and demonstrated high specificity (99.9%) and sensitivity
(95.0–100%) [21]. In this study, all sera samples were tested for nAbs against
the RBD using the sVNT. According to the manufacturer’s instructions, a percent
inhibition of
Using the GenScript sVNT as the reference standard, sensitivity, specificity,
overall percent agreement, and Cohen’s Kappa coefficient were calculated to
assess the performance of each automated immunoassay. Cohen’s kappa coefficient
(
A total of 110 samples from SARS-CoV-2-confirmed asymptomatic individuals and the 70-pre-pandemic samples were screened for the presence of nAbs using the sVNT. Out of the 110 samples, 105 were positive (95.5%). All pre-pandemic samples were negative, indicating a 100% specificity of the sVNT assay. Then, the positive 105 samples were used to evaluate the performance and correlation of the three immunoassays in comparison with the sVNT.
The diagnostic performance of each automated immunoassay using the sVNT as a reference test is summarized in Table 2. The specificity of all three assays was 100% (95% CI: 94.8–100.0). Architect showed the lowest sensitivity in detecting Abs that correlates with neutralizing activity at 81.0% (95% CI: 77.2–84.7), followed by CL-900i® at 94.3% (95% CI: 89.8–98.7), and VITROS® which showed the highest sensitivity at 99.0% (95% CI: 93.8–100.0). The overall agreement with the sVNT ranged from 88.6% (95% CI: 83.9–93.3, kappa = 0.77), for Architect, to 99.3% (95% CI: 97.9–100.0, kappa = 0.98) for VITROS®. CL-900i® showed an overall agreement of 96.6% (95% CI: 93.9–99.3, kappa = 0.93).
Automated assay | Mindray CL-900i® SARS-CoV-2 IgG | Abbott architect SARS-CoV-2 IgG | VITROS® anti-SARS-CoV-2 total test |
---|---|---|---|
Sensitivity % (95% CI) | 94.3 (89.8–98.7) | 81.0 (77.2–84.7) | 99.0 (93.8–100.0) |
Specificity % (95% CI) | 100.0 (94.8–100.0) | 100.0 (94.8–100.0) | 100.0 (94.8–100.0) |
Overall agreement % (95% CI) | 96.6 (93.9–99.3) | 88.6 (83.9–93.3) | 99.3 (97.9–100.0) |
Cohen’s kappa coefficient |
0.93 (0.89–0.97) | 0.77 (0.71–0.84) | 0.98 (0.96–1.0) |
ROC curves optimized cut-off index | |||
Sensitivity using optimized cut-off indices % (95% CI) | 98.1 (93.3–99.7) | 87.6 (79.9– 92.6) | 99.0 (94.7–99.9) |
Specificity using optimized cut-off indices % (95% CI) | 100.0 (94.8–100.0) | 100.0 (94.8–100.0) | 100.0 (90.4–100.0) |
The correlation and linear regression analysis between the readings obtained
from each automated immunoassay and neutralizing activity (percent inhibition)
from the sVNT are illustrated in Fig. 1. Spearman’s correlation coefficients
(
Correlation and linear regression analysis between each
SARS-CoV-2 immunoassay ratio results and the surrogate virus neutralization test
(sVNT) percent inhibition in asymptomatic COVID-19 individuals
(n = 105). Spearman’s correlation coefficient
(
As shown in Fig. 2, ROC curve analyses showed that all three assays had an AUC
that exceeded 0.99, indicating an excellent performance for all assays
(CL-900i®: 0.9976, Architect: 0.9966, VITROS®:
0.9997, p
Empirical Receiver Operating Characteristic (ROC) curve analysis for each automated immunoassay to estimate the optimal threshold levels for predicting the presence of neutralizing antibodies against SARS-CoV-2 in asymptomatic COVID-19 individuals (n = 105). The sensitivity and specificity values correspond to the plotted points in the graphs which were used to calculate the area under the curve (AUC) and p-value for each curve plot. Based on the area under the ROC curve, the Youden Index cutoff values that maximize the sum of sensitivity and specificity were determined. The significance level was set at 0.05.
The ongoing COVID-19 pandemic requires high-throughput serological assays that can provide strong correlations with nAbs against SARS-CoV-2 and enable studying the immunity induced by natural infection and vaccines. Although conventional virus neutralization tests (cVNT) are the gold standard for detecting nAbs, they are cumbersome, time-consuming, require BSL-3 facilities, and cannot be implemented in routine practice [21]. On the other hand, sVNT is a quick and straightforward test that does not require specialized facilities and has shown an excellent performance compared with the cVNT in several studies [21, 47, 48]. However, this assay is not available yet in high throughput automated format for screening large populations compared to commercial automated assays. Therefore, we aimed to evaluate the performance of three automated immunoassays (CL-900i®, Architect, and VITROS®) using the GenScript sVNT as a reference test. Also, we aimed to assess the correlation of these assays with neutralizing activity against SARS-CoV-2. A panel of 110 samples collected from RT–PCR confirmed individuals and 70 pre-pandemic sera were used.
Following the initial screening of all samples using the sVNT, nAbs were detectable in 95.5% (105/110) of the sera samples. The sVNT did not show any cross-reactivity with the pre-pandemic samples, neither did any of the three automated assays, with 100% specificities. VITROS® demonstrated the highest sensitivity (99.0%), followed by CL-900i® (94.3%) and Architect (81.0%). These findings were concordant with a previous study that reported a 100% sensitivity for VITROS® using a microneutralization test as a reference test [49]. A possible explanation of the superior sensitivity demonstrated by VITROS® could be attributed to the fact that it detects the total Abs (IgM, IgA, and IgG) against the virus, compared to the other two assays (CL-900i® and Architect) that only detect IgG Abs. Thus, targeting multiple types of Abs could increase the chance of detecting Abs that correlate with neutralizing activity, and enhance the assay’s sensitivity. Further, although a different sample cohort was used, the sensitivity obtained by CL-900i® was comparable to the one reported in a previous study we conducted (~96%), at which the performance of three immunoassays was assessed using the sVNT test [15]. In line with our findings, Architect also showed a comparable performance in another study that reported a sensitivity of 80.5% using a microneutralization assay as the reference test [49, 50].
The correlation between the ratios obtained from each automated assay and the
percent inhibition of nAbs targeting the RBD was also examined. Both
CL-900i® and VITROS® demonstrated a strong
correlation with the sVNT compared to Architect, which showed a moderate
correlation (Fig. 1). This could be because CL-900i® detects Abs
against both the S and N proteins, while VITROS® solely targets
Abs against S1 subunit, which also contains the RBD. Similar studies have shown
that the RBD is a potent target for nAbs and that serology assays that detect Abs
against the S1 subunit or the RBD alone strongly correlate with neutralization
activity [21, 48, 51, 52, 53]. A study comparing the performance of five serology
assays with neutralization test showed that assays targeting the S protein
including DiaSorin CLIA, Beckman Coulter CLIA, and Euroimmun ELISA had the
highest correlation coefficients (rho = 0.72, 0.68, and 0.63, respectively),
similar to our findings (Fig. 1). On the other hand, the correlation between
Architects and nAbs was weaker due to the fact it only targets Abs against the N
protein. An earlier study also demonstrated a modest correlation between nAb
titers and immunoassays detecting Abs against the N protein [51]. In that study,
Pearson correlation coefficients were calculated between SARS-CoV-2
neutralization titers (EC
ROC curve analysis was performed to determine the optimal cutoff indices for each automated immunoassay that could predict the presence of nAbs against SARS-CoV-2 (Fig. 2). The calculated cutoff values showed that the manufacturer’s recommended thresholds are adequate for predicting the presence of nAbs and providing a strong correlation with the sVNT. Further, these cutoff indices can be adjusted depending on the clinical setting or research context in which they can be lowered to improve the sensitivity without affecting the specificity in high prevalence settings.
Our study had some limitations, including the small sample size and the fact that all samples were collected from asymptomatic male individuals. However, these samples were collected as part of a previous population-based study we conducted on the craft and manual workers in Qatar [27] who represent an important section of Qatar by comprising 60% of the total population. Also, the selected samples were obtained from various time points (7–35 days) after the positive RT-PCR test to account for variability and delay of elicited immune response among different individuals. Further, other studies have shown that male individuals produce weaker immune responses, particularly nAbs, than females [54, 55]. Hence, identifying the thresholds of protective immunity in male individuals is essential to avoid false-negative results and accurately estimate herd immunity. Another limitation is that our pre-pandemic sera samples did not include seropositive samples for other human coronaviruses, potentially leading to an overestimated specificity. However, our control group did include Dengue and ANA seropositive samples which have been reported to cause cross-reactivity with SARS-CoV-2 in other studies [56, 57].
In conclusion, our findings showed that both CL-900i® and VITROS® immunoassays demonstrated excellent performance in terms of sensitivity, specificity, and overall correlation with the sVNT, suggesting that they could serve as reliable and high-throughput assays for predicting the presence of protective nAbs. The identification of such reliable assays is of particular importance following the mass vaccination campaigns taking place worldwide. They could be used for screening vaccinated populations and recovered COVID-19 patients to measure the effectiveness of the developed vaccines and ensure efficient herd immunity.
Conceptualization: GKN and LJA, HMY; Methodology: AI, FMS, HAJ, DWA, IAA; Formal Analysis: HAJ; Validation: GKN, HAJ, FMS; Investigation: AI, FMS, HQ, GKN Resources: GKN; Data Curation: AI, GKN, HAJ, FMS; Writing—Original Draft Preparation: AI, FMS, HAJ; Writing, Review & Editing: HMY, GKN, LJA, AI, HQ; Visualization: HAJ, FMS, GKN; Supervision: GKN, DWA; Project Administration: GKN, DWA; Funding Acquisition: GKN. All authors have read and agreed to the published version of the manuscript.
This project was approved by the Institutional Review Boards at QU (QU-IRB 1492-E/21).
We would like to thank the medical technologist at the Medical Commission, Qatar for their help in performing the Vitros and the Architect immunoassay.
This research was funded by Qatar National Research Fund (a member of Qatar Foundation), grant No. RRC-2-032 and Qatar University internal grant QUERG-CMED-2020-2.
The authors declare no conflict of interest.
sVNT, surrogate virus-neutralizing test; ROC, Receiver operating characteristic; nAb, neutralizing antibodies; RBD, receptor-binding domain; COVID-19, Coronavirus Disease 2019; VNAs, virus neutralization assays; BSL-3, biosafety level 3; CT, Cycle threshold; CLIA, chemiluminescence immunoassay; CMIA, chemiluminescent microparticle immunoassay; ELISA, enzyme-linked immunosorbent assay; AUC, area under the curve; RT-PCR, Real-time polymerase chain reaction.