Can commercial automated immunoassays be utilized to predict neutralizing antibodies after SARS-CoV-2 infection? A comparative study between three different assays

1. Abstract Background : High-throughput assays that can infer neutralizing activity against SARS-CoV-2 are of great importance for assessing the immunity induced by natural infection and COVID-19 vaccines. We aimed to evaluate the performance and degree of correlation of three fully automated anti-SARS-CoV-2 immunoassays with neutralization activity using a surrogate virus-neutralizing test (sVNT) from GenScript, targeting the receptor-binding domain. Methods : 110 sera collected from PCR-confirmed asymptomatic COVID-19 individuals were tested for neutralizing antibodies (nAbs) using the sVNT. Positive samples were tested on three automated immunoassays targeting different viral antigens: Mindray CL-900i®, Abbott Ar-chitect, and Ortho VITROS®. The diagnostic sensitivity, specificity, agreement, and correlation with the sVNT were assessed. Receiver operating characteristic (ROC) curve analysis was performed to determine optimal thresholds for predicting the presence of neutralizing activity by each assay. Results : All three assays showed 100% specificities.


Abstract
Background: High-throughput assays that can infer neutralizing activity against SARS-CoV-2 are of great importance for assessing the immunity induced by natural infection and COVID-19 vaccines.We aimed to evaluate the performance and degree of correlation of three fully automated anti-SARS-CoV-2 immunoassays with neutralization activity using a surrogate virus-neutralizing test (sVNT) from GenScript, targeting the receptor-binding domain.Methods: 110 sera collected from PCR-confirmed asymptomatic COVID-19 individuals were tested for neutralizing antibodies (nAbs) using the sVNT.Positive samples were tested on three automated immunoassays targeting different viral antigens: Mindray CL-900i®, Abbott Architect, and Ortho VITROS®.The diagnostic sensitivity, specificity, agreement, and correlation with the sVNT were assessed.Receiver operating characteristic (ROC) curve analysis was performed to determine optimal thresholds for predicting the presence of neutralizing activity by each assay.Results: All three assays showed 100% specificities.
The highest sensitivity was 99.0%, demonstrated by VIT-ROS®, followed by 94.3%, for CL-900i®, and 81.0%, for Architect.Both VITROS® and CL-900i® had the strongest correlation with the sVNT (ρ = 0.718 and ρ = 0.712, respectively), while Architect showed a moderate correlation (ρ = 0.618).ROC curve analysis indicated that the manufacturer's recommended cutoff values are adequate for predicting the presence of nAbs and providing a strong correlation with the sVNT.Conclusion: VITROS® and CL-900i® serological assays, which detect antibodies against SARS-CoV-2 spike protein, could serve as reliable assays to predict neutralization activity after infection or vaccination.

Introduction
Following the emergence of the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) in China, the infectious agent of Coronavirus Disease 2019 (COVID-19), the world faced a global health crisis with over 121 million confirmed cases and 2.68 million deaths worldwide [1].Since the release of the SARS-CoV-2 genome sequence in January 2020, all efforts were directed towards the development of effective anti-viral drugs and vaccines [2].Up to date, more than 180 vaccines have been developed and many of which have moved into phase III of clinical trials and received the emergency use authorization (EUA) by the U.S. Food and Drug Administration (FDA) [3].Consequently, mass immunization campaigns were launched worldwide to ease some of the restrictions that were enforced by COVID-19.Even so, a substantial ongoing surge in cases and fatalities was still seen across many countries, indicating that the rollout of vaccination probably came out too late in these countries [4].
Neutralizing antibodies (nAbs) against SARS-CoV-2 play a key role in the quest for protective immunity and have been the reference method for assessing protection against several viruses after vaccination, including smallpox, polio, and influenza viruses [5][6][7][8].The antibody immune responses against SARS-CoV-2 have been extensively described in literature, where the limited pre-existing immunity was suggested to account for the remarkable rise in cases worldwide [4].It was shown in several studies that SARS-CoV-2 infected individuals develop protective humoral immunity that persists for at least six months following infection [9][10][11][12].Although not suited for early diagnosis, serological assays have been widely used to detect antibody responses against SARS-CoV-2 and identify the disease prevalence in populations [1,13].Also, serological assays are important for measuring the efficacy of containment measures and screening convalescent sera for therapeutic and prophylactic purposes [12].
The clinical performance of the commercially available serology assays detecting Abs against SARS-CoV-2 has been extensively evaluated by numerous stud-ies [12,[14][15][16].In these studies, immunoassays that target the spike (S) protein of SARS-CoV-2 or the receptorbinding domain (RBD) have shown the best performance in terms of sensitivity and specificity [10].However, measuring the antibody response against the SARS-CoV-2 is not sufficient for assessing protective immunity since most of these Abs are binding Abs that do not neutralize the virus [17].On the other hand, nAbs are more indicative of protective immunity as they block the interaction between the virus and the host cells [18].Methods for detecting nAbs include virus neutralization assays (VNAs) which are the gold standard for measuring protective immunity following infection or vaccination [17][18][19][20].However, VNAs are cumbersome, time-consuming, require biosafety level 3 (BSL-3) facilities, and cannot be implemented in routine practice [21].Thus, correlation studies between highthroughput commercial SARS-CoV-2 serology platforms with nAbs are needed.In this regard, few studies have investigated the correlation between VNAs and SARS-CoV-2 immunoassays [9,16,[22][23][24].A recent study comparing six SARS-CoV-2 immunoassays with microneutralization assay showed variable performances depending on the target antigen with a sensitivity and specificity ranging between 43.8-87.8%and 68.3-97.5%,respectively.Thus, due to variability of reported performances among the assays, further investigation is needed to accurately identify reliable assays that provide the best correlation with neutralizing activity against SARS-CoV-2.Such assays are crucial to evaluate the effectiveness of the currently approved vaccines, determine the durability of the produced nAbs, and identify whether the vaccines are effective against the recently emerged SARS-CoV-2 variants [25,26].
In this study, we aimed to evaluate the performance characteristics of three fully automated immunoassays (CL-900i®, Architect, and VITROS®) targeting different SARS-CoV-2 antigens in comparison with a surrogate virus-neutralizing test (sVNT) that detects nAbs against SARS-CoV-2 RBD.

Study design, ethical approval, and clinical samples
The performance of the following three CEmarked SARS-CoV-2 chemiluminescent immunoassays was evaluated: CL-900i® which detects IgG Abs against viral S and nucleocapsid (N) proteins; Architect i4000SR which detects IgG Abs against the N protein; and VIT-ROS® which detects total Abs (IgG, IgA, and IgM) against S1 antigen.The sera used were collected from volunteer individuals between July 26 and September 9, 2020, as a nationwide survey sub-study [27].The study was approved by the Institutional Review Boards at Qatar University (QU-IRB 1492-E/21).
A total of 110 sera samples from RT-PCRconfirmed asymptomatic individuals were used to determine the sensitivity of each immunoassay was assessed in comparison to the sVNT.Details regarding the demographic and clinical characteristics of the participants are shown in Supplementary Table 1.Nasopharyngeal swab specimens from all participants were tested for SARS-CoV-2 using the Superscript III One-Step RT-PCR reaction mix with PlatinumR Taq DNA polymerase (Cat No. 12594100, ThermoFisher, Waltham, MA, USA) after RNA extraction using Qiagen RNA extraction kit.Each sample was tested using three sets of primers: one set targeting the E gene for screening and the other two sets targeting the RdRp gene for confirmation as described in [28].Quant Studio 6 Flex real-time PCR System was used, and cycle threshold (CT) values below 30 were considered positive, while CT values between 30-33 were considered reactive.

Neutralization assay (sVNT)
A SARS-CoV-2 surrogate virus neutralization test (sVNT) was used as a reference test in this study (Cat.No. L00847, GenScript Biotech, NJ, USA) for detecting nAbs against SARS-CoV-2.This assay utilizes the same format as enzyme-linked immunosorbent assay (ELISA) by detecting Abs that competitively inhibit the interaction between recombinant SARS-CoV-2 RBD-HRP fusion protein and recombinant ACE2 protein coated on a 96-well plate [39].The assay was shown to have a high degree of correlation with the conventional pseudovirus neutralization test (pVNT, R2 = 0.84) and demonstrated high specificity (99.9%) and sensitivity (95.0-100%) [21].In this study, all sera samples were tested for nAbs against the RBD using the sVNT.According to the manufacturer's instructions, a percent inhibition of ≥20% was considered positive (nAbs were detected), and a percent inhibition of <20% was considered negative (nAbs were not detected).

Statistical analysis
Using the GenScript sVNT as the reference standard, sensitivity, specificity, overall percent agreement, and Cohen's Kappa coefficient were calculated to assess the performance of each automated immunoassay.Cohen's kappa coefficient (κ) is a robust statistical measure of agreement used to test inter-rater reliability and to assess the possibility of agreement occurrence by chance [40,41].Kappa value can be interpreted as follows: a value of ≤0 indicates no agreement, 0.01-0.20 is a poor agreement, 0.21-0.40 is a fair agreement, 0.41-0.60 is a moderate agreement, 0.61-0.80 is substantial agreement, and 0.81-1.00 is an almost perfect agreement [40].Correlation and linear regression analysis between each automated immunoassay and the sVNT percent inhibition were performed.Nonparametric Spearman's correlation coefficient (ρ) was calculated with 95% confidence interval (95% CI), where a coefficient of <0.3 suggests no or negligible correlation, 0.3-0.5 is a weak correlation, 0.5-0.7 is a moderate correlation, 0.7-0.9 is a strong correlation, and >0.9 is a very strong correlation [42].Receiver operating characteristic (ROC) curve analysis and Youden index were used to assess the assays thresholds (cutoff indices) and identify optimized ones.A nonparametric ROC analysis was performed for each immunoassay to calculate the area under the curve (AUC).The bigger the AUC, the more accurate a tool in terms of diagnostic performance.The relation between AUC and diagnostic accuracy applies as follows: an AUC of <0.5 suggests no discrimination (ability to diagnose patients with and without the disease or condition based on the test), 0.7-0.8 is deemed to be acceptable, 0.8-0.9 is deemed excellent, and >0.9 is deemed outstanding [43,44].Youden's index is often used as a summary measure of the ROC curve to evaluate the overall discriminative power of a diagnostic procedure and to compare it with other tests [45].Youden's index (J) was calculated using the formula: J = max (sensitivity + specificity) -1 to help determine the optimal thresholds for each assay [46,47].All statistical analyses were performed using Microsoft Excel 365 software and GraphPad Prism software (Version 9.0.0,San Diego, CA, USA).

Sample selection and nAbs screening using sVNT
A total of 110 samples from SARS-CoV-2confirmed asymptomatic individuals and the 70-prepandemic samples were screened for the presence of nAbs using the sVNT.Out of the 110 samples, 105 were positive  (95.5%).All pre-pandemic samples were negative, indicating a 100% specificity of the sVNT assay.Then, the positive 105 samples were used to evaluate the performance and correlation of the three immunoassays in comparison with the sVNT.
The correlation and linear regression analysis between the readings obtained from each automated immunoassay and neutralizing activity (percent inhibition) from the sVNT are illustrated in Fig. 1.Spearman's correlation coefficients (ρ) showed a statistically significant positive correlation for all three automated assays with the sVNT (p < 0.001).The strongest correlation was shown by immunoassays targeting the S protein; VITROS® (ρ = 0.718, Fig. 1C), followed by CL-900i® (ρ = 0.712, Fig. 1A).Architect immunoassay, which targets the N protein, demonstrated the weakest correlation compared to the other two assays (ρ = 0.618, Fig. 1B).Linear regression analysis showed that all constructed models could statistically significantly predict the dependent variable (% inhibition) based on the cutoff index (COI) generated by each immunoassay with a predication precision ranging between 13.9-15.6%(standard error of estimate, SEE).The best regression model fitting the data was shown by VITROS® (r 2 = 403, SEE = 13.9%),followed by Architect (r 2 = 0.308, SEE = 15.1%) and Mindray (r 2 = 0.275, SEE = 15.6%).

Discussion
The ongoing COVID-19 pandemic requires highthroughput serological assays that can provide strong correlations with nAbs against SARS-CoV-2 and enable studying the immunity induced by natural infection and vaccines.Although conventional virus neutralization tests (cVNT) are the gold standard for detecting nAbs, they are cumbersome, time-consuming, require BSL-3 facilities, and cannot be implemented in routine practice [21].On the other hand, sVNT is a quick and straightforward test that does not require specialized facilities and has shown an excellent performance compared with the cVNT in several studies [21,47,48].However, this assay is not available yet in high throughput automated format for screening large populations compared to commercial automated assays.Therefore, we aimed to evaluate the performance of three automated immunoassays (CL-900i®, Architect, and VIT-ROS®) using the GenScript sVNT as a reference test.Also, we aimed to assess the correlation of these assays with neutralizing activity against SARS-CoV-2.A panel of 110 samples collected from RT-PCR confirmed individuals and 70 pre-pandemic sera were used.
Following the initial screening of all samples using the sVNT, nAbs were detectable in 95.5% (105/110) of the sera samples.The sVNT did not show any crossreactivity with the pre-pandemic samples, neither did any of the three automated assays, with 100% specificities.VIT-ROS® demonstrated the highest sensitivity (99.0%), followed by CL-900i® (94.3%) and Architect (81.0%).These findings were concordant with a previous study that reported a 100% sensitivity for VITROS® using a microneutralization test as a reference test [49].A possible explanation of the superior sensitivity demonstrated by VITROS® could be attributed to the fact that it detects the total Abs (IgM, IgA, and IgG) against the virus, compared to the other two assays (CL-900i® and Architect) that only detect IgG Abs.Thus, targeting multiple types of Abs could increase the chance of detecting Abs that correlate with neutralizing activity, and enhance the assay's sensitivity.Further, although a different sample cohort was used, the sensitivity obtained by CL-900i® was comparable to the one reported in a previous study we conducted (~96%), at which the performance of three immunoassays was assessed using the sVNT test [15].In line with our findings, Architect also showed a comparable performance in another study that reported a sensitivity of 80.5% using a microneutralization assay as the reference test [49,50].The correlation between the ratios obtained from each automated assay and the percent inhibition of nAbs targeting the RBD was also examined.Both CL-900i® and VITROS® demonstrated a strong correlation with the sVNT compared to Architect, which showed a moderate correlation (Fig. 1).This could be because CL-900i® detects Abs against both the S and N proteins, while VIT-ROS® solely targets Abs against S1 subunit, which also contains the RBD.Similar studies have shown that the RBD is a potent target for nAbs and that serology assays that detect Abs against the S1 subunit or the RBD alone strongly correlate with neutralization activity [21,48,[51][52][53].A study comparing the performance of five serology assays with neutralization test showed that assays targeting the S protein including DiaSorin CLIA, Beckman Coulter CLIA, and Euroimmun ELISA had the highest correlation coefficients (rho = 0.72, 0.68, and 0.63, respectively), similar to our findings (Fig. 1).On the other hand, the correlation between Architects and nAbs was weaker due to the fact it only targets Abs against the N protein.An earlier study also demonstrated a modest correlation between nAb titers and immunoassays detecting Abs against the N protein [51].In that study, Pearson correlation coefficients were calculated between SARS-CoV-2 neutralization titers (EC 50 ) and the ratios reported by Roche and Abbot immunoassays.The correlation coefficient was 0.29 for Roche and 0.47 for Abbot, denoting a moderate correlation with nAbs for both assays.
ROC curve analysis was performed to determine the optimal cutoff indices for each automated immunoassay that could predict the presence of nAbs against SARS-CoV-2 (Fig. 2).The calculated cutoff values showed that the manufacturer's recommended thresholds are adequate for predicting the presence of nAbs and providing a strong correlation with the sVNT.Further, these cutoff indices can be adjusted depending on the clinical setting or research context in which they can be lowered to improve the sensitivity without affecting the specificity in high prevalence settings.
Our study had some limitations, including the small sample size and the fact that all samples were collected from asymptomatic male individuals.However, these samples were collected as part of a previous population-based study we conducted on the craft and manual workers in Qatar [27] who represent an important section of Qatar by comprising 60% of the total population.Also, the selected samples were obtained from various time points (7-35 days) after the positive RT-PCR test to account for variability and delay of elicited immune response among different individuals.Further, other studies have shown that male individuals produce weaker immune responses, particularly nAbs, than females [54,55].Hence, identifying the thresholds of protective immunity in male individuals is essential to avoid false-negative results and accurately estimate herd immunity.limitation is that our prepandemic sera samples did not include seropositive samples for other human coronaviruses, potentially leading to an overestimated specificity.However, our control group did include Dengue and ANA seropositive samples which have been reported to cause cross-reactivity with SARS-CoV-2 in other studies [56,57].
In conclusion, our findings showed that both CL-900i® and VITROS® immunoassays demonstrated excellent performance in terms of sensitivity, specificity, and overall correlation with the sVNT, suggesting that they could serve as reliable and high-throughput assays for predicting the presence of protective nAbs.The identification of such reliable assays is of particular importance following the mass vaccination campaigns taking place worldwide.They could be used for screening vaccinated populations and recovered COVID-19 patients to measure the effectiveness of the developed vaccines and ensure efficient herd immunity.

Ethics approval and consent to participate
This project was approved by the Institutional Review Boards at QU (QU-IRB 1492-E/21).

Acknowledgment
We would like to thank the medical technologist at the Medical Commission, Qatar for their help in performing the Vitros and the Architect immunoassay.

Funding
This research was funded by Qatar National Research Fund (a member of Qatar Foundation), grant No. RRC-2-032 and Qatar University internal grant QUERG-CMED-2020-2.

Fig. 2 .
Fig. 2. Empirical Receiver Operating Characteristic (ROC) curve analysis for each automated immunoassay to estimate the optimal threshold levels for predicting the presence of neutralizing antibodies against SARS-CoV-2 in asymptomatic COVID-19 individuals (n = 105).The sensitivity and specificity values correspond to the plotted points in the graphs which were used to calculate the area under the curve (AUC) and p-value for each curve plot.Based on the area under the ROC curve, the Youden Index cutoff values that maximize the sum of sensitivity and specificity were determined.The significance level was set at 0.05.