The severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) and resulting COVID-19 disease have presented a relevant challenge in laboratory diagnostics [1, 2]. During the current pandemic, several SARS-CoV-2 diagnostic platforms have been developed and tested for clinical use. Serological assays are essential in the detection of anti-SARS-CoV-2 antibodies and in monitoring convalescent and vaccine-induced immunity [3]. Mainly, they play a supplementary but indispensable role in diagnosing suspected cases with negative swab, in asymptomatic or past SARS-CoV-2 infection patients, for epidemiological assessment, assuring the contact-tracing of positive cases and identifying new foci, and in evaluating vaccine efficacy and therapeutic antibody development [4, 5, 6, 7, 8].

During the last 18 months, different serologic tests detecting specific SARS-CoV-2 antibodies were developed, some of which give quantitative results; however, these tests may detect all binding antibodies to a specific target, but they do not provide correct information on the potency of functional antibodies that may be associated with protective responses. In general, when analysing the correlation between antibody titer and neutralizing capacity it is necessary to distinguish between humoral immunity induced by natural infection and humoral immunity induced by vaccination. In fact, only in the latter do the amount of antibodies and the neutralizing capacity correlate very well [9, 10, 11, 12]. Therefore, in parallel to the development of reliable quantitative serological tests, there is an urgent need for fast and sensible functional assays to improve the understanding of SARS-CoV-2 immune responses, their kinetics over time and the knowledge of antibody persistence [13]. It is unclear how long SARS-CoV-2 neutralizing antibodies last and, as a result, whether they can provide long-term protection [14]. Although, some studies have shown a correlation between binding and neutralizing antibodies, neutralizing antibody assays are still a gold standard to judge the immunity tone of a patient [15, 16].

Notably, the Virus Neutralization Test (VNT), and more specifically the Plaque Reduction Neutralization Test (PRNT) [17], is considered the gold standard for serological detection measuring neutralizing antibodies. It detects functional antibodies and its results demonstrate blocking virus infection, thus representing a strong correlative indicator of protection from disease. As PRNT variants, some Neutralization Tests (NTs) were developed by observing the Focus Reduction (FRNT) or the CytoPathic Effect (CPE) [18]. The principles are basically identical to those of PRNT, and are based on the single microscopy observation of infected foci or CPE reduction, instead of counting plaques.

All these assays use SARS-CoV-2 replicating virus. VNTs are inconvenient in that they require BioSafety Level (BSL)-3 laboratories and often require intense and time consuming (3–7 days) work as well as not being readily amenable to automatization [19].

Alternatively, pseudotype-based neutralization assays represent an improvement towards VNT with the advantage of being able to work at a lower containment level (BSL-2), in which case infectivity can be measured by marker gene expression using ELISA plate readers. However, they still require 1–2 days, skilled operators, and are affected by variations between laboratories and different preparations of viral pseudotypes.

Many efforts have been made to develop other methods to test sera for the presence of neutralizing antibodies that are more practical, simple, and at the same time, reliable.

These are commercial assays that detect SARS-CoV-2-specific antibodies directed against the Spike Receptor Binding Domain (S-RBD) protein and are capable of detecting antibodies showing neutralizing activity [20, 21]. This year, several surrogate Virus Neutralization Tests (sVNTs) have appeared on the market that, based on the manufacturers’ description, claim excellent correlation with the neutralizing titer.

Neutralizing antibodies, in most cases act by interacting with the binding between the S-RBD protein on the viral surface and its receptor, the angiotensin-2 converting enzyme (ACE-2) receptor, expressed on the target cell. Neutralizing antibodies can block virus entry and infection in the host cell through interaction with S-RBD [22, 23]. This mechanism of virus entry inhibition into the target cell reduces viral replication and consequently viral load. Indeed, the appearance of neutralizing antibodies correlates with host immunity and protects, at least temporarily, against future reinfection [24, 25]. Recent studies have shown that the level of neutralizing antibodies can vary markedly between patients with severe and mild-to-moderate symptoms [26, 27].

However, there are no current data available regarding the titer of neutralizing antibodies providing patients protection from reinfection or from disease. Whilst the mechanism of protection certainly involves cellular immunity, there are supporting data which suggest neutralizing antibodies may be a correlate of protection [24, 27, 28].

To increase comparability of the results of serologic tests detecting neutralizing antibodies to support the identification of a possible protective correlate, it is crucial that all tests are calibrated and developed using a well-characterized and globally traceable reference standard. In this regard, an International Standard (IS) containing a known amount of anti-SARS-CoV-2 immunoglobulins with well-characterized neutralizing activity was made available by the WHO in December 2020 with the aim of allowing the accurate calibration of assays to the International Unit (IU) for neutralising antibodies or Binding Antibody Units (BAU) for binding antibodies, thereby reducing interlaboratory variation and creating a common language for reporting data [29]. In light of these premises, comparative performance data and proper validation are essential to guide the appropriate use of serology in COVID-19 diagnostics [13].

Our principal aim was to compare four different assay systems to determine in vitro SARS-CoV-2 neutralizing antibody concentrations and correlating the antibody levels with the gold standard VNT. Finally, we investigated whether the sVNTs performances were closer to the gold standard, comparing them with the traditional anti-SARS-CoV-2 S-RBD or S1 based antibody assays.

Eighty-eight consecutive serum samples (F:M ratio 1.4:1, mean age = 57.0 $\pm$ 14.0 years) screened for anti-S protein antibodies at the Clinical Chemistry Laboratory of “Tor Vergata” University Hospital (Rome, Italy), were tested using four different commercial sVNTs for anti-SARS-CoV-2 IgG/IgA/IgM neutralizing antibodies and with the current gold standard live VNT. Our study cohort reasonably consisted of a mixed population (vaccinated subjects, recovered patients from COVID-19, and healthy individuals) and no longitudinal samples were included. Clinical data were obtained by retrospective medical records reviewed only for patients with discordant results between VNT and sVNTs assays. Table 1 illustrates the main features of each serological assay related to its method, declared cut-off, conversion factor, dynamic range and assay principle, according to the manufacturer’s instructions. The linear dynamic ranges have been extended, conforming to the manufacturers’ instructions. Additionally, the sera were also tested using eight different commercial kits for anti-SARS-CoV-2 IgG or total antibodies (tAb) detection, S-RBD or S1 based (Supplementary File 1).

In this study, all samples have been tested with four different commercial sVNTs for anti-SARS-CoV-2 neutralizing antibodies detection and compared with the VNT titer.

In addition, a qualitative test was used to evaluate the overall agreement rate between each sVNT and VNT. As shown in Table 2, a high agreement rate was demonstrated for all tests, ranging from 95.45% (Euroimmun) to 98.86% (Diesse, Mindray and Snibe). For each sVNT overall agreement, it has been associated a 2 $\times$ 2 contingency table (Supplementary File 2) presenting the positive/negative number of samples for each serological assay compared to the True Positive (TP) and True Negative (TN) resulted by the gold standard (VNT). Furthermore, we assessed the overall agreement between eight traditional anti-SARS-CoV-2 antibody assays (S-RBD or S1 based) and VNT titers (Table 3). To this purpose, the VNT-agreement rates between anti-SARS-CoV-2 antibody assays and sVNTs (Fig. 1) were compared. The scatter plot graph shows that the sVNT mean value (98.01 $\pm$ 1.705%) was significantly higher than traditional anti-SARS-CoV-2 antibody assays (95.45 $\pm$ 1.921%) with a p 0.05 (independent T test).

Fig. 1.

Overall agreement of eight traditional anti-SARS-CoV-2 antibody assays (S-RBD or S1 based) vs sVNTs. The sVNTs mean value (98.01 $\pm$ 1.705%) was significantly higher than anti-SARS-CoV-2 antibody assays (95.45 $\pm$ 1.921%) with a p 0.05 (Independent T test).

In Table 4, the discordant results for each sVNT and VNT assay were compared to the patients’ history: Euroimmun displayed four discordant cases; Diesse, Mindray and Snibe only one case. The discordances were possible false positive sVNT results except one possible false negative VNT case.

All quantified sVNTs results were graphically correlated to VNT in Fig. 2. The dot plot graph illustrates on the vertical axis the sVNT antibody values and on the horizontal axis the neutralization titers, expressed as the reciprocal of the serum dilution, showing an increase in VNT titers correlated to the antibody concentrations. Spearman’s statistical analysis was used in order to find the correlation of neutralizing antibodies levels versus VNT. The highest coefficient (r = 0.810) was obtained with Euroimmun sVNT (%) (Fig. 2D) and the lowest (r = 0.666) with Mindray sVNT (IU/mL) (Fig. 2B). Diesse sVNT (BAU/mL) achieved an r = 0.696 (Fig. 2A), while Maglumi sVNT (IU/mL) achieved an r = 0.779 (Fig. 2C). The statistical significance level estimated was p 0.001 in all data. Finally we calculated the median value for all commercial assays at 1/160 VNT titer. Mindray, Maglumi and Diesse had similar median values and not significantly different; 715.6 IU/mL, 726.6 IU/mL and 967.0 IU/mL respectively. Euroimmun had a median value of 90.53%.

Fig. 2.

Four graphs of correlation between sVNTs and VNT: (A) Diesse sVNT (BAU/mL) with r = 0.696. (B) Mindray sVNT (IU/mL) with r = 0.666. (C) Maglumi sVNT (IU/mL) with r = 0.779. (D) Euroimmun sVNT (% IH) with r = 0.819. Spearman’s test was used for correlation analysis.

After the first year of the COVID-19 outbreak, the role of serology shifted its focus from the acute diagnostic strategy of seroprevalence and protective herd immunity to (i) controlling the pandemic, to (ii) the critical measurements of individuals’ neutralizing antibody levels for vaccine clinical trials, and finally (iii) research studies and disease prevention.

Traditionally SARS-CoV-2 antibody assays detect the isotypes of different antibodies and use different antigenic proteins. However, since competition with ACE2 may reveal the presence of potential neutralizing antibodies for RBD binding, recently various assays mainly based on the ACE2-RBD competition have spread widely and are known as sVNT.

Despite WHO efforts, only some of the commercial kits available today use the IU suggested for the neutralization assays. In fact, in our study only two out of the four sVNTs tested express their results as IU (Mindray and Maglumi), one as a percentage of HI (Euroimmun) and one as BAU/mL (Diesse). Also, VNT uses NTs instead of IU. Conversely, the harmonization process for anti-SARS-CoV-2 antibody assays has been achieved since most commercial assays use BAU/mL. Since the minimal neutralizing antibody threshold of protection from infection is still unknown, as is how long this protection might last after natural infection or vaccination, there is a great need for assays profiling patient responses to infection, and to establish acquired immunity against SARS-CoV-2 [30, 31, 32].

The sVNTs, given their high throughput nature and no need for biosafety level 3 containment, make them accessible to the wider community, enabling greater availability for effective mass testing.

Our data confirm its potential routine use as a screening test of neutralizing antibodies in a diagnostic laboratory scenario, given the high percentage of agreement with the gold standard test (95.45%–98.86%) [10]. In addition, agreement between sVNT values and VNT was higher than SARS-CoV-2 immunoassays agreement (98.01% vs 95.02%). Regarding the association between NTs and quantitative results, we observed a higher but not significant Spearman coefficient (sVNTs: r = 0.738 vs. SARS-CoV-2 immunoassays: r = 0.711).

However, the moderate correlation observed suggests that VNT cannot be replaced by the two examined assays groups, with the purpose of assessing functional antibody activity. Interestingly, the Euroimmun sVNT is the test with the highest number of qualitative discrepancies but with the strongest quantitative correlation, meaning the different aspects of the test. Our results are partially in agreement with a recent study [10] which showed an average correlation of 0.473, assessed by Spearman’s test, between VNT and three different SARS-CoV-2 immunoassays. In detail, the correlation assessed by Matusali et al. [10] was lower compared to our results (0.473 vs. 0.724), possibly due to the different population studied (COVID-19 patients’ sera vs routine sera) and to the assays examined. In fact, the tests used in our study were all anti-SARS-CoV-2 antibody assays S-RBD or S1 based, excluding nucleocapsid based assays with a lower neutralizing capacity activity [10]. Despite great efforts in recent months to develop reliable assays, the VNT still remains the gold standard, given its intrinsic characteristics. Indeed, the sVNTs may reveal only a portion of neutralizing antibodies and not measure the total neutralizing activity directed against epitopes outside the RBD, such as the N-terminal domain of the S protein [33, 34, 35, 36].

Our correlation data between sVNT vs VNT show significant Spearman coefficients, with a statistical significance level p 0.001 in all data, to confirm that high antibody concentrations correspond to high in vitro titer values, showing the applicable correlation for use in practice. From our data, it was shown that the best sVNT was Euroimmun assays, nevertheless, the new family of serological “ACE2-RBD competitive assays” are produced for fast and large-scale rapid testing, where a medium-good agreement is sufficient. Completely, they might help to provide a screening program to reduce the high number of samples undergoing VNT and thus, might be used as monitoring tools for a proper administration of vaccination or for an individual booster dose timing schedule.

Most evaluation studies have mainly compared serological assays with VNT results [37, 38, 39, 40, 41, 42, 43]. but, to the best of our knowledge, there are no studies that have considered the two different assay families separately, anti-SARS-CoV-2 antibody assays (S-RBD or S1 based) and sVNTs, compared with each other and against the gold standard.

The main limitation of the study is the small number of negative patients included due to the fact that we studied a routine population during the pandemic and the vaccination campaign, consequently scarce of negative samples. Therefore the bias resulting by a higher number of negative samples might alter the overall agreement. A second limitation of this study was that we did not use the Genscript (cPass) [44] as the sVNT reference method. Indeed, cPass is widely accepted as a reference in the literature and it is fully approved by the FDA. Unfortunately, at this stage it was not possible to perform further comparisons.

Finally, we calculated the median value for all commercial assays at 1/160 VNT titer. Mindray, Maglumi and Diesse had similar median values while Euroimmun could not be compared because it has a percentage unit of measurement. However, all tests show that as the concentration of serum antibodies increases, so does the VNT titer.

However, the strength of our work is that it is a so-called “real-life” research study typically designed to better reflect aspects of routine care.

Starting from these premises, our comparison study aims to contribute to the refinement of anti-SARS-CoV-2 antibody testing strategies for public health use.

Further studies on larger sample numbers are required to better understand the strengths and limitations of commercially available serological assays and for the flow chart optimization of COVID-19 serological tests, including both families of tests at different levels.

Serological studies are an aid to questions about the duration of immune protection and vaccine effectiveness. Although VNT still remains the gold standard, given the good agreement between VNT and sVNT, sVNT has proved to be a potential valuable tool for screening larger populations, due to shorter response times, lower costs, and good performance. In the next future sVNT could be used to personalize decision on vaccination shedule or disease therapy.

These should be presented as follows: MP, MI, MM, MN designed the research study. MI, MP, MN, VG, FT, SS performed the research. CC, AA, MM, PC, MB and SB provided critical revision of the manuscript and valuable comments. MI, MP, MN, LS, ER, FT, SS, BL analyzed the data. MI, MP, MM, MN, ER wrote the manuscript. All authors contributed to editorial changes in the manuscript. All authors read and approved the final manuscript.

The study was performed according to local ethical approval protocol no. R.S.44.20. Informed consent was obtained from all subjects enrolled in the study. The study was in accordance with the Helsinki Declaration, as revised in 2013.

The Authors would like to thank Diesse (Diagnostica Senese, Siena, Italy), Snibe (Diagnostic, Shenzhen, China), Mindray (Medical, Shenzhen, China), Euroimmun, (Lübeck, Germany) for kindly having provided kits and Diesse for having performed live virus neutralization test (VNT) for this study, and Giada Mattiuzzo (National Institute for Biological Standards and Control, UK) for discussion and critical reading of the manuscript.

This research received no external funding.

The authors declare that they have no conflict of interest. AA, ER are serving as the Editorial Board members of this journal. We declare that AA and ER had no involvement in the peer review of this article and has no access to information regarding its peer review. Full responsibility for the editorial process for this article was delegated to AP.