Clinical research is the cornerstone of innovation in medical development. Randomized clinical trials (RCTs) form the basis of evidence-based practice [1, 2]. Negative trials are considered those that fail to reject the null hypothesis for the primary endpoint, or rephrased, those trials that failed to achieve the prespecified aims of the investigators and/or sponsors. RCTs are accompanied with a substantial investment burden and they are often funded by pharmaceutical companies [3, 4]. Therefore, it is common perception to view negative trials as a poor investment or even being valueless. However, this attribute is in fact incorrectly interpreted. Negative trials provide good evidence of which treatment lacks an effect, considering the prespecified primary outcome, or has an effect but in lesser extent than expected. Also, it is possible that inadequate design and poor conduction of the RCT could lead to negative results, even for interventions that have an effect [5]. Nevertheless, negative results prevent adoption of those interventions in clinical practice [6]. In addition, they lead to decreasing or aborting further research investment in those interventions. However, in 2017, WHO stated that 50% of clinical trials went unreported, often because of negative results [7]. These unreported results of trials could lead to a biased picture of the risk and benefits of medicines and medical devices and result in the use of suboptimal and even harmful products. In another statement the WHO calls for reporting results of older but still unpublished trials and outlines steps to improve linkages between clinical trial registry entries and their published results [8].

The importance of scientific research is gauged through the number of citations and the impact of the journal where the research was published [9]. Recently, we observed an increase in media attention regarding scientific research, which is getting distributed more progressively through a wide range of online sources [10]. It is anticipated that one third of the global population is going to use social media in 2021 [11]. One of the advantages of this new trend is that scientific research information is more accessible for the general public. Therefore, new metrics for the measurement of published papers impact on general public have been developed.

The Altmetric attention score (AAS) is one of these new metrics, and it offers a measure of the “media impact” of a scientific publication. It provides a numeric score derived from an algorithm, which allocates a score according to the detected online attention. This attention is weighted according to the source. For instance, a mention on news media generates a score of 8 while a mention on Facebook generates a score of 0.25 [12]. AAS is designed to complement traditional bibliometrics by the utilization of the information from digital platforms, and provision of immediate quantification of the impact of the scientific paper [13]. With the recent frequent utilization of AAS as a measure of scientific performance, we were interested in understanding the impact of this new score on scholarly article metrics. If a correlation between an articles AAS and it’s number of citations exist, one could argue that by promoting research articles on social media and other digital platforms the scientific impact of a article could also be increased.

A recently published study (939 articles, 76% were observational studies), showed that among cardiovascular (CV) trials, the AAS had a moderate correlation with citation counts at 3 years [14]. Poor correlation between the AAS and the number of citations has been demonstrated in the cross-sectional analysis among high-impact general medicine journals [15].

The impact of positive and negative CV RCTs using citation data and the AAS and their correlation has not been evaluated. The aim of this study was to compare the publication impact of positive and negative CV RCTs through scholarly (number of citation) and new (AAS) bibliometrics measures. We also evaluated the possibility of the presence of publication bias associated with statistical significance among included trials.

A total of 262 RCTs were included, which involved patients with CV disease and reported CV outcomes as their primary outcome. The two independent reviewers (DŠ, JK) showed a high inter-rater reliability (Cohen’s Kappa = 0.867, 95% CI 0.837–0.897) which indicates almost perfect agreement. The majority of the published trials investigated PCOs (76%). Median number of participants in trials was 575 (IQR 188–2204). More than half of CV RCTs were industry funded (57%) and investigated effect of a drug (56%) (Table 1).

There was no significant difference between positive and negative trials, considering the number of their citations (p = 0.66). However, positive trials had a tendency towards a higher AAS (p = 0.058). Positive trials had higher likelihood to be non-inferior by their design (OR 3.8, 95% CI 1.8–9.1, p = 0.001), and noninferiority trials were more often funded by the industry (OR 2.7, 95% CI 1.3–5.8, p = 0.008), when adjusted for time since published. Interestingly, positive trials used surrogate markers more often as their primary outcome (OR 2, 95% CI 1.1–3.8, p = 0.027), when adjusted for time since published (Table 1). Open access papers had a lower AAS (p = 0.01) and were less cited (p = 0.024) than papers published non-open access. This unexpected observation could be explained with additional analyses which showed that trials that were not published as open access were more often positive trials (p = 0.006) and were more often published in the category “Medicine, General & Internal” (p = 0.0001). In addition, more non-inferiority trials were present among non-open access papers vs open access papers (p = 0.047). Furthermore, 76.9% of trials published non-open access vs 53.8% of trials published open access were industry funded (p = 0.007).

More positive CV RCTs were published (p = 0.002). We analysed the frequencies of p values between 0.01 and 0.09 and speculated whether they are uniformly distributed. However, we have noticed a decreasing trend of their distribution with a discrete step at value of 0.05 (Fig. 2), which is commonly considered a threshold for claiming statistical significance. This observation raises doubts regarding the presence of publication bias, which has already been described in the literature [19]. To really confirm those suspicions, further investigation of unpublished RCTs in trial registries (e.g., ClinicalTrials.gov) should be done.

Fig. 2.

Distribution of p values at 0.01–0.09 range for primary outcomes. Number of trials in different p value ranges (for primary outcome) shown graphically.

“Medicine, General & Internal” journals had more participants (p 0.001), a higher number of citations (p 0.001) and AAS (p 0.001), compared to “Cardiac & Cardiovascular Systems” journals. In addition, they were more commonly industry funded (p = 0.049) (Table 1). Trials with an AAS larger than median (112), had a higher likelihood to be published in “Medicine, General & Internal” journal (OR 14, 95% CI 7.8–25.9, p 0.001).

Correlation between the AAS and the number of citations was moderately positively correlated ($\rho$ = 0.47, p 0.001), even after analysing positive and negative trials data separately. The correlation strength was similar when performing subgroup analyses considering the type of journal (“Medicine, General & Internal” journal vs “Cardiac & Cardiovascular Systems” journal), as well as study outcome (positive vs negative).

The highest correlation has been noticed for positive trials when log transformed and divided with time (months) ($\rho$ = 0.57, p 0.001) (Fig. 3).

Fig. 3.

Scatter plot of log transformed data of positive trials investigating correlation between the AAS and the number of citations divided with time (months) since publication. Moderate correlation has been observed between the log transformed AAS and the log transformed number of citations of positive trials.

There was no time trend associated with publication of industry funded (p = 0.37), negative (p = 0.34) and non-inferiority trials (p = 0.39), or trials that used surrogate outcomes (p = 0.62). Only 3 out of 262 trials have used Bayesian statistics for data analysis.

The results of our study suggest that there is no significant difference among recently published positive and negative CV RCTs regarding their impact measured through scholarly bibliometrics (number of citation). In other words, positive CV RCTs are not more cited then negative CV RCTs. However, we have noticed a tendency toward a higher AAS for recently published positive CV RCTs compared to the negative ones (p = 0.058). A possible reason may be a higher tendency of the general public to discuss results of positive trials on social media platforms. Another reason could be the selective social media promotion of positive cardiovascular trials by those with scientific, financial or other interests. However, this speculation needs to be further investigated.

The AAS and number of citations had a moderately positive correlation among CV RCTs which suggest that social media promotion of a CV RCT article could increase it’s citation rate and scientific impact. The correlation strength did not change when performing subgroup analyses, considering the type of journal (“Medicine, General & Internal” journal vs “Cardiac & Cardiovascular Systems” journal) and study outcome (positive vs negative). This is in line with previous research that has investigated this type of correlation [15, 20].

Analysing the AAS, we have noticed that values above median AAS were strongly associated with publications in “Medicine, General & General” journals (OR 14, 95% CI 7.8–25.9, p 0.001). Possible explanation for this observation could be a strong twitter community of “Medicine, General & Internal” journals which can have huge influence on the dissemination of research among social platforms. Also this contributes to the formation of the AAS by itself. A much higher proportion of CV RCTs (76%) published in “Medicine, General & Internal” journals than in “Cardiac & Cardiovascular Systems” journals investigated the effect of intervention on PCOs, which is not the case in some other medical settings such as oncology and hematology [20]. In addition, the majority of articles published in “Medicine, General & Internal” journals were funded by industry (p = 0.049), which can be explained by a higher financial requirement for the conduction of larger trials which are commonly published in those journals.

Positive trials had a high likelihood to be non-inferior by their design (OR 3.8, 95% CI, 1.8–9.1, p = 0.001). This observation can be caused by the fact that margins in noninferiority trials are often chosen without clearly explained reasons, together with an asymmetry in how the trials are interpreted. The non-inferiority margin is a pre-specified amount in effect for which the investigated intervention is allowed to be worse than the comparator to still be called non-inferior. If the margin is prespecified to be larger than it should be it becomes harder for the investigated intervention to fall outside of this margin and be called inferior. Unfortunately, some studies show that 58% of non-inferiority margins provide no reason as to why they are selected [21]. Also, some studies suggest inconsistency in the CONSORT recommendation for the interpretation of non-inferiority trials [22].

The limitations of this study should be addressed. We have only analysed high quality evidence research from the CV setting, which was published in high impact medical journals in 2017, 2018 and 2019. Therefore, a larger observation period and the inclusion of a larger sample of journals and research with lower quality by their design could possibly render different results. Only primary manuscripts of randomized trials were included in our analysis. Since the interpretation of the trial results is made through the effect of intervention on the primary outcome, we have not included secondary manuscripts, such as reported sub-analyses or follow up analyses. However, it should be emphasized that secondary analyses often get a lot of media attention which would have possibly affected the results of our study in case we included them in our analysis. Additional limitations of our paper could lie in the fact that the citation data could be incomplete, since we only used the WoS database for citation count which can provide a different profile of citation compared to other scholarly searched engines such as Google scholar [23]. Although this could affect absolute citation counts, it’s effect on the relative citation effects between the positive and negative trials should be irrelevant. Although we have not performed power analysis, similar studies have used same time periods so we have speculated that this period should provide us enough trials to give us enough power to detect significant results. Despite some borderline results in terms of statistical significance that we have noticed in our analysis, we have not added more studies since this was not prespecified in our protocol.

To our knowledge, this is the first paper which aimed to assess and compare the difference in impact of positive and negative CV randomized trials, measured through number of citations and AAS.

We did not find any difference between the impact of positive and negative CV RCTs considering number of citations. The higher number of published positive CV RCTs could raise suspicion on presence of publication bias associated with statistical significance, but further investigation of unpublished CV RCTs in trial registries (ClinicalTrials.gov) is needed to evaluate if this suspicion is true or false. Additional research is also necessary in order to shed more light on the role of using social media in the dissemination of CV research results and increasing it’s scientific impact.

AAS, Altmetric Attention Score; CV, cardiovascular; EHJ, European Heart Journal; IQR, interquartile range; JACC, Journal of the American College of Cardiology; JAMA, Journal of the American Medical Association; NEJM, The New England Journal of Medicine; OR, odds ratio; PCO, patient-centred outcome; RCT, randomized clinical trial; WoS, Web of Science.

MS and JK designed the research study. MS, JK, DŠ, DR, ML, AR, MP and EJ performed the research. DŠ, DR, ML, AR, MP and EJ analyzed the data. MS, JK, DŠ, DR wrote the manuscript. ML, AR, MP and EJ reviewed the manuscript. All authors contributed to editorial changes in the manuscript. All authors read and approved the final manuscript.

Not applicable.

The authors would like to thank Darko Galic and Tanja Miketic for proofreading the article.

This research received no external funding.

The authors declare no conflict of interest.