Externally validated datasets were generated from The Cancer Genome Atlas-Liver Hepatocellular Carcinoma (TCGA-LIHC) database (https://portal.gdc.cancer.gov) and the Gene Expression Omnibus (https://www.ncbi.nlm.nih.gov/geo, ID: GSE76427 and GSE64041) [25]. SHI-sequencing data and MHCC-LM3 cell sequencing data used in this study can be accessed from Supplementary Material and used after the consent of the corresponding author upon reasonable request.

All clinical samples of HCC tissues and matched adjacent normal liver tissues were obtained from patients at Sun Yat-sen University Cancer Center between 2015 and 2018. These were used for high-throughput sequencing, quantitative real-time polymerase chain reaction (qPCR), western blot, and immunohistochemistry (IHC) analyses. All clinical samples and data were accessed after obtaining approval of the Ethics Committee of Sun Yat-sen university Cancer Center (No. 202011). Written informed consents from patients were waived due to the retrospective nature of our study.

Hep-3B cells were purchased from the National Collection of Authenticated Cell Cultures (https://www.cellbank.org.cn/). MHCC-LM3 cells were obtained from the Liver Cancer Institute, Fudan University (Shanghai, China). All cells were grown in Dulbecco’s modified Eagle’s medium (DMEM, Gibco, New York, NY, USA) with high glucose and supplemented with 10% foetal bovine serum (FBS, Gibco) in a 37 °C incubator with 5% CO${}_2$ atmosphere. Recombinant human TNF-$\alpha$ (Cat No: PRP1013) and TRAIL (Cat No: PRP1028) proteins were purchased from Abbkine (Wuhan, China).

Cell lines were authenticated by the Genomics Unit at TSINGKE, using Short Tandem Repeat profiling with a commercial kit from NUHIGHbio (Cat No: NH9347, NUHIGHbio, Suzhou, China). Mycoplasma test was performed in all cell lines every other week using the MycoAlert Mycoplasma Detection Kit (Vazyme, Nanjing, China). The mycoplasma test is negative.

The full-length sequence of RCAN2 was cloned into the lentiviral expression vector pReceiver-Lv105 (GeneCopoeia, Guangzhou, China) to construct the pReceiver-Lv105-RCAN2 recombinant plasmid. Target sequences were synthesized and inserted into the shRNA expression vector psi-LVRU6GP (GeneCopoeia) in order to construct shRNA of RCAN2. The shRNA of ETS homologous factor (EHF) was constructed by the vector pLenti-U6 (OBIO, Shanghai, China) and the shRNA with a nontargeting sequence was used as a negative control.

Virus packaging in HEK 293T cells was performed by co-transfection of recombinant plasmid with Opti-MEM (Gibco) and Lenti-Pac HIV Expression Packaging Kit (GeneCopoeia). Viral particles were harvested and used to transduce HCC cells after 48 h of transfection. HCC cells (1 $\times$ 10${}^5$), including Hep-3B and MHCC-LM3 cells, were infected with 2 mL of viral particles and 5 µg/mL polybrene (YEASEN, Shanghai, China). Transfected cells were cultured in medium supplemented with puromycin at least 2 weeks to select stably expressing cells for the later experiment. The expression of RCAN2 was validated by qPCR and western blot.

Total RNA from clinical tissues and HCC cells was isolated and purified with TRIzol reagent (Invitrogen, Carlsbad, CA, USA) for later sequencing. High-throughput sequencing of samples was performed on an Illumina HiSeq 4000 platform (GENEWIZ, Suzhou, China). Sequencing reads were then cleaned and processed by hisat2, fastqc, samtools and subread.

Total RNA was extracted from cells and tissues using TRIzol reagent (Invitrogen). ReverTra Ace qPCR RT Master Mix (TOYOBO, Shanghai, China) was used to perform reverse transcription, and qPCR was then performed using the THUNDERBIRD SYBR qPCR Mix (TOYOBO). The specific reaction conditions were as follows: 95 °C for 15 s followed by 40 cycles of 95 °C for 5 s and 60 °C for 30 s. $\beta$-actin was used as the internal reference gene to normalize other gene expression levels. The primers used are list in Table 1.

Total protein from clinical tissues and HCC cells was extracted using the RIPA buffer Kit (KeyGEN BioTECH Corporation, Nanjing, China). The protein concentration was determined using the Pierce Rapid Gold BCA Protein Assay Kit (Thermo Scientific, Waltham, MA, USA). Proteins were separated by sodium dodecyl sulfate polyacrylamide gel electrophoresis (SDS-PAGE) and transferred to nitrocellulose membranes (Bio-Rad, Hercules, CA, USA). The membranes were incubated with primary antibodies overnight at 4 °C and then with secondary antibodies at room temperature for 1–2 hours. Antibodies against RCAN2, caspase-3, caspase-8, caspase-9 and DR5 were purchased from Proteintech (http://www.ptgcn.com/). The antibody against EHF was purchased from Thermo Fisher Scientific (https://www.thermofisher.cn/), while all other antibodies were purchased from CST (https://www.cellsignal.cn/).

In vitro, the proliferation rate of HCC cells was measured using the Cell Counting Kit-8 (CCK-8; Dojindo, Kumamoto, Japan) according to the manufacturer’s instructions. Hep-3B and MHCC-LM3 cells were seeded into 96-well plates at 2 $\times$ 10${}^3$ cells per well in a final volume of 100 µL. Proliferation was measured at days 0, 1, 3, and 5 using a 10% CCK-8 solution. The absorbance at 450 nm was measured after incubation for 2 h at 37 °C and used to calculate the number of viable cells.

Tissue microarrays were constructed by Guangzhou Yuebin Biotech Laboratory (Guangzhou, China). Slides with tissue microarray sections were incubated overnight (8–12 h) at 4 °C with the antibody to RCAN2 (1:250). The reaction time with chromogenic reagent was 30 s. After IHC staining, all slides were scanned using a Vectra 2 scanning system (Perkin Elmer, Waltham, MA, USA) and analysed by IHC scores with the Inform analysis system (Perkin Elmer). The median IHC score was used as the cut-off value, with patients categorized into high- and low-expression groups.

Annexin V-FITC/propidium iodide (PI) staining assay (4A Biotech Corporation, Beijing, China) and Annexin V-PE/7-AAD staining assay (YEASEN, Shanghai, China) were used to detect cells in early and late apoptotic stages with the image-flow cytometry assay. The transfected cells were harvested and washed in cold PBS and resuspended in 500 µL solution with 5 µL Annexin V-PE and 10 µL 7-AAD. For the detection of MHCC-LM3 apoptotic cells following recovery of EHF expression, cells were incubated in 500 µL solution with 5 µL Annexin V-FITC and 10 µL PI. After incubation for 10–15 min at 15–25 °C, the cells were evaluated in a CytoFLEX S machine (Beckman Coulter, Brea, CA, USA) and analysed with CytExpert software (v 2.4.0.28, Beckman Coulter) for automated image-flow cytometry. A total of 20,000 cells were analysed per sample. Data were later analysed using Flowjo software (v10.5.3, BD Biosciences, Sussex, NJ, USA).

Four- to six-week-old male BALB/c nude mice were used for xenograft experiments. For the subcutaneous HCC implantation model, 3 $\times$ 10${}^6$ cells were suspended in 100 µL of serum-free DMEM. Each nude mouse was inoculated in the subcutaneous tissue above the right groin with a micro syringe. All mice were sacrificed 4 weeks later. Tumour diameter and weight were recorded, and the diameter was calculated as follows: tumour diameter = (longest diameter + shortest diameter)/2. The animal study was approved by the Animal Care Review Committee of the Sun Yat-sen University (L025501202107011).

Differences between variables were assessed by Wilcoxon analysis, Mann-Whitney test or two-tailed Student’s t test. Kaplan–Meier analysis and log-rank tests were used to compare survival between groups. Multivariate Cox analysis was performed to identify independent factors of prognosis. Data are presented as the mean $\pm$ significant difference (SD), with the significance threshold considered as p 0.05.

The workflow for this study is shown in Fig. 1. The TCGA-LIHC database was used to examine the correlation between gene expression and TMB. This resulted in the identification of 99 TMB signature-related genes when using the thresholds of p 0.05 and $|$R$|$ $>$0.2. By using the expression of TMB signature-related genes as the clustering standard, 353 patients were divided into three subgroups: integrated cluster (IC) 1 (n = 197), IC2 (n = 155) and IC3 (n = 1) (Supplementary Fig. 1A,B). Due to its small sample size, IC3 was eliminated from the study, leaving only IC1 and IC2. The overall survival of IC1 patients was worse than that of IC2 patients (p = 0.031, Fig. 2A), while the TMB score of IC1 patients was also significantly higher (p 0.001, Fig. 2B). Using a p-adjusted value of 0.05 as the standard for significant difference, we identified 5443 differentially expressed genes between IC1 and IC2.

Next, the differentially expressed genes (n = 5443) selected above were studied by Kyoto Encyclopedia of Genes and Genomes (KEGG) enrichment analysis of biological function. This revealed that PI3K-Akt signalling was the top enrichment pathway (Fig. 2C), and suggests that phosphorylation-related pathways may underlie the phenotypic differences between IC1 and IC2. Cox proportional hazard analysis was used to identify 5 key genes related to prognosis: Galectin 3 (LGALS3), C-X-C motif chemokine ligand 1 (CXCL1), Solute carrier family 22 member 1 (SLC22A1), C-type lectin domain family 3 member B (CLEC3B) and RCAN2 (Fig. 2D). RCAN2 is a member of the regulator of calcineurin family that controls various cellular functions by affecting calcineurin activity. It was therefore selected for subsequent in vitro and in vivo studies.

The correlation coefficient between RCAN2 and TMB was calculated for multiple cancer types in the TCGA database. RCAN2 expression was found to correlate with TMB in 18 of 33 cancer types examined (Fig. 2E, Supplementary Table 1). In the large majority of these cancer types (17/18), a significant inverse correlation was observed between RCAN2 expression and TMB. This result suggests that accumulation of mutations reduces RCAN2 expression in tumours, thereby affecting prognosis.

RCAN2 mRNA was extracted from 10 clinical samples, including HCC and adjacent normal tissues, and the level of RCAN2 was measured by qPCR. Corresponding tissues were used to validate RCAN2 protein expression by IHC staining. The results of these assays showed that the RCAN2 mRNA and protein expression were lower in HCC than in adjacent normal tissues (Fig. 3A,B).

To investigate the relationship between RCAN2 protein expression and the prognosis of HCC, surgical specimens were collected from 144 HCC patients for IHC staining. RCAN2 protein expression in each specimen was analysed and recorded as the IHC score using the Inform analysis system (Perkin Elmer, USA). Patients were subsequently divided into high-RCAN2 and low-RCAN2 groups according to the median IHC score of 120. The overall survival of the high-RCAN2 group was significantly better than the low-RCAN2 group (log-rank p = 0.0009, Fig. 3C,D).

The basic levels of mRNA and protein expression of RCAN2 in tumour cells and normal hepatocytes HL-7702 are shown in Supplementary Fig. 2. Based on the above results, we selected Hep-3B with a high mRNA expression level and LM3 with a low of mRNA expression level for our study.

To explore the mechanism by which RCAN2 expression might impact the prognosis of HCC, the RCAN2-OE and RCAN2-SH plasmids were transfected into HCC cells (Hep-3B and MHCC-LM3). The expression of RCAN2 in these cells was then evaluated using qPCR and western blot analysis (Fig. 4A).

The proliferation rate of HCC cells was assessed using the CCK-8 assay to determine cell viability at different time points. HCC cells that overexpressed RCAN2 showed a reduced proliferation rate, whereas HCC cells with RCAN2 knockdown showed enhanced proliferation (Fig. 4B). These findings suggest that upregulation of RCAN2 can effectively inhibit the proliferation of HCC cells in vitro.

The biological function of RCAN2 in BALB/c nude mice was also investigated using MHCC-LM3 cells with altered RCAN2 expression. Following subcutaneous injection of cells, the tumour diameter and weight were measured to evaluate tumour growth (Fig. 4C,D). Mice injected with cells overexpressing RCAN2 (RCAN2-OE) showed significantly smaller diameter (p = 0.021) and weight (p = 0.0003) compared to mice injected with cells overexpressing vector only (vector-OE). In addition, mice injected with cells that were knocked down for RCAN2 (RCAN2-SH) showed significantly greater tumour diameter (p = 0.008) and weight (p = 0.0063) than the negative control (NC) group. These findings suggest that RCAN2 has a significant biological role in HCC, and that its overexpression can effectively suppress both the in vitro and in vivo growth of HCC.

To explore the mechanism by which RCAN2 regulates the proliferation of HCC, we screened for genes related to RCAN2 in the TCGA-LIHC database. All genes were ordered by their R value and analysed through GSEA analysis. The top three signalling pathways found to be associated with RCAN2 expression were TNF-$\alpha$ signalling via NF$\kappa$B, oxidative phosphorylation, and immune inflammation-related pathways (Fig. 5A). This suggests the TNF-$\alpha$-related extrinsic apoptotic pathway may play a pivotal role in HCC cells with differential expression of RCAN2.

Based on the results of GSEA analysis, flow cytometry was employed to evaluate apoptosis in cells with altered RCAN2 expression. Concentration of 10 ng/mL TNF-$\alpha$ and 100 ng/mL TRAIL protein were used to induce extrinsic apoptosis in HCC cells, as described in the previous studies [26, 27]. Flow cytometry revealed a higher proportion of apoptosis in cells with RCAN2-OE compared to the vector-OE group (p 0.0001 for Hep-3B, and p = 0.033 for MHCC-LM3). In contrast, the proportion of apoptotic cells decreased in HCC cells with knockdown of RCAN2 (p = 0.005 for Hep-3B, and p = 0.001 for MHCC-LM3; Fig. 5B).

Under different incubation conditions with various apoptotic inducers, changes were also observed in the expression of apoptosis-related proteins, including caspase-8, cleaved caspase-8, caspase-9, caspase-3 and cleaved caspase-3 (Fig. 6). It showed that external apoptotic inducers could improve the expression of executive proteins in the apoptotic pathway in HCC cells with RCAN2 overexpression, including caspase-8, cleaved caspase-8, caspase-9, caspase-3 and cleaved caspase-3. Meanwhile, apoptosis-related proteins were down-regulated in RCAN2-SH group. Of note, caspase-9 is implicated in the mitochondrial apoptotic pathway, suggesting that RCAN2 may modulate the apoptosis of HCC cells via the mitochondrial pathway of extrinsic apoptosis.

To explore the mechanism by which RCAN2 regulates apoptosis in HCC, we screened genes related to RCAN2 in three independent databases (GSE76427, GSE6404 and SHI-sequencing data) using a threshold of p 0.05. Five key genes (RORB, GAS6, SEMA6A, RFTN1 and EHF) were selected after intersecting the related genes in the three databases (Fig. 7A).

The relationship between five key genes and RCAN2 was validated in the MHCC-LM3 cell sequencing dataset. Compared with the RCAN2-SH group, the expression of EHF in the NC group was significantly upregulated, while the expression of the other four genes was not significantly different (Fig. 7B). Because of the positive correlation between RCAN2 and EHF in multiple independent databases, EHF was selected for subsequent mechanism verification.

We also detected the correlation in mRNA and protein expression between RCAN2 and EHF was examined in vitro. In transfected HEP-3B and MHCC-LM3 cells, the changes in EHF mRNA and protein expression mirrored those of RCAN2, as determined by qPCR and western blot (Fig. 7C,D). This result suggests that overexpression or knockdown of RCAN2 can upregulate or inhibit the expression of EHF (Fig. 7D), and corroborates a previous study that found EHF acts as an upstream transcription factor which regulates the expression of DR5 [28].

It is unclear RCAN2 promotes the apoptosis of HCC cells by regulating the expression of EHF. To address this, we transfected an EHF-SH plasmid into MHCC-LM3 cells with RCAN2-OE, as well as the corresponding control cells. Cell proliferation and apoptosis were measured by CCK-8 kit and image-flow cytometry assay, respectively. The proportion of apoptotic cells in the image-flow cytometry assay was reduced after the recovery of EHF expression (Fig. 8A). Moreover, the proliferation rate of HCC cells also increased after the recovery of EHF expression (Fig. 8B). At day 4, the cell proliferation rate of the RCAN2-OE/NC group was lower than that of the vector-OE/NC group (p = 0.004). When the expression of EHF was knocked down, the proliferation rate of cells improved significantly (RCAN2-OE/EHF-SH group vs. RCAN2-OE/NC group, p = 0.0003).

After 12 h treatment of HCC cells with recombinant TRAIL, the expression levels of caspase-8, cleaved caspase-8, caspase-9, caspase-3 and cleaved caspase-3 were evaluated to assess apoptosis. It showed that the protein expression of DR5, caspase-8, cleaved caspase-8, caspase-9, caspase-3 and cleaved caspase-3 were downregulated after the recovery of EHF expression (Fig. 8C). This result suggests that knockdown of EHF can reverse the effect of RCAN2 upregulation on the apoptosis of HCC cells, thereby restoring cell proliferation. Based on these preliminary observations, we propose that RCAN2 can promote extrinsic apoptosis of HCC cells by upregulating the expression of EHF and DR5, thus inducing apoptosis through the caspase-8/9/3 pathway.