Assessment of three-dimensional RNA structure prediction in CASP15.

Computation of RNA-Puzzles-style metrics The RNA-puzzles-style assessment relied mainly on the Root Mean Square Deviation (RMSD) measure complemented by the Deformation Index (DI) 22 . The RMSD is the usual measure of distance between all atoms (excluding H atoms) of the two superimposed structures. The DI score complements the RMSD values by introducing features specific to RNA in the metric in the following way. The pairs formed by the nucleotides are identified, counted, and annotated in the experimental structure. They are broadly classified as either of the Watson-Crick complementary type (WC, comprising AU, GC, or GU pairs whose geometry are compatible with the standard Watson-Crick-Franklin double helix) or of the non-Watson-Crick type (NWC). The base–base network, i.e. WC, NWC, and stacking interactions in both reference and predicted models are extracted using the MC-Annotate 23 tool. We then compute, for each of the three types of base-base interactions, the number of correctly predicted pairs, the true positive (TP), the number of predicted pairs with no correspondence in the reference model, the false positive (FP), and the number of pairs in the reference model that are not present in the predicted model, the false negative (FN). The Interaction Network Fidelity (INF) is then computed as the Matthews Correlation Coefficient, the geometric mean of the positive predictive value and sensitivity as in Gorodkin 24 , 25 : INF = ( T P T P + F P ) × ( T P T P + F N ) , The DI is then computed as: RMSD/INF. Several partial INF values (and respective DI) can be computed considering only the Watson-Crick (WC) base pairs (INF WC ), the non-Watson-Crick (NWC) base pairs (INF NWC ), both WC and NWC base pairs (INF BPS ), or the stacking interactions (INF STACK ). Finally, the Deformation Profile is a distance matrix computed as the average RMSD between the individual bases of the predicted and the reference models while superimposing each nucleotide of the predicted model over the corresponding nucleotide of the reference model one at a time. It is computed using the “dp.py” command from the “SIMINDEX” package 22 . For simplification, we also calculate the sum, mean and median of the deformation profile to account for the general accuracy of the prediction. The stereochemical correctness of the predicted models was evaluated with MolProbity 26 , which provides quality validation for 3D structures of proteins and nucleic acids. For the latter, MolProbity performs several automatic analyses, from checking the lengths of H-bonds present in the model to validating the compliance with the rotameric nature of the RNA backbone 26 , 27 . As a single measure of stereochemical correctness, we chose the clash score, i.e., the number of all types of steric clashes per thousand residues 28 . The assessment also considered the coordinate comparison metric TM-score as computed in RNA-Align 29 and the Mean of Circular Quantities 30 to assess accuracy in the torsion angle space. All the source codes and an example notebook are available at: https://github.com/RNA-Puzzles/RNA_assessment .

Computation of CASP-style metrics

Independently from the RNA-Puzzles-style computations, we assessed the accuracy of the submitted models in a manner closer to recent CASP assessments for protein structure prediction through Z RNA , a weighted Z-score average of several different assessment metrics. To perform the Z RNA evaluation, we developed the casp-rna pipeline, which encompasses our workflow for data wrangling, job parallelization, and ranking visualizations. In consideration of RNA as a flexible molecule in which irregular loops may affect RMSD measures, Z RNA explored additional metrics beyond RMSD to capture the global accuracy, local accuracy, and geometries of RNA. We selected the following tools for our ranking scheme: (1) US-align 31 , which was used to compute TM-score through a heuristic alignment approach improving on the original RNA-align 29 ; (2) Local-Global Alignment 32 which yielded GDT_TS, the average percentage of aligned C4’ atoms (rather than that of Cɑ in proteins) at cutoffs of 1 Å, 2 Å, 4 Å, and 8 Å; (3) rna-tools 33 , a toolkit used to determine the accuracy of contact classifications among base stackings, Watson-Crick interactions, and non-canonical interactions. INF scores were calculated from interaction predictions dependent on ClaRNA 34 ; (4) OpenStructure 35 , 36 , a framework used to find lDDT, a metric that measures structural similarity (unlike for proteins, our implementation of lDDT for RNA did not penalize for stereochemical violations); and (5) PHENIX, which reports a clashscore metric for all non-hydrogen bonded atom pairs that overlap worse than 0.4 Å 26 , 37 . For TM-score and GDT_TS, superposition of models and experimental models were calculated with default atoms for those packages, C3’ and C4’, respectively (repeating GDT_TS calculations with different atoms P, C3’, and C4’ gave negligible differences). Two alignment modes were considered for GDT_TS: a fixed residue-residue correspondence approach and an automated search for the best superposition, ignoring sequence; these gave nearly identical group rankings, so we opted for the former approach. INF scores were computed with ClaRNA to help increase robustness of base pair assignment for low resolution models; these values were slightly different than but highly correlated with INF scores computed with MC-Annotate, the tool normally used by RNA-Puzzles ( Supplemental Figure 1 ). Similar to the assessment of protein models in past CASP assessments, we employ a two-pass procedure for Z-scores 13 , 38 . For each target and for each of the considered metrics, the Z-score (difference with the mean, normalized by the standard deviation) was calculated by taking the mean and standard deviation for the best model from each group with respect to each considered metric. To prevent distortion from very poor outlier predictions, models with initial Z-scores that fall under a tolerance threshold of −2 were discarded, and the Z-scores were recomputed with the new mean and standard deviation. After this second pass, models with Z < −2 were re-assigned Z = −2. For Z-scores that involved linear combinations of multiple components (e.g., Z RNA ), the Z-score values for individual components were then summed. To prevent penalization of novel methods that might give poor models for some targets, the sums of just the positive Z RNA over all targets were used to make final rankings. For targets where experimentalists provided multiple conformations to either represent experimental uncertainty or bona fide conformational diversity (e.g., different copies in the crystallographic asymmetric unit or multiple conformations captured by cryo-EM 39 ), predictor models were compared to all available experimental models. Groups were rewarded based on their best score. Code for the analysis of submitted models, assessment tools, and documentation using casp-rna are available as an open-source repository at https://github.com/DasLab/casp-rna . Metrics are also available for interactive viewing on the CASP15 website at https://predictioncenter.org/casp15/results.cgi?tr_type=rna .

Generation of simple template-based structures as comparison models

As baselines for the accuracy of predicted models, we prepared template-based structures generated using homology models with the rna_thread application in Rosetta 3 (version tag v2019.27-dev60818–134-g04678680f9c). 40 For the CPEB3 ribozymes (R1107 and R1108), we generated template-based structures using the HDV ribozyme structure (PDB ID: 3NKB). We used residues 2–9, 11–39, 43–47, and 57–72 in this HDV ribozyme structure to model residues 3–8, 10–43, and 54–69 in the CPEB3 ribozymes, avoiding loop residues that were not homologous between the structures. For the class I Pre-Q1 riboswitch, we compared the type III structure R1117 to template-based structures derived from the type I structure (PDB ID: 3Q50). We used residues 2–5, 7–20, and 23–33 in the type I Pre-Q1 riboswitch structure to model residues 2–30 in the target type III Pre-Q1 riboswitch structure, again avoiding loop residues that were not homologous between the models. Non-homologous residues were left out of these simple template-based structures.

Computation of map-to-model metrics for cryo-EM targets

All models for the 6 targets determined by cryo-EM (R1126, R1128, R1136, R1138, R1149, and R1156) were assessed directly against the experimental maps. The RNA-protein targets (R1189 and R1190) were excluded from this analysis because none of the predicted models for these targets fit sufficiently well into the density to give robust alignments, but in principle this analysis is compatible with RNA-protein targets. First, models were fit into maps using two approaches. Models were aligned to the reference models (built by experimentalists into density maps) using US-align 31 and then fit locally using the command fitmap in ChimeraX 41 . We also tested an iterative phenix.dock_in_map 42 procedure. For the well-fitting models, there was very little difference between these two methods and thus the fitmap method was selected. The following programs were used to measure the listed metrics, in all cases using default parameters (1) Phenix 42 , for cross-correlation of the map and model masked by the area around the model (CC mask ), cross-correlation of the N highest density peaks in the model-generated map to the map (CC volume ), cross-correlation of the N highest density peaks in the model-generated map and N highest density peaks in the map (CC peaks ), and map-to-model Fourier shell correlation (FSC) values (N is the number of grid points inside the molecular mask); (2) TEMPy 43 , for cross-correlation coefficient (CCC), mutual information (MI), least-square fit (LSF), envelope score (ENV), and segment-based Mander’s overlap coefficient (SMOC); (3) ChimeraX 41 and in-house script for atomic inclusion 44 and density occupancy; and (4) MapQ 45 , for Q-score. An RMSD filter was selected for each target based on visual inspection. Ranking of all the models was carried out by Z-score, following the two-pass procedure described in section 2.2 . Code for the analysis can be found at https://github.com/DasLab/CASP15_RNA_EM .

Scoring against

X-ray data and molecular replacement (MR) All models for the four targets determined by X-ray crystallography (R1107, R1108, R1116 and R1117) were assessed directly against the X-ray data by superimposing them on the target structure with RNAalign 29 and calculating the Log Likelihood Gain (LLG) with respect to the diffraction data using Phaser 46 . For R1108 and R1117, with two RNA molecules in the asymmetric unit, the LLG was calculated for a single copy of the model ideally placed on chain A. A ranking of groups was derived from Z-scores computed from equal weighting of LLG, TFZ (translation-function Z-score from the model search), and CC (correlation coefficient of the map based on phases from the ideally placed model compared to the map computed by the experimentalists with their final phases). These ranking Z-scores were based on the same two-pass procedure as described in section 2.2 . Molecular Replacement was carried out using the CCP4 package 47 via CCP4 Cloud 48 and specifically the programs Phaser 46 and MOLREP 49 . Map correlation coefficients were calculated with the phenix.get_cc_mtz_pdb tool 42 . MR strategies were chosen with reference to the accuracy achieved for different targets: highly accurate predictions typically succeed unmodified while extensive manual intervention can be required with poorer predictions. For R1117, the models were used unedited from all groups. For the other targets, where overall modeling was less accurate, different editing approaches were used with the models from group TS232 (AIchemy_RNA2). For R1107 and R1108, RNA model superposition was carried out with Theseus 50 and nucleotides with higher structural variance values were removed in 10% intervals. The group 232 model_1 after removal of 10, 20, 30, 40 or 50% of nucleotides with highest structural divergence across the models was then used as a search model. MR also made use of models of the U1 small nuclear ribonucleoprotein A protein (U1ABD) component, which were generated using the AlphaFold 2 51 network in its local ColabFold implementation 52 . For R1116, a version of Slice’N’Dice 53 modified to work with RNA inputs was used to split model 1 from group TS232 into three structural segments using the Birch algorithm from the SciKit toolbox 54 .

Supplementary Material 1