Validation methods for low-resolution fitting of atomic structures to electron microscopy data

Introduction

Due to dramatic improvements in experimental methods and computational techniques, electron cryo-microscopy (cryo-EM) has matured into a powerful collection of methods that allow the high-resolution visualization of the structure and the dynamics of an extraordinary range of biological assemblies in their native aqueous environment. Recent hardware and software developments have revolutionized the field [1]. The increased signal-to-noise ratio of a new generation of cameras that detect electrons directly [2] in combination with their ability to correct for beam-induced movements, have allowed the field to obtain structural information even for particles with low or no symmetry at resolutions around 3 Å [3–8], sufficient to build de novo structural models [9].

However, the majority of reconstructions obtained by cryo-EM are of insufficient resolution for such direct structure determination. In fact, currently over 70% of the reconstructions deposited in the electron microscopy data bank [10] do not reach a resolution of better than 10 Å. While at resolutions between 5 and 10 Å secondary structural elements are often visible as rods (α-helices) and sheets (β-sheets), at resolutions below the 10-Å mark, internal features of the reconstructions are not straightforward to interpret (Figure 1).

As cryo-EM methodology continues to improve, atomic-resolution reconstructions are likely to become more common. These reconstructions will likely be of highly rigid molecules. At the same time, the advances in cryo-EM technology will also open the door to structure determination of complexes that were previously too small, too heterogeneous, too flexible, or otherwise challenging, albeit at lower resolution. In addition, electron cryo-tomography has become a powerful alternative for structure determination of samples that are not amenable to single-particle approaches. New software developments and careful experimental design [11] enable the determination of structures from cryo tomograms at around 8 Å, but resolutions below 20 Å are more common. As a net-effect, the majority of cryo-EM reconstructions are likely to remain at the resolution range worse than 10 Å for the foreseeable future.

Fitting of atomic components into cryo-EM density maps insufficient for direct de-novo model building is routinely used to understand the structure and function of these macromolecular machines. Many fitting methods have been developed, but standard protocols for successful fitting remain to be established. Broadly, fitting methods can be divided into two major groups, rigid-body and flexible fitting methods. In rigid-body fitting approaches the atomic structures of components are fitted as single units. These units can be composed of entire proteins, domains, or even smaller groups of structural element. In flexible fitting approaches the entire atomic structures are allowed to distort in some way to improve the fit with the reconstruction, subject to constraints such as molecular dynamics force-fields or normal modes to counter-balance fitting of spurious noise. Comprehensive reviews of the various fitting approaches that are available were provided in several recent articles [12–14].

Despite the plethora of available fitting techniques, generally accepted criteria for assessing the accuracy and quality of the fitted models have not been established yet [15]. In this review, our aim is to present the general concepts underlying current validation ideas as they relate to fitting of atomic models into cryo-EM reconstructions, with an emphasis on reconstructions with resolutions that do not reach the sub-nanometer range.

Accuracy versus precision

In the context of fitting, it is important to emphasize the difference between accuracy and precision (Figure 2). In short, a fit is precise if similar fits are obtained with repeated runs. In contrast, a fit is accurate if it is close to the true structure (or ensemble of structures) underlying the data. Consensus approaches that compare results from different fitting methods [16,17] or from multiple scoring functions [18] using a single data set, for example, can only inform on the precision of the fit. While it has been claimed that precision gives a lower bound for the accuracy, this is not necessarily true (Figure 2). In fact, in the context of fitting atomic structures into cryo-EM reconstructions, the fitting that appears less precise can actually be more accurate than a fit with the same center position and a narrower spread. The reason is that cryo-EM reconstructions, unlike crystal structures, often represent an ensemble of conformers that coexisted in the sample at the time of freezing, reflecting the structural dynamics of the complex in solution. This can also lead to anisotropy of the resolution in the reconstructions, further complicating the issue.

It is not immediately clear how precision can be quantified within the context of fitting atomic models. This issue is of major importance especially at low resolution, where ambiguities may arise from the geometry of the reconstruction alone [19] and an objective criterion to allow favoring one solution over another is needed. One approach towards this goal is the use of statistical methods to define confidence intervals (see below), which will then allow to define an objective precision estimate. However, the real quantity that is of interest is the accuracy or how close the obtained fit is to the true structure. Without knowledge of the true structure accuracy cannot be directly assessed.

Sources of errors

Every cryo-EM reconstruction has some uncertainty due to the presence of noise and its generally limited resolution. If only such random errors exist, a quantified precision measure may actually be a reasonable estimate for the accuracy of a fit. More troublesome in this context are potential systematic errors such as those originating in the electron microscopy data-collection and reconstruction procedures. These include misestimations of the magnification, incomplete corrections of the microscope’s contrast transfer function, uneven distribution of projection images in Euler space, and misestimation of the resolution. All of these can introduce bias and throw off the fits from the true solution in a non-random fashion. Other factors that can bias fitting results in significant ways include incompleteness of the structure to be fitted in relation to the target reconstruction and the possible presence of conformational mixtures. Generally, rigid-body techniques are less susceptible to generating artifacts in the presence of errors simply because the underlying structure is kept intact and only six parameters per rigid body, three for the center-of-mass position and three for the orientation, need to be determined.

Another source of systematic errors is the fact that, at low resolution, the amount of structural information that can be obtained is limited. Care must be taken that the number of refinable parameters used during fitting does not exceed the number of independent observations in the reconstruction. Otherwise, overfitting will inevitably ensue. Even fitting of a single rigid body using only the six rotational and translational degrees of freedom can lead to ambiguities in the resulting models at intermediate resolution [19]. Generally, all available experimental restrains that are independent of the fitting process should be employed for validation purposes. However, inclusion of such restrains into the fitting process can sometimes be helpful to resolve ambiguities.

Recently developed methods for incorporating data from other sources such as proteomics [20], Förster resonance energy transfer [21], or sparse distance restraints [22] into the fitting process have shown some very encouraging potential [23,24], but they also have to be handled with care. An example is a discrepancy in fitting-based interpretation of the ryanodine receptor [25,26]. In the earlier study, a homology model was fitted into the region of a reconstruction that was indicated by antibody labeling. In the other study, fitting with an atomic structure of a fragment was performed followed by an evaluation of ambiguities using confidence intervals. The differences in fitting protocols completely altered the final fit. More recent fits based on high-resolution reconstructions [27,28] confirmed the statistics based fit [26] to be correct indicating that the labeling constraint [25] threw off the fitting rather than improved the fit (Figure 3).

Crossvalidation

One technique that can help overcoming overfitting caused by an inadequate ratio of refinable parameters to observables is crossvalidation [29]. In crossvalidation, the data is split into a “training set” and a “test set”. Calculations are then performed on the training set while evaluation is done using the test set, so that an adequate set of parameters can be determined (Figure 4). While the fit to the training set will continue to improve due to overfitting, the test set fit will at some point worsen, thus giving optimal values for the fitting parameters involved. This scheme corresponds to a 2-fold crossvalidation also known as holdout method. This is the simplest form of k-fold crossvalidation and has the disadvantage that it usually needs to set apart a sizable portion of the data into the test set that is not accessible for use in the fitting and that an “unfortunate” split can lead to severe misestimations of the parameters. k-fold crossvalidation overcomes this problem by repeating the calculations with k different splits that cover the whole data set and thus are less amenable to artifacts caused by a specific split and also allows a larger fraction of the data to participate in the fitting calculations.

In structural biology, the most common form of crossvalidation is a 2-fold crossvalidation scheme implemented in Fourier space for X-ray crystallography in the form of the “free R-factor” [30]. A crucial prerequisite for crossvalidation to work is that the information in the test set is independent from that in the work set. For the Fourier terms in X-ray crystallography this assumption is usually justified. In electron microscopy the Fourier terms are strongly correlated so that the free R-factor it is not applicable in a direct analogy to crystallography. Several factors introduce correlations between Fourier terms in cryo-EM reconstructions. If the particle is of limited size in real space, placed in an empty box, then there are automatically correlations between neighboring Fourier components. Equally, low-pass filtering, a common operation in cryo-EM, has the same effect. The alignment of images during the reconstruction process also tends to introduce correlation in the noise of these images [31].

Crossvalidation in cryo-EM fitting

In the current context of fitting atomic structures into cryo-EM reconstructions, two conceptually different approaches have been proposed [32,33]. The first approach [32] relies on splitting the data according to frequency ranges where low-resolution frequency data with significant signal-to-noise ratio is used as a training set for model building and a shell of high-frequency data is used as a test set. As expected, significant correlations between the training and test sets can be detected. The authors propose a work-around where they compare the signal in the test set with a signal from a “perfectly overfitted” bead model, which is used as a baseline. Results using test cases with resolutions between 5 and 10 Å are promising but whether the method works at lower resolutions remains to be seen, especially because correlations between shells will be a more serious problem for lower resolution (the Fourier components with a spatial frequency corresponding to 8 and 9 Å are closer in Fourier space than the 3 and 4 Å components).

In the second crossvalidation approach [33] the data is split randomly into two independent sets and reconstructions are built independently from each set. One of these reconstructions is used as a training set, the second is used as test set. This type of methodology can be used to select sensible weighting terms between the density and all-atom energy contributions using the program Rosetta [34]. However, the authors find that at resolution below 12 Å significant correlation between the two independent maps occur, outweighing the usefulness of the approach at resolution ranges below 12 Å [33].

Confidence intervals

A confidence interval is a range of values that is believed, with some stated level of confidence, to contain the true value of interest. In general, high levels of confidence can be achieved with wider intervals, while narrower, more precise intervals carry less confidence. Thus, there is a trade off between precision and confidence and, in fact, any statement of precision without a corresponding confidence level is incomplete. The advantage of providing a range of values for the estimate is that it will be more likely to include the correct one. Generally, the width of a confidence interval can be interpreted as a measure of precision while the confidence level of an interval is a measure of accuracy. However, this statement is only true if all error sources are accounted for correctly. If undetected systematic errors are present, this relationship is broken.

The use of confidence intervals was shown to be a powerful tool in rigid-body fitting approaches for interpreting cryo-EM reconstructions [35]. In this approach, a global search is followed by a global statistical analysis of the score distribution resulting in the definition of confidence intervals. All fits that have scores within that confidence interval satisfy the data within the error margin defined by the errors in the data and the chosen confidence level. If all errors are accounted for, this will give excellent estimates for precision and accuracy. To account for systematic errors, it is an advantage to use as many independent data sets under varying conditions as possible. Crossvalidation by splitting data into random halves is also an option [35].

Structural parameters of interest can be evaluated as properties of the sets satisfying the confidence interval criterion. For example, the uncertainty of each atom position of the fitted structure can be approximated by calculating the root-mean-square deviation for each atom using all members of the set. The statistical nature of the approach allows the use of standard statistical tests, such as Student’s t-test, to evaluate the significance of differences between models in different orientations [19] or functional states [23] and to help model the corresponding conformational changes in a robust and reliable way.

Conclusions

Fitting of atomic structures into low-resolution reconstructions from cryo-EM need de facto standards and tools for assessing the quality and estimating the accuracy of the resulting fits. It is clear that rigorous and objective evaluation criteria are still needed to validate conclusions drawn from fitting of high-resolution structures into lower-resolution reconstructions from electron microscopy. Currently, several promising approaches based on cross-validation and statistical tools to obtain confidence intervals are in development but generally accepted standards are lacking. Thus, better validation criteria are likely to continue to be the subject of intense development in the near future.