# Table of Contents - [About CryoSPARC™ | CryoSPARC Guide](#about-cryosparc-cryosparc-guide) - [Licensing | CryoSPARC Guide](#licensing-cryosparc-guide) - [Non-commercial license agreement | CryoSPARC Guide](#non-commercial-license-agreement-cryosparc-guide) - [Obtaining A License ID | CryoSPARC Guide](#obtaining-a-license-id-cryosparc-guide) - [How to Download, Install and Configure | CryoSPARC Guide](#how-to-download-install-and-configure-cryosparc-guide) - [(Optional) Hosting CryoSPARC Through a Reverse Proxy | CryoSPARC Guide](#-optional-hosting-cryosparc-through-a-reverse-proxy-cryosparc-guide) - [CryoSPARC Installation Prerequisites | CryoSPARC Guide](#cryosparc-installation-prerequisites-cryosparc-guide) - [CryoSPARC Cluster Integration Script Examples | CryoSPARC Guide](#cryosparc-cluster-integration-script-examples-cryosparc-guide) - [Accessing the CryoSPARC User Interface | CryoSPARC Guide](#accessing-the-cryosparc-user-interface-cryosparc-guide) - [CryoSPARC Architecture and System Requirements | CryoSPARC Guide](#cryosparc-architecture-and-system-requirements-cryosparc-guide) - [Management and Monitoring (≤v4.7) | CryoSPARC Guide](#management-and-monitoring-v4-7-cryosparc-guide) - [Flat vs Hierarchical Navigation | CryoSPARC Guide](#flat-vs-hierarchical-navigation-cryosparc-guide) - [File Browser | CryoSPARC Guide](#file-browser-cryosparc-guide) - [Keyboard Shortcuts | CryoSPARC Guide](#keyboard-shortcuts-cryosparc-guide) - [CryoSPARC Tools | CryoSPARC Guide](#cryosparc-tools-cryosparc-guide) - [How to Access CryoSPARC Live | CryoSPARC Guide](#how-to-access-cryosparc-live-cryosparc-guide) - [Managing Data | CryoSPARC Guide](#managing-data-cryosparc-guide) - [Instance Management | CryoSPARC Guide](#instance-management-cryosparc-guide) - [CryoSPARC Live Tutorial Videos | CryoSPARC Guide](#cryosparc-live-tutorial-videos-cryosparc-guide) - [A Tour of the CryoSPARC Interface | CryoSPARC Guide](#a-tour-of-the-cryosparc-interface-cryosparc-guide) - [Tags | CryoSPARC Guide](#tags-cryosparc-guide) - [Downloading and Exporting Data | CryoSPARC Guide](#downloading-and-exporting-data-cryosparc-guide) - [View and Download Results | CryoSPARC Guide](#view-and-download-results-cryosparc-guide) - [Admin Panel | CryoSPARC Guide](#admin-panel-cryosparc-guide) - [Dashboard | CryoSPARC Guide](#dashboard-cryosparc-guide) - [Waves as Vectors | CryoSPARC Guide](#waves-as-vectors-cryosparc-guide) - [Blueprints | CryoSPARC Guide](#blueprints-cryosparc-guide) - [v3 User Interface Guide | CryoSPARC Guide](#v3-user-interface-guide-cryosparc-guide) - [Managing a CryoSPARC Live Session from the CLI (v5.0+) | CryoSPARC Guide](#managing-a-cryosparc-live-session-from-the-cli-v5-0-cryosparc-guide) - [Prerequisites and Compute Resources Setup | CryoSPARC Guide](#prerequisites-and-compute-resources-setup-cryosparc-guide) - [Job Relationships | CryoSPARC Guide](#job-relationships-cryosparc-guide) - [Job: Simulate Data (Legacy) | CryoSPARC Guide](#job-simulate-data-legacy-cryosparc-guide) - [Simulations | CryoSPARC Guide](#simulations-cryosparc-guide) - [Queue Job, Inspect Job and Other Job Actions | CryoSPARC Guide](#queue-job-inspect-job-and-other-job-actions-cryosparc-guide) - [UI Overview | CryoSPARC Guide](#ui-overview-cryosparc-guide) - [Live Jobs and Session-Level Functions | CryoSPARC Guide](#live-jobs-and-session-level-functions-cryosparc-guide) - [Create and Build Jobs | CryoSPARC Guide](#create-and-build-jobs-cryosparc-guide) - [Project and Workspace Management | CryoSPARC Guide](#project-and-workspace-management-cryosparc-guide) - [Local Refinement | CryoSPARC Guide](#local-refinement-cryosparc-guide) - [Job: Simulate Data (GPU) | CryoSPARC Guide](#job-simulate-data-gpu-cryosparc-guide) - [Job: Local Resolution Estimation | CryoSPARC Guide](#job-local-resolution-estimation-cryosparc-guide) - [Webinar Recordings | CryoSPARC Guide](#webinar-recordings-cryosparc-guide) - [Job: Local Filtering | CryoSPARC Guide](#job-local-filtering-cryosparc-guide) - [Job: ResLog Analysis | CryoSPARC Guide](#job-reslog-analysis-cryosparc-guide) - [Job: ThreeDFSC (Wrapper) (Legacy) | CryoSPARC Guide](#job-threedfsc-wrapper-legacy-cryosparc-guide) - [Job: Particle Subtraction | CryoSPARC Guide](#job-particle-subtraction-cryosparc-guide) - [FAQs and Troubleshooting | CryoSPARC Guide](#faqs-and-troubleshooting-cryosparc-guide) - [About CryoSPARC Live | CryoSPARC Guide](#about-cryosparc-live-cryosparc-guide) - [Automated Workflows | CryoSPARC Guide](#automated-workflows-cryosparc-guide) - [Helical Reconstruction | CryoSPARC Guide](#helical-reconstruction-cryosparc-guide) - [Job: Average Power Spectra | CryoSPARC Guide](#job-average-power-spectra-cryosparc-guide) - [Job: Generate Micrograph Thumbnails | CryoSPARC Guide](#job-generate-micrograph-thumbnails-cryosparc-guide) - [Job: Exposure Tools | CryoSPARC Guide](#job-exposure-tools-cryosparc-guide) - [Job: Cache Particles on SSD | CryoSPARC Guide](#job-cache-particles-on-ssd-cryosparc-guide) - [Job: Check for Corrupt Particles | CryoSPARC Guide](#job-check-for-corrupt-particles-cryosparc-guide) - [Tutorial: EER File Support | CryoSPARC Guide](#tutorial-eer-file-support-cryosparc-guide) - [Job: Select Volume | CryoSPARC Guide](#job-select-volume-cryosparc-guide) - [Tutorial: Phase Plate Data | CryoSPARC Guide](#tutorial-phase-plate-data-cryosparc-guide) - [Tutorial: Blob Picker Tuner | CryoSPARC Guide](#tutorial-blob-picker-tuner-cryosparc-guide) - [Job: Reassign Particles to Micrographs | CryoSPARC Guide](#job-reassign-particles-to-micrographs-cryosparc-guide) - [Managing a CryoSPARC Live Session from the CLI (≤v4.7) | CryoSPARC Guide](#managing-a-cryosparc-live-session-from-the-cli-v4-7-cryosparc-guide) - [Tutorial: Maximum Box Sizes for Refinement | CryoSPARC Guide](#tutorial-maximum-box-sizes-for-refinement-cryosparc-guide) - [Tutorial: Float16 Support | CryoSPARC Guide](#tutorial-float16-support-cryosparc-guide) - [Tutorial: Particle Picking Calibration | CryoSPARC Guide](#tutorial-particle-picking-calibration-cryosparc-guide) - [Tutorial: BILD files | CryoSPARC Guide](#tutorial-bild-files-cryosparc-guide) - [Job: Split Volumes Group | CryoSPARC Guide](#job-split-volumes-group-cryosparc-guide) - [Case Study: Processing EMPIAR-10291 (300 Micrographs) to 3.4Å in 1 hour 25 minutes | CryoSPARC Guide](#case-study-processing-empiar-10291-300-micrographs-to-3-4-in-1-hour-25-minutes-cryosparc-guide) - [Case Study: Exploratory data processing by Oliver Clarke | CryoSPARC Guide](#case-study-exploratory-data-processing-by-oliver-clarke-cryosparc-guide) - [Tutorial: Patch Motion and Patch CTF | CryoSPARC Guide](#tutorial-patch-motion-and-patch-ctf-cryosparc-guide) - [Job: Volume Alignment Tools | CryoSPARC Guide](#job-volume-alignment-tools-cryosparc-guide) - [Job: Volume Tools | CryoSPARC Guide](#job-volume-tools-cryosparc-guide) - [Installing 3DFlex Dependencies (v4.1–v4.3) | CryoSPARC Guide](#installing-3dflex-dependencies-v4-1-v4-3-cryosparc-guide) - [Tutorial: Dynamic Masking in Refinements (v5.0+) | CryoSPARC Guide](#tutorial-dynamic-masking-in-refinements-v5-0-cryosparc-guide) - [Tutorial: Ewald Sphere Correction | CryoSPARC Guide](#tutorial-ewald-sphere-correction-cryosparc-guide) - [Tutorial: Negative Stain Data | CryoSPARC Guide](#tutorial-negative-stain-data-cryosparc-guide) - [Tutorial: Orientation Diagnostics | CryoSPARC Guide](#tutorial-orientation-diagnostics-cryosparc-guide) - [Tutorial: 3D Variability Analysis (Part Two) | CryoSPARC Guide](#tutorial-3d-variability-analysis-part-two-cryosparc-guide) - [Performance Metrics | CryoSPARC Guide](#performance-metrics-cryosparc-guide) - [Tutorial: Symmetry Relaxation | CryoSPARC Guide](#tutorial-symmetry-relaxation-cryosparc-guide) - [Tutorial: 3D Flexible Refinement | CryoSPARC Guide](#tutorial-3d-flexible-refinement-cryosparc-guide) - [Tutorial: CTF Refinement | CryoSPARC Guide](#tutorial-ctf-refinement-cryosparc-guide) - [Tutorial: EPU AFIS Beam Shift Import | CryoSPARC Guide](#tutorial-epu-afis-beam-shift-import-cryosparc-guide) - [Tutorial: 3D Variability Analysis (Part One) | CryoSPARC Guide](#tutorial-3d-variability-analysis-part-one-cryosparc-guide) - [Tutorial: Helical Processing using EMPIAR-10031 (MAVS) | CryoSPARC Guide](#tutorial-helical-processing-using-empiar-10031-mavs-cryosparc-guide) - [Tutorial: 3D Flex Mesh Preparation | CryoSPARC Guide](#tutorial-3d-flex-mesh-preparation-cryosparc-guide) - [Tutorial: Tips for Membrane Protein Structures | CryoSPARC Guide](#tutorial-tips-for-membrane-protein-structures-cryosparc-guide) - [Tutorial: Mask Creation | CryoSPARC Guide](#tutorial-mask-creation-cryosparc-guide) - [New Live Session: Start to Finish Guide | CryoSPARC Guide](#new-live-session-start-to-finish-guide-cryosparc-guide) - [Tutorial: Common CryoSPARC Plots | CryoSPARC Guide](#tutorial-common-cryosparc-plots-cryosparc-guide) - [Tutorial: 3D Classification | CryoSPARC Guide](#tutorial-3d-classification-cryosparc-guide) - [Case Study: Yeast U4/U6.U5 tri-snRNP | CryoSPARC Guide](#case-study-yeast-u4-u6-u5-tri-snrnp-cryosparc-guide) - [Case Study: End-to-end processing of encapsulated ferritin (EMPIAR-10716) | CryoSPARC Guide](#case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716-cryosparc-guide) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) --- # About CryoSPARC™ | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/readme.md) . [](https://guide.cryosparc.com/#what-is-cryosparc-tm) What is CryoSPARC™? ------------------------------------------------------------------------------ CryoSPARC is a state of the art scientific software platform for cryo-electron microscopy (cryo-EM) used in research and drug discovery pipelines. CryoSPARC is used to reconstruct and visualize cryo-EM structures of biological targets including membrane proteins, viruses and complexes, from raw movies to high resolution maps. Learn more: [https://cryosparc.com/](https://cryosparc.com/) [](https://guide.cryosparc.com/#what-is-cryosparc-live-tm) What is CryoSPARC Live™? ---------------------------------------------------------------------------------------- CryoSPARC Live is an extension of CryoSPARC that delivers real-time cryo-EM data processing, quality assessment and feedback as data is collected. Learn more: [https://cryosparc.com/live](https://cryosparc.com/live) [](https://guide.cryosparc.com/#licensing) Licensing --------------------------------------------------------- CryoSPARC and CryoSPARC Live can be licensed for non-profit academic use and commercial use. Please see: [Licensing](https://guide.cryosparc.com/licensing) [](https://guide.cryosparc.com/#citation) Citation ------------------------------------------------------- If you use CryoSPARC in your work, please cite as follows. ### [](https://guide.cryosparc.com/#cryosparc-algorithms) CryoSPARC algorithms Please cite the following papers as appropriate: * **General CryoSPARC/CryoSPARC Live use, including preprocessing, 2D classification, Ab-initio reconstruction, Refinement:** [Punjani, A., Rubinstein, J.L., Fleet, D.J. & Brubaker, M.A. cryoSPARC: algorithms for rapid unsupervised cryo-EM structure determination. Nature Methods 14, 290-296 (2017).](https://www.nature.com/articles/nmeth.4169) * **Local (per-particle) motion correction:** [Rubinstein, J.L. & Brubaker, M.A. Alignment of cryo-EM movies of individual particles by optimization of image translations. Journal of Structural Biology 192 (2), 188-195 (2015).](https://www.sciencedirect.com/science/article/pii/S1047847715300459) * **Non-uniform refinement:** [Punjani, A., Zhang, H. & Fleet, D.J. Non-uniform refinement: adaptive regularization improves single-particle cryo-EM reconstruction. Nat Methods **17,** 1214–1221 (2020).](https://www.nature.com/articles/s41592-020-00990-8) * **3D Variability Analysis:** [Punjani, A. & Fleet, D.J. 3D variability analysis: Resolving continuous flexibility and discrete heterogeneity from single particle cryo-EM. Journal of Structural Biology, Volume 213, Issue 2, 2021. https://doi.org/10.1016/j.jsb.2021.107702](https://doi.org/10.1016/j.jsb.2021.107702) * **3D Flexible Refinement:** [Punjani, A. & Fleet, D.J. 3DFlex: determining structure and motion of flexible proteins from cryo-EM. Nature Methods (2023). https://doi.org/10.1038/s41592-023-01853-8](https://doi.org/10.1038/s41592-023-01853-8) ### [](https://guide.cryosparc.com/#cryosparc-implementations) CryoSPARC implementations * **ResLog analysis:** [Stagg, S.M., Noble, A.J., Spilman, M. & Chapman, M.S. ResLog plots as an empirical metric of the quality of cryo-EM reconstructions. Journal of Structural Biology 185 (3), 418-426 (2014).](https://www.sciencedirect.com/science/article/pii/S1047847713003377?via%3Dihub) * **CTF refinement and aberration correction:** [Zivanov, J., Nakane, T. & Scheres, S. H. W. Estimation of high-order aberrations and anisotropic magnification from cryo-EM data sets in _RELION_\-3.1. IUCrJ 7, 253-267 (2020).](https://dx.doi.org/10.1107%2FS2052252520000081) * **Reference-based motion correction/Bayesian polishing:** [Zivanov, J., Nakane, T. & Scheres, S. H. W. A Bayesian approach to beam-induced motion correction in cryo-EM single-particle analysis. IUCrJ 6, 5-17 (2019).](https://doi.org/10.1107/S205225251801463X) ### [](https://guide.cryosparc.com/#wrappers-to-third-party-tools) Wrappers to third-party tools Users should obtain their own software licenses (as applicable) for the below programs, for which wrappers are available in CryoSPARC. * **MotionCor2:** Shawn Q. Zheng, Eugene Palovcak, Jean-Paul Armache, Yifan Cheng and David A. Agard (2016) Anisotropic Correction of Beam-induced Motion for Improved Single-particle Electron Cryo-microscopy, Nature Methods, submitted. BioArxiv: [http://biorxiv.org/content/early/2016/07/04/061960](http://biorxiv.org/content/early/2016/07/04/061960) * **CTFFIND:** [Rohou, A. & Grigorieff, N. CTFFIND4: Fast and accurate defocus estimation from electron micrographs. Journal of Structural Biology 192 (2), 216-221 (2015).](https://www.sciencedirect.com/science/article/pii/S1047847715300460?via%3Dihub) * **Gctf:** Gctf: Jack (Kai) Zhang. Zhang, K. (2016). Gctf : Real-time CTF determination and correction. Journal of Structural Biology, 193(1), 1-12. [https://doi.org/10.1016/j.jsb.2015.11.003](https://doi.org/10.1016/j.jsb.2015.11.003) * **Topaz:** Bepler, T., Morin, A., Rapp, M. et al. Positive-unlabeled convolutional neural networks for particle picking in cryo-electron micrographs. Nat Methods 16, 1153–1160 (2019) doi:10.1038/s41592-019-0575-8 and Bepler, T., Noble, A.J., Berger, B. Topaz-Denoise: general deep denoising models for cryoEM. bioRxiv 838920 (2019) doi: [https://doi.org/10.1101/838920](https://doi.org/10.1101/838920) * **3DFSC:** [Tan, Y.Z., Baldwin, P.R., Davis, J.H., Williamson, J.R., Potter, C.S., Carragher, B. & Lyumkis, D. Addressing preferred specimen orientation in single-particle cryo-EM through tilting. Nature Methods 14, 793-796 (2017).](http://dx.doi.org/10.1038/nmeth.4347) * **DeepEMhancer:** R. Sanchez-Garcia, J. Gomez-Blanco, A. Cuervo et al., “DeepEMhancer: a deep learning solution for cryo-EM volume post-processing”, Communications Biology, vol. 4, no. 874, 2021. Available: 10.1038/s42003-021-02399-1. ### [](https://guide.cryosparc.com/#dependencies) Dependencies #### [](https://guide.cryosparc.com/#cudnn) **cuDNN** libcudnn.so.8 is distributed with CryoSPARC as of v3.2, pursuant to the terms of NVIDIA's Software License Agreement (SLA) for cuDNN: [https://docs.nvidia.com/deeplearning/cudnn/sla/index.html](https://docs.nvidia.com/deeplearning/cudnn/sla/index.html) #### [](https://guide.cryosparc.com/#scikit-cuda) **scikit-cuda** A modified version of scikit-cuda is included with cryosparc\_compute as of v3.2, pursuant to the scikit-cuda license terms: [https://scikit-cuda.readthedocs.io/en/latest/](https://scikit-cuda.readthedocs.io/en/latest/) > Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution. Neither the name of Lev E. Givon nor the names of any contributors may be used to endorse or promote products derived from this software without specific prior written permission. > > THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. [](https://guide.cryosparc.com/#cryosparc-in-scientific-studies) CryoSPARC in Scientific Studies ----------------------------------------------------------------------------------------------------- Hundreds of structural studies have used CryoSPARC for cryo-EM data processing: [Google Scholar: cryoSPARC](https://scholar.google.ca/scholar?cites=6690181732944497496&as_sdt=2005&sciodt=0,5&hl=en) [](https://guide.cryosparc.com/#development) Development ------------------------------------------------------------- CryoSPARC was originally a research project with origins at the University of Toronto in 2014. As of 2016, all research and development for CryoSPARC is done by [Structura Biotechnology Inc.](https://structura.bio/) , a scientific software company based in Toronto, Canada. By combining our expertise in image processing, algorithm development and professional software engineering, we aim to keep CryoSPARC at the forefront of software for cryo-EM. To that end, we are constantly working on new algorithms and software features which we release on an ongoing basis. CryoSPARC's GPU-accelerated code is written entirely from scratch in-house, with exception of certain wrappers to third party tools that are clearly indicated in the documentation. Many of the algorithms in CryoSPARC are novel developments for cryo-EM image processing and links to publications can be found throughout this documentation. ### [](https://guide.cryosparc.com/#major-version-history) Major Version History * CryoSPARC v5.0 was released on January 27, 2026. * CryoSPARC v4.0 was released on October 3, 2022 and has been followed by several subsequent releases up to v4.7.1. * CryoSPARC v3.0 was released on December 9, 2020 and has been followed by subsequent version v3.1, v3.2 and v3.3. * CryoSPARC v2.0 (released August 17, 2018) was followed by a number of new releases up to v2.15.0 (released May 13, 2020). * CryoSPARC v0.2.1 was the first public version of CryoSPARC (released February 7, 2017) and was followed by a number of new releases up to v0.6.5 (released January 12, 2018). For release notes, see: [https://cryosparc.com/updates](https://cryosparc.com/updates) © 2026 Structura Biotechnology Inc. All rights reserved. CryoSPARC™ and CryoSPARC Live™ are trademarks of Structura Biotechnology Inc. [NextLicensing](https://guide.cryosparc.com/licensing) Last updated 1 month ago --- # Licensing | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/licensing.md) . [](https://guide.cryosparc.com/licensing#non-profit-use) Non-profit use ---------------------------------------------------------------------------- CryoSPARC™ and CryoSPARC Live™ are available free of charge for non-profit academic research. Non-Profit Academic Research is defined as follows: **Non-Profit Academic Research** means practicing, making, using, improving upon, importing and exporting (but not selling, leasing or otherwise monetizing) academic or scholarly research, for individual (personal) or academic institutional research purposes, in good faith, and expressly excludes, without limitation, purposes that are intended to (or result in, whether by intent or otherwise): (i) create a commercial advantage for any Person; (ii) generate monetary compensation for products or services; (iii) generate commercialization rights for any Person; (iv) be used in an ongoing business concern; or (v) result in an ongoing business concern obtaining any intellectual property rights in any research or results linked to the Non-profit Academic Research. To request a license for non-profit academic research, please fill out the form on our website: [![Logo](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2Fcryosparc.com%2Ffavicon.png&width=20&dpr=3&quality=100&sign=5ec89076&sv=2)CryoSPARC | Download CryoSPARCCryoSPARC](https://cryosparc.com/download) The full text of the non-commercial license agreement is available here for reference: [Non-commercial license agreement](https://guide.cryosparc.com/licensing/non-commercial-license-agreement) [](https://guide.cryosparc.com/licensing#for-profit-commercial-or-industry-use) For-profit, commercial or industry use --------------------------------------------------------------------------------------------------------------------------- Please contact [sales@structura.bio](mailto:sales@structura.bio) for inquiries relating to for-profit or industry licensing, including academic-industry collaborations on proprietary projects and fee-for-service data processing. [](https://guide.cryosparc.com/licensing#questions-about-licensing) Questions about licensing? --------------------------------------------------------------------------------------------------- If you aren't sure which license applies to your use case, or have any questions around licensing, please contact us at [sales@structura.bio](mailto:sales@structura.bio) with your questions. [PreviousAbout CryoSPARC™](https://guide.cryosparc.com/) [NextNon-commercial license agreement](https://guide.cryosparc.com/licensing/non-commercial-license-agreement) Last updated 4 months ago * [Non-profit use](https://guide.cryosparc.com/licensing#non-profit-use) * [For-profit, commercial or industry use](https://guide.cryosparc.com/licensing#for-profit-commercial-or-industry-use) * [Questions about licensing?](https://guide.cryosparc.com/licensing#questions-about-licensing) --- # Non-commercial license agreement | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/licensing/non-commercial-license-agreement.md) . [](https://guide.cryosparc.com/licensing/non-commercial-license-agreement#cryosparc-non-commercial-software-license-agreement) CryoSPARC Non-Commercial Software License Agreement --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- _Last Updated May 27, 2020_ This Non-Commercial Software License Agreement (the "Agreement") is made between you (the "Licensee") and Structura Biotechnology Inc. (the "Licensor"). By installing or otherwise using CryoSPARC (the "Software"), you agree to be bound by the terms and conditions of this Agreement as may be revised from time to time at Licensor's sole discretion. If you do not agree to the terms and conditions of this Agreement, do not install or use the Software. 1. NON-COMMERCIAL USE. Licensor hereby grants to Licensee one (1) non-exclusive, non-transferable license (the "License") to install and use the Software for Non-Profit Academic Research and processing of cryo-electron microscopy ("cryo-EM") data (the "Intended Purpose"). "Software" includes the executable computer programs, code and any related printed, electronic and online documentation, manuals, training aids, user guides, system administration documentation and any other files that may accompany the code. "Non-Profit Academic Research" means practicing, making, using, improving upon, importing and exporting (but not selling, leasing or otherwise monetizing) academic or scholarly research, for individual (personal) or academic institutional research purposes, in good faith, and expressly excludes, without limitation, purposes that are intended to (or result in, whether by intent or otherwise): (i) create a commercial advantage for any Person; (ii) generate monetary compensation for products or services; (iii) generate commercialization rights for any Person; (iv) be used in an ongoing business concern; or (v) result in an ongoing business concern obtaining any intellectual property rights in any research or results linked to the Non-profit Academic Research. 2. RESTRICTIONS. Licensee may not: (i) modify, enhance, reverse-engineer, decompile, disassemble or create derived forms of the Software; (ii) copy the Software; (iii) sell, sub-license, lease, assign, transmit, distribute or otherwise transfer rights in/to the Software; (iv) allow third-party use of Licensee's installation of the Software; or (v) pledge, hypothecate, alienate or otherwise encumber the Software to any third party. Use of the Software is restricted to the Intended Purpose only. 3. NO WARRANTY. The Software is provided "as is" without warranty of any kind. Licensor makes no representations, warranties or covenants to Licensee, either express or implied, with respect to the Software or with respect to any Confidential Information (as defined herein) disclosed to Licensee. Licensor specifically disclaims any implied warranty or condition of non-infringement, merchantable quality or fitness for a particular purpose. Licensee acknowledges that the Software is of an experimental nature, that no particular results can be guaranteed, and that it has been advised by Licensor to undertake its own due diligence with respect to all matters arising from the Agreement. 4. LIMITATION OF LIABILITY. In no event is Licensor liable for any damages on any basis, in contract, tort or otherwise, of any kind and nature whatsoever, arising in respect of this Agreement, howsoever caused, including damages of any kind and nature caused by Licensor’s negligence or by a fundamental breach of contract or any other breach of duty whatsoever. Licensee is advised to safeguard important data, use caution and not rely in any way on the correct functioning or performance of the Software and/or accompanying materials. 5. NO IMPROVEMENTS. Licensor is under no obligation to provide Improvements to the Software. "Improvements" means any improvements, updates, variations, modifications, alterations, additions, error corrections, enhancements, functional changes or other changes to the Software, including, without limitation: (i) improvements or upgrades to improve software efficiency and maintainability; (ii) improvements or upgrades to improve operational integrity and efficiency; (iii) changes or modifications to correct errors; and (iv) additional licensed computer programs to otherwise update the Software. 6. NO FUTURE ENTITLEMENT. Nothing in this Agreement shall be construed as creating any obligation on Licensor to continue to develop, commercialize, offer, make available or support (i) the Software; or (ii) any feature, functionality or Improvement as may be encompassed in the Software from time to time. 7. SUSPENSION. Licensor reserves the right to modify, suspend or discontinue, temporarily or permanently, the Software, with or without notice and without liability to Licensee. 8. OWNERSHIP. Licensor retains title to and ownership of the Software and any Improvements. Nothing in this Agreement shall be construed as granting any express or implied ownership rights to Licensee in respect of the Software, associated documentation or Confidential Information, including but not limited to any patent, copyright, trademark or other intellectual property right. 9. INTELLECTUAL PROPERTY. All Intellectual Property, Intellectual Property Rights and distribution rights associated with or arising from the Software or Licensor’s Confidential Information remain exclusively with Licensor. “Intellectual Property” includes, without limitation, all technical data, designs, specifications, software, data, drawings, plans, reports, patterns, models, prototypes, demonstration units, practices, inventions, methods and related technology, processes or other information, and all rights therein, including, without limitation, patents, copyrights, industrial designs, trade-marks and any registrations or applications for the same and all other rights of intellectual property therein, including any rights for which arise from the above items being treated by the Parties as trade secrets or confidential information (the rights being “Intellectual Property Rights”). 10. CONFIDENTIAL INFORMATION. “Confidential Information” means any and all confidential or proprietary information of Licensor or Licensee which may be exchanged between the Parties at any time prior to and during the term of this Agreement, including, without limitation, business and marketing information, technology, know-how, ideas, reports, techniques, methods, processes, uses, composites, skills, and configurations of any kind. Without limiting the generality of the foregoing, Licensor’s Confidential Information includes: (i) the Software, including its features, functionality, performance, application and use; (ii) the computer code underlying the Software, including source and compiled code and all associated documentation and files; (iii) information relating to the performance or quality of the Software; (iv) the details of any technical assistance provided to Licensee during the term of this Agreement; (v) any other products or service made available to Licensee by Licensor during the term of this Agreement; and (vi) information regarding Licensor’s business operations or research and development activities. Neither party shall: (i) disclose, either directly or indirectly, any Confidential Information or any part thereof belonging to the other party, to any person except as is specifically contemplated in this Agreement; or (ii) use any Confidential Information or any part thereof belonging to the other party, for any purpose except as is specifically contemplated in this Agreement. The obligations of confidentiality set forth herein shall not apply to the extent that the information: (i) was already known to the relevant party without restriction at the time the information was disclosed to such party; (ii) was generally available to the public or otherwise was part of the public domain at the time of its disclosure to the relevant party; (iii) became generally available to the public or otherwise part of the public domain after its disclosure to the relevant party through no act or omission of such party; or (iv) was disclosed to the relevant party without restriction by a third party who, to the best of such party's knowledge and belief, had no obligation not to disclose such information. 11. FEEDBACK. Licensee may communicate to Licensor, whether or not at Licensor’s request, suggestions and comments regarding the Software, including without limitation, performance, user interface, experiment results, and errors (collectively, “Feedback”). Licensor shall have worldwide, non-exclusive, perpetual, irrevocable, royalty-free, fully-paid up rights to use such Feedback. Without limiting the generality of the foregoing, Licensor shall have the unencumbered right to make, use, copy, modify, sell, distribute, sub-license, and create derivative works of/incorporating the Feedback as part of any product, technology, service, specification or other documentation and to publicly perform or display, import, broadcast, transmit, distribute, license, offer to sell, and sell, rent, lease or lend anonymized copies of the Feedback (and derivative works thereof) as part of any product. 12. PERFORMANCE DATA AND ANALYTICS. Licensor may collect usage and performance data relating to Licensee’s installation of the Software. including, without limitation, data relating to: (i) software use, including the number of users, projects and experiments associated with an installation; (ii) error information, including error messages and user-submitted feedback; (iii) performance data, including experiment run times and failed experiments; (iv) hardware utilization, including the number of active nodes and memory usage; and (v) license status information, including confirmation of valid license status. 13. TERMINATION. Licensor reserves the right to terminate this Agreement immediately and without notice in the event Licensee fails to comply with any provision of this Agreement. On termination of this Agreement, whether by reason of expiry or otherwise, Licensee shall promptly discontinue use of the Software, destroy its installation of the Software and, at Licensor's request, return the Software to Licensor at no cost to Licensor. Licensor may exercise any or or more of the remedies available to it under the terms of this Agreement, in addition to any remedy available at law. Failure of Licensor to enforce a right under this Agreement shall not act as a waiver of that right. 14. GOVERNING LAW. This Agreement is made in Ontario and governed by and construed in accordance with the laws of the Province of Ontario and the federal laws of Canada applicable therein. The Parties attorn to the exclusive jurisdiction of the Courts of the Province of Ontario. 15. SURVIVAL. The provisions of subsections 2-12 shall survive termination of this Agreement. 16. SEVERABILITY. If any term, covenant, condition or provision of this Agreement is held by a court of competent jurisdiction to be invalid, void or unenforceable, it is the Parties' intent that such provision be reduced in scope by the court only to the extent deemed necessary by that court to render the provision reasonable and enforceable and the remainder of the provisions of this Agreement will in no way be affected, impaired or invalidated as a result. 17. NO AGENCY. No provision of this Agreement or action by the Parties will establish or be deemed to establish any partnership, joint venture, principal-agent or employer-employee relationship in any way, or for any purpose, between Licensor and Licensee. 18. ENTIRE AGREEMENT. This Agreement including all schedules hereto, constitutes the entire agreement between the Parties concerning the subject matter hereof and supersedes all prior or collateral agreements, communications, representations, understandings, negotiations and discussions, oral or written. © 2020 Structura Biotechnology Inc. [PreviousLicensing](https://guide.cryosparc.com/licensing) [NextCryoSPARC Architecture and System Requirements](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements) Last updated 4 months ago --- # Obtaining A License ID | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/obtaining-a-license-id.md) . To obtain a License ID for CryoSPARC, go to [cryosparc.com/download](http://cryosparc.com/download) , fill out the form and submit it. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252Fsync%252F7558e24f45dd59789c4c2f917fa053f86759ac48.png%3Fgeneration%3D1589377614584595%26alt%3Dmedia&width=768&dpr=3&quality=100&sign=4de95bb2&sv=2) License request form ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252Fsync%252F136354b8b9406b9804bb08a944826e64135e66b8.png%3Fgeneration%3D1589377614009401%26alt%3Dmedia&width=768&dpr=3&quality=100&sign=3ade4f9f&sv=2) Confirmation message once form is submitted Once you fill out the form, you will receive an email confirming your request. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252Fsync%252Ff62d8f653789b31f79434ff1f1560117fd9ab9f2.png%3Fgeneration%3D1589377613858009%26alt%3Dmedia&width=768&dpr=3&quality=100&sign=b218217d&sv=2) Email notifying that your request was successful We endeavour to respond to all requests within 24 business hours. Once verified, you will receive a second email containing your License ID. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252Fsync%252F2dda859ad4b02f22b36cba4856dc0d5b35ecbf26.png%3Fgeneration%3D1589377614018636%26alt%3Dmedia&width=768&dpr=3&quality=100&sign=731f6606&sv=2) Email containing your cryoSPARC License ID and next steps [PreviousHow to Download, Install and Configure](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure) [NextDownloading and Installing CryoSPARC](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/downloading-and-installing-cryosparc) Last updated 4 months ago --- # How to Download, Install and Configure | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure.md) . [](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure#step-1-confirm-prerequisites) Step 1: Confirm Prerequisites ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Review [CryoSPARC Architecture and System Requirements](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements) and complete all applicable [CryoSPARC Installation Prerequisites](https://guide.cryosparc.com/setup-configuration-and-management/cryosparc-installation-prerequisites) [](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure#step-2-obtain-a-cryosparc-license-id) Step 2: Obtain a CryoSPARC License ID ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- CryoSPARC is available free of charge for [non-profit academic use](https://guide.cryosparc.com/licensing) . To download the software, you will need a CryoSPARC License ID, which can be requested via the form at [https://cryosparc.com/download/](https://cryosparc.com/download/) . Please contact [sales@structura.bio](mailto:sales@structura.bio) for inquiries relating to for-profit or industry licensing, including academic-industry collaborations on proprietary projects and fee-for-service data processing. [Obtaining A License ID](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/obtaining-a-license-id) [](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure#step-3-download-and-install-cryosparc) Step 3: Download and Install CryoSPARC ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Please see the detailed installation steps applicable to your system setup. [Downloading and Installing CryoSPARC](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/downloading-and-installing-cryosparc) [PreviousCryoSPARC Installation Prerequisites](https://guide.cryosparc.com/setup-configuration-and-management/cryosparc-installation-prerequisites) [NextObtaining A License ID](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/obtaining-a-license-id) Last updated 5 months ago * [Step 1: Confirm Prerequisites](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure#step-1-confirm-prerequisites) * [Step 2: Obtain a CryoSPARC License ID](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure#step-2-obtain-a-cryosparc-license-id) * [Step 3: Download and Install CryoSPARC](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure#step-3-download-and-install-cryosparc) --- # (Optional) Hosting CryoSPARC Through a Reverse Proxy | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/optional-hosting-cryosparc-through-a-reverse-proxy.md) . As discussed in [Accessing the CryoSPARC User Interface](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc) , there are various ways in which users can access the CryoSPARC web interface such as through a VPN connection or SSH tunnel. If you would like to host the CryoSPARC interface in a secure manner at a predictable URL, this can be done through a reverse proxy server. Reverse proxy servers allow for more control over how a user accesses a web application interface over other methods. By controlling incoming network traffic it is able to host the application at a static URL (for example `https://cryosparc.institution.edu`) and ensure all correspondence is secured via HTTPS. The method in which you host CryoSPARC through a reverse proxy is similar to hosting any other web application. However, the following serve as our recommended minimum requirements: * All incoming traffic should be served through HTTPS (via a SSL certificate) * HTTPS traffic requires a valid SSL certificate provided by a certificate authority (CA) for the domain in which you are hosting the interface. * If the server listens for incoming HTTP traffic, forward all connections to a more secure protocol (HTTPS) * Ensure traffic is also mediated by an organization-level authentication barrier (for example single sign-on). CryoSPARC should _not_ be served via the public internet without any additional authentication checks. There are many ways to generate a SSL certificate for your domain, however, this will most likely be specific to your institution or organization. If you're unsure of how to generate a SSL certificate for your private network, please consult with your system or network administrator for guidance. Each institution or private network can have a specific setup requiring custom rules and/or proxy configuration considerations. Generally the example configurations below should be compatible with common reverse proxy installations. Please consult with your system or network administrator for guidance regarding institution-specific protocols for reverse proxy hosting. The following section will provide example configuration files for common reverse proxy servers given CryoSPARC is running on base port `61000`. [](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/optional-hosting-cryosparc-through-a-reverse-proxy#nginx) NGINX ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This [NGINX](https://nginx.org/en/) configuration takes advantage of [authenticated origin pulls](https://developers.cloudflare.com/ssl/origin-configuration/authenticated-origin-pull/explanation/) for an added layer of security between the reverse-proxy and a downstream proxy/load balancer. Copy server { listen 80; listen [::]:80; server_name private.domain.dev; return 302 https://$server_name$request_uri; } server { # SSL configuration listen 443 ssl http2; listen [::]:443 ssl http2; ssl on; ssl_certificate /etc/certs/domain.dev/origin-cert.pem; ssl_certificate_key /etc/certs/domain.dev/private-key.pem; ssl_client_certificate /etc/certs/domain.dev/origin-pull-ca.pem; ssl_verify_client on; server_name private.domain.dev; access_log /var/log/nginx/private.domain.dev.access.log; error_log /var/log/nginx/private.domain.dev.error.log; location / { proxy_pass http://127.0.0.1:61000; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header X-Forwarded-For $remote_addr; proxy_request_buffering off; proxy_buffering off; client_max_body_size 0; } } An alternative configuration from our [Discussion Forum](https://discuss.cryosparc.com/t/how-to-run-cryosparc-on-an-https-url-using-an-ssl-certificate/2614/7) that serves the application over HTTPS and redirects incoming HTTP requests. [](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/optional-hosting-cryosparc-through-a-reverse-proxy#apache) Apache ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ The following is a simplified [Apache HTTP Server](https://httpd.apache.org/) configuration that illustrates the `RewriteRule`. For production use, we recommend HTTPS instead of HTTP. Additional configuration, not shown here, is required to enable HTTPS. [PreviousAccessing the CryoSPARC User Interface](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc) [NextSoftware Updates and Patches](https://guide.cryosparc.com/setup-configuration-and-management/software-updates) Last updated 1 month ago * [NGINX](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/optional-hosting-cryosparc-through-a-reverse-proxy#nginx) * [Apache](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/optional-hosting-cryosparc-through-a-reverse-proxy#apache) Copy server { listen 80; server_name ; access_log /var/log/nginx/.http.access.log; error_log /var/log/nginx/.http.error.log; location / { return 301 https://$server_name$request_uri; } } server { listen 443 ssl; server_name ; access_log /var/log/nginx/.access.log; error_log /var/log/nginx/.error.log; location / { proxy_pass http://127.0.0.1:61000; proxy_http_version 1.1; proxy_set_header Upgrade $http_upgrade; proxy_set_header Connection 'upgrade'; proxy_set_header X-Forwarded-For $remote_addr; proxy_request_buffering off; proxy_buffering off; client_max_body_size 0; } ssl_certificate /etc/letsencrypt/live//fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live//privkey.pem; # managed by Certbot } Copy ProxyRequests Off RewriteEngine on ProxyPass / http://localhost:61000/ ProxyPassReverse / http://localhost:61000/ RewriteCond %{HTTP:UPGRADE} ^WebSocket$ [NC] RewriteCond %{HTTP:CONNECTION} ^Upgrade$ [NC] RewriteRule .* ws://localhost:61000%{REQUEST_URI} [P] --- # CryoSPARC Installation Prerequisites | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/setup-configuration-and-management/cryosparc-installation-prerequisites.md) . [](https://guide.cryosparc.com/setup-configuration-and-management/cryosparc-installation-prerequisites#id-1.-nvidia-driver) 1\. Nvidia Driver -------------------------------------------------------------------------------------------------------------------------------------------------- CryoSPARC worker installations on workstations, dedicated GPU nodes or clusters require a recent version of the Nvidia driver and a Nvidia GPU. The list below specified the required Nvidia Driver version for a range of CryoSPARC versions. * **CryoSPARC v5.0+** * Requires **Nvidia Driver version 570.26 or newer**. Note that NVIDIA Blackwell devices are only compatible with the open driver. * A system CUDA installation is not needed. CryoSPARC includes CUDA 12.8 which drops support for NVIDIA GPUs with compute capability 3.5 (Kepler). Only GPUs with compute capability 5.0 (Maxwell) to 12.0 (Blackwell) are supported. * **CryoSPARC v4.4 to CryoSPARC v4.7** * Requires **Nvidia Driver version 520.61.05 or newer.** * A system CUDA installation is not needed. CryoSPARC includes CUDA 11.8.0. * **CryoSPARC ` portion in the `http://localhost:` URL to which you will point your browser. The second port number in both examples is prescribed by the web application port number configured during CryoSPARC installation. Based on the two ssh examples above, you would point your browser to `http://localhost:62222` ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FKBswnEzfLtWOcOJa8Uew%252Fv4-0-0-installation-changes-accessing-ui-login-screen-0.png%3Falt%3Dmedia%26token%3D4904c1b7-6b09-4ee9-a117-edf2ce847b1d&width=768&dpr=3&quality=100&sign=d0253153&sv=2) CryoSPARC UI login page [](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc#more-complex-ssh-requirements-and-configurations) More complex SSH requirements and configurations ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- If you need to hop over one or more "jump" hosts to access the CryoSPARC server or alternative "tunnel" server, please refer the [OpenSSH wikibook](https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Proxies_and_Jump_Hosts#Passing_Through_One_or_More_Gateways_Using_ProxyJump) for suggested `~/.ssh/config` configurations. [](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc#reverse-proxy) Reverse Proxy ------------------------------------------------------------------------------------------------------------------------------------------------------------- Refer to the following guide for more information on hosting the CryoSPARC web application via a reverse proxy server: [(Optional) Hosting CryoSPARC Through a Reverse Proxy](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/optional-hosting-cryosparc-through-a-reverse-proxy) [](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc#appendix) Appendix --------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc#appendix-a-setting-up-password-less-ssh-access-to-a-remote-workstation) Appendix A: Setting up password-less SSH access to a remote workstation Set up SSH keys for password-less access (only if you currently need to enter your password each time you ssh into the compute node). 1. If you do not already have SSH keys generated on your local machine, use `ssh-keygen` to do so. Open a terminal prompt on your local machine, and enter: _Note: this will create an RSA key-pair with no passphrase._ 2. Copy the RSA public key to the remote compute node for password-less login: _Note:_ `_remote_username_` _and_ `_remote_hostname_` _are your username and the hostname that you use to SSH into your compute node. This step will ask for your password._ ### [](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc#appendix-b-using-ssh-forwarding-with-compression-to-reduce-data-usage) Appendix B: Using SSH Forwarding with compression to reduce data usage Supply `-C` to the port tunnelling command to request compression of all data. This can help when downloading maps from the CryoSPARC UI, as masks can be greatly compressed. From `man ssh`: For example: ### [](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc#appendix-c-using-hardware-accelerated-openssh-ciphers) Appendix C: Using Hardware Accelerated OpenSSH Ciphers If your system supports Intel or AMD AES-NI, you can take advantage of hardware accelerated ciphers that dramatically improve the performance of your SSH connection. To find out if your system supports this, [follow this tutorial.](https://www.cyberciti.biz/faq/how-to-find-out-aes-ni-advanced-encryption-enabled-on-linux-system/) If your system has these features enabled, supply the argument `-o Ciphers=aes128-gcm@openssh.com` or `-o Ciphers=aes256-gcm@openssh.com` (depending on what your system supports, but AES 256 is preferred) to the port forwarding command. For example: ### [](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc#appendix-d-custom-ssl-certificate-authority-bundle) Appendix D: Custom SSL Certificate Authority Bundle cryoSPARC requires internet access from the main process to verify your license and perform updates. At minimum, CryoSPARC should have access to our license server at `https://get.cryosparc.com/`. On some older systems, or if your system is behind a HTTP proxy, CryoSPARC may have trouble getting the required SSL certificates to validate this requires. If you have a Certificate Authority (CA) bundle on your system, you may specify its path for CryoSPARC to use and apply. Add the following line to `cryosparc_master/config.sh` (substitute `/path/to/cabundle` with the path to the CA bundle on your system): [PreviousCryoSPARC Cluster Integration Script Examples](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/cryosparc-cluster-integration-script-examples) [Next(Optional) Hosting CryoSPARC Through a Reverse Proxy](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/optional-hosting-cryosparc-through-a-reverse-proxy) Last updated 1 year ago * [VPN Access](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc#vpn-access) * [SSH Access and Tunneling](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc#ssh-access-and-tunneling) * [SSH Local Port Forwarding](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc#ssh-local-port-forwarding) * [More complex SSH requirements and configurations](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc#more-complex-ssh-requirements-and-configurations) * [Reverse Proxy](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc#reverse-proxy) * [Appendix](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc#appendix) * [Appendix A: Setting up password-less SSH access to a remote workstation](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc#appendix-a-setting-up-password-less-ssh-access-to-a-remote-workstation) * [Appendix B: Using SSH Forwarding with compression to reduce data usage](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc#appendix-b-using-ssh-forwarding-with-compression-to-reduce-data-usage) * [Appendix C: Using Hardware Accelerated OpenSSH Ciphers](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc#appendix-c-using-hardware-accelerated-openssh-ciphers) * [Appendix D: Custom SSL Certificate Authority Bundle](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc#appendix-d-custom-ssl-certificate-authority-bundle) Copy ssh -L 62222:localhost:61000 myname@csserver.lab Copy ssh -L 62222:csserver.lab:61000 myname@sshserver.lab Copy ssh-keygen -t rsa -N "" -f $HOME/.ssh/id_rsa Copy ssh-copy-id remote_username@remote_hostname Copy -C Requests compression of all data (including stdin, stdout, stderr, and data for forwarded X11, TCP and UNIX-domain connections). The compression algorithm is the same used by gzip(1), and the “level” can be controlled by the CompressionLevel option for protocol version 1. Compression is desirable on modem lines and other slow connections, but will only slow down things on fast networks. Copy ssh -N -f -L localhost:62222:localhost:61000 remote_hostname -C Copy ssh -N -f -L localhost:62222:localhost:61000 remote_hostname -C -o Ciphers=aes256-gcm@openssh.com Copy export REQUESTS_CA_BUNDLE="/path/to/cabundle" --- # CryoSPARC Architecture and System Requirements | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements.md) . [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#cryosparc-system-architecture-overview) CryoSPARC System Architecture Overview -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- CryoSPARC is a backend and frontend high-performance computing software system that provides data processing and image analysis capabilities for single particle cryo-EM, along with a rich browser-based user interface and command line tools. CryoSPARC can be deployed on-premises or in the cloud. CryoSPARC is designed to be run only within a trusted private network. CryoSPARC instances are not security-hardened against malicious actors on the network and should never be hosted directly on the internet or an untrusted network without a separate controlled authentication layer. ### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#master-worker-pattern) Master-worker pattern The system is based on a master-worker pattern. * The master processes (web application, core application and MongoDB database) run together on one machine (**master** node). The master node requires relatively lightweight resources (4+ CPUs, 16GB+ RAM, 250GB+ HDD storage) * Worker processes run on any available/configured machine that has NVIDIA GPUs (**worker** node). The worker is responsible for all actual computation and data handling and is dispatched by the master node. The same node can function as both master and worker. The master-worker architecture allows CryoSPARC to be installed and scaled up flexibly on a variety of hardware, including a single workstation, groups of workstations, cluster nodes, HPC clusters, cloud nodes, and more. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252Fsync%252Fbfb7df18a6b4e84f7621b6f455ce716f416908a1.png%3Fgeneration%3D1589377613075053%26alt%3Dmedia&width=768&dpr=3&quality=100&sign=2d096b6d&sv=2) Core components included in the CryoSPARC system [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#typical-cryosparc-system-setups) Typical CryoSPARC System Setups ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ CryoSPARC can support a heterogeneous mixture of all typical setups in a single instance. This means you can start with installing CryoSPARC on a single workstation, then connect a worker node or cluster as your data processing requirements scale. ### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#single-workstation) Single Workstation ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252Fsync%252F210539c505176428c649972eed84be5de704dab3.png%3Fgeneration%3D1589377608042147%26alt%3Dmedia&width=768&dpr=3&quality=100&sign=24619fde&sv=2) Single CryoSPARC workstation example, where the master and worker processes run on a single machine. Both the CryoSPARC **master** and CryoSPARC **worker** processes may run on the same machine. The only requirement is that GPU resources are available for the CryoSPARC worker processes. This is the simplest setup. ### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#master-worker) Master-Worker ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FSIQAomjnCoH7xqLK3WfW%252Fv5.0.4-master-worker-v20260422.png%3Falt%3Dmedia%26token%3D7c22119f-0c39-4004-8a9e-55a4c4ac15ae&width=768&dpr=3&quality=100&sign=11d76be0&sv=2) Single-master, multiple-worker example. All nodes must have access to a shared file system. Nodes may be assigned to a dedicated lane (such as lanes B and C in this example) or combined into a common lane (lane A). In the master-worker setup, the worker processes run on one or more GPU servers ("nodes"). The master processes may be run on a light-weight dedicated server or on a GPU server that is configured similarly to a single workstation (see [above](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#single-workstation) ). This is the most flexible setup for installing CryoSPARC. There are three main requirements for this setup, which are also explained in greater detail in the [installation sections of this document](https://guide.cryosparc.com/setup-configuration-and-management/cryosparc-installation-prerequisites) : **1) All nodes have access to a shared file system.** This file system is where the project directories are located, allowing all nodes to read and write intermediate results as jobs start and complete. **2) The master node** has password-less SSH access to each of the worker nodes. SSH is used to execute jobs on the worker nodes from the master node. **3) All worker nodes** have TCP access to 10 consecutive ports on the master node. These ports are used for metadata communication via HTTP API requests. If master processes are run on a GPU server, heavy GPU processing loads can lead to instability if the GPU worker node hangs or runs out of RAM, causing the master processes running the web application and database to also hang. In this _master-worker_ pattern * processing jobs are queued to a specific scheduler _lane_ * a scheduler _lane_ is a collection of one or more (GPU) worker nodes * each worker node is associated with no more than one lane One may combine multiple worker nodes into a lane if there is no concern over which specific node will process a given CryoSPARC job. One may prefer single-node lanes if more direct control is desired over where a given job will run. This may be the case when hardware specifications, such as the amount of RAM or GPU models, vary significantly between nodes. A potential drawback is that a job queued to such a lane would remain queued to the selected lane even if resources are idle on another lane. ### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#clusters) Clusters ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252Fsync%252F14bfec6461b63760244d5d2d99989f43ca705bc5.png%3Fgeneration%3D1589377611998970%26alt%3Dmedia&width=768&dpr=3&quality=100&sign=fdd01825&sv=2) CryoSPARC cluster integration example where both nodes have access to a shared file system The master node can also spawn or submit jobs to a cluster scheduler system (e.g., [Slurm Workload Manager](https://slurm.schedmd.com/overview.html) ). This integration is transparent, and works similar to the master-worker setup explained above, except all resource scheduling is handled by the cluster scheduler, and CryoSPARC's scheduler is only used for orchestration and management of jobs. Similar requirements are present: **1) All nodes** have access to a shared file system. This file system is where the project directories are located, allowing all nodes to read and write results as jobs start and complete. **2) All worker nodes** have TCP access to 10 consecutive ports on the master node (default ports are 39000-39009). These ports are used for metadata communication via HTTP Remote Procedure Call (RPC) based API requests. For a **cluster** setup, the master node can be a regular cluster node (or even a login node) if this makes networking requirements easier, but the CryoSPARC master processes must be run continuously. If the master is to be run on a regular cluster node, the node may need to be requested from your scheduler in interactive mode or for an indefinitely running job. Project directories are created in locations specified by CryoSPARC users. If administering a multi-user cluster instance, ensure that users create project directories in locations where both the master and worker nodes have access. #### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#supported-cluster-schedulers) Supported cluster schedulers CryoSPARC supports most cluster schedulers, including SLURM, SGE and PBS. Please see [here for more details about how CryoSPARC connects with a cluster system](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/downloading-and-installing-cryosparc#connect-a-cluster-to-cryosparc) . [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#cryosparc-system-requirements) CryoSPARC System Requirements -------------------------------------------------------------------------------------------------------------------------------------------------------------------- The following are requirements for every master and worker node in the system unless otherwise specified. Component Requirement Architecture x86-64 (Intel or AMD) Operating System Modern Linux OS. Please see the section on [Operating System](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#operating-system) for more details and limitations. Shell [Bash](https://www.gnu.org/software/bash/) User Account `cryosparcuser` Software Nvidia driver (worker nodes only). [See details on Nvidia and CUDA requirements](https://guide.cryosparc.com/setup-configuration-and-management/cryosparc-installation-prerequisites#1.-nvidia-driver) . Filesystem Shared file system across all nodes ### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#cryosparc-master-node-requirements) CryoSPARC Master Node Requirements The following are requirements specific to the master node. Component Minimum Requirement Recommended **CPU** 4+ cores 8+ cores at 2.8GHz+ **RAM** 16GB+ 32GB DDR4 **System Storage** 250GB+ HDD 500GB SSD **Fast Local Storage** Not Required Not Required **GPU** Not Required Not Required **Network** 1Gbps link to storage servers 10Gbps link to storage servers A 10Gbps connection is recommended to the storage servers given raw cryo-EM movies can be several TB in size, and I/O bottlenecks are more of a concern than processing power for pre-processing jobs in CryoSPARC. Although a CPU with a higher core count is recommended, a CPU with a faster clock rate is more advantageous due to how master processes are implemented. Enough System Storage is required to host the `cryosparc_master` installation package and database folder. Each CryoSPARC project occupies between 100MB and 5GB of database storage, depending on the size of the project. 500GB is enough for approximately 200 medium-sized projects. _Note that this excludes the space required for CryoSPARC project data in bulk storage, which could be in terabytes for larger projects._ ### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#worker-node-cluster-worker-minimum-requirements) Worker Node/Cluster Worker Minimum Requirements The following are requirements for each worker node/cluster worker. Component Minimum Requirement Recommended **CPU** 2+ cores **per GPU** 4 cores **per GPU** **CPU Memory Bandwidth** 50+ GB/s 100+ GB/s **RAM** 32GB+ **per GPU** 64GB DDR4 **per GPU** **System Storage** 25GB+ HDD 50GB+ SSD **Fast Local Storage** 1TB SSD 2TB PCIe SSD **GPU** 1+ NVIDIA GPU with [CC 3.5+](https://developer.nvidia.com/cuda-gpus#compute) , 11GB+ VRAM 1+ NVIDIA Tesla V100, RTX2080Ti, RTX3090, etc **Network** 1Gbps link to storage servers 10Gbps link to storage servers High CPU memory bandwidth is especially important for [CryoSPARC Live preprocessing](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup#preprocessing-lane) . System RAM is very important for worker nodes and should scale proportionately to the number of GPUs available for processing on the system. Enough System Storage is required to host the `cryosparc_worker` installation package. Fast local storage is also necessary as reconstruction jobs require random access to particle images. SSDs provide high throughput in this context. See the section on [Solid State Storage](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#solid-state-storage-ssds) for more details. ### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#operating-system) Operating System For CryoSPARC v5.0+, the operating system must support GLIBC 2.28 or greater. Therefore, the oldest compatible operating systems are Rocky/RHEL 8 and Ubuntu 20.04. For Ubuntu, version 22.04 or 24.04 is recommended. We recommend running CryoSPARC on an up-to-date long-term support Linux distribution, such as Ubuntu (22.04, 24.04), Rocky Linux (8, 9, 10) or a related distribution. As of June 2026, CryoSPARC versions up to 5.0.6 are incompatible with the version 7 kernel that is the default for Ubuntu 26.04. ### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#disks-and-compression) Disks and compression Fast disks are a necessity for processing cryo-EM data efficiently. Fast **sequential** read/write throughput is needed during **pre-processing** stages (e.g., motion correction) where the volume of data is very large (tens of TB) while the amount of computation is relatively low (sequential processing for motion correction, CTF estimation, particle picking, etc.) Spinning disk arrays in a RAID configuration are used to store large raw data files, and often cluster file systems are used for larger systems. As a rule of thumb, to saturate a 4-GPU machine during pre-processing, a **sustained sequential read of 1000MB/s is required**. Compression can greatly reduce the amount of data stored in movie files, and also greatly speeds up preprocessing because decompression is actually faster than reading uncompressed data straight from disk. Typically, counting-mode movie files are stored in LZW compressed TIFF format without gain correction, so that the gain reference file is stored separately and must be applied on-the-fly during processing (which is supported by CryoSPARC). Compressing gain corrected movies can often result in much **worse** compression ratios than compressing pre-gain corrected (integer count) data. CryoSPARC supports LZW compressed TIFF format, EER format and BZ2 compressed MRC format natively. In either case, the gain reference must be supplied as an MRC file. TIFF, EER and BZ2 compression are implemented as multi-core decompression streams on-the-fly. ### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#solid-state-storage-ssds) Solid State Storage (SSDs) SSD space is optional on a per-worker node basis but is **highly recommended** for worker nodes that will be running refinements and reconstructions using particle images. Nodes reserved for pre-processing (motion correction, particle picking, CTF estimation, etc) **do not** need to have an SSD. CryoSPARC particle processing algorithms rely on random-access patterns and multiple passes through the data, rather than sequentially reading the data at once. Using a storage medium that allows for fast random reads will speed up processing dramatically. CryoSPARC manages the SSD cache on each worker node transparently. Files are automatically cached, re-used across the same project and deleted if more space is needed. [Please see the SSD Caching guide for more information.](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/tutorial-ssd-particle-caching-in-cryosparc) The size of your typical single particle cryo-EM datasets will inform the size of SSD you choose to use. For a sample calculation, see: [![Logo](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Ficon%252FCeHcWPT0EsCRkHxr4GZX%252Fstructura-profile-purple.png%3Falt%3Dmedia%26token%3D39f4a203-8992-4197-acd4-2191eec1b4fa&width=48&height=48&sign=66b523e0&sv=2)Guide: SSD Particle Caching in CryoSPARC | CryoSPARC Guideguide.cryosparc.com](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/tutorial-ssd-particle-caching-in-cryosparc#hardware) ### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#graphical-processing-units-gpus) Graphical Processing Units (GPUs) At least one worker node must have GPUs available to run the complete set of CryoSPARC jobs. Non-GPU workers may run CPU-only jobs. The GPU memory (VRAM) in each GPU limits the maximum particle box size for reconstruction. Typically, a GPU with 12GB VRAM can handle a box size of up to 700^3, and up to 1024^3 in some job types. Please ensure each connected worker includes a recent version of the Nvidia Driver compatible with your GPU. See [this section](https://guide.cryosparc.com/setup-configuration-and-management/cryosparc-installation-prerequisites#1.-nvidia-driver) for details. [Download the latest driver for your GPUs here](https://www.nvidia.com/Download/index.aspx) . Visit [Troubleshooting](https://guide.cryosparc.com/setup-configuration-and-management/troubleshooting#gpu-issues) to resolve common GPU errors. #### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#selecting-gpus) Selecting GPUs When acquiring GPUs to use with CryoSPARC, the following considerations may be useful. * CryoSPARC almost exclusively uses single-precision operations on the GPU. As such, consumer cards generally have a much better price/performance ratio than enterprise cards. Enterprise cards do have their own benefits such as reliability, better cooling for servers, longer support timelines, and compatibility with other applications that may use double-precision math. * The most important metric of a GPU is the device VRAM memory bandwidth. This is the rate at which the GPU can read and write from its own memory. This is generally more important than GPU core count or clock speed, as almost all operations on the GPU are memory-bandwidth limited. When selecting GPUs, this is the primary metric to compare (along with price). For example, the NVIDIA RTX 3090 has 24GB of memory at 936 GB/s bandwidth, the NVIDIA A100 has up to 80GB memory at 1935 GB/s bandwidth, and the NVIDIA A4000 has 16GB at 448 GB/s. * GPU memory size is the main limiting factor in terms of the box-sizes that can be handled during a 3D refinement. Other than this, memory size does not have any impact on speed. 11GB consumer cards can generally handle all processing steps (including motion correction of K3 data, etc) for particle box sizes up to 600^3. * GPU-CPU interconnect bandwidth (eg. PCIE) is generally not a bottleneck (e.g., for most job types, we get similar benchmark performance on 8x or 16x PCIE lanes) but IO bandwidth reading data from cluster storage/local SSD is usually a significant factor in performance. This is especially true for preprocessing and CryoSPARC Live, as movies, micrographs, and particles need to be read, written, and transferred rapidly to keep up with collection and GPUs can process the data very quickly. * In many cases, older or slower GPUs can often perform almost equally as well as the newest, fastest GPUs because most computations in CryoSPARC are **not** bottlenecked by GPU compute speed, but rather by GPU memory bandwidth and disk I/O speed. ### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#browser-requirements) Browser Requirements The CryoSPARC web interface works best on the latest version of [Google Chrome](https://www.google.com/chrome/index.html) . [Firefox](https://www.mozilla.org/en-US/firefox/new/) and [Safari](https://www.apple.com/safari/) are also an option, although some features may not work as intended. Internet Explorer is not supported. [See this guide](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc) for more information on accessing the CryoSPARC web interface. [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#additional-configuration-notes) Additional Configuration Notes ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#network-accessibility) Network Accessibility Network security must be an important factor in the installation and ongoing management of any CryoSPARC instance. Access to the network that hosts a CryoSPARC instance must be carefully controlled, as CryoSPARC instances are not security-hardened against malicious actors on the network. CryoSPARC is designed to be run only within a trusted private network. CryoSPARC instances should never be directly hosted on the internet or an untrusted network, without a separate controlled authentication layer. CryoSPARC’s User Interface does include a user management system, and CryoSPARC user accounts and passwords help control access to the interface within a trusted private network, but please note that CryoSPARC passwords are not intended as a barrier against malicious access. ### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#root-access) Root Access The CryoSPARC system is specifically designed not to require root access to install or use. The reason for this is to avoid security vulnerabilities that can occur when a network application (web interface, database, etc.,) is hosted as the root user. For this reason, the CryoSPARC system must be installed and run as a regular UNIX user (`cryosparcuser`), and all input and output file locations must be readable and writable as this user. In particular, this means that project input and output directories that are stored within a regular user's home directory need to be accessible by `cryosparcuser`, or else (more commonly) another location on a shared file system must be used for CryoSPARC project directories. ### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#multi-user-environment) Multi-user environment If you are installing the CryoSPARC system for use by many users (for example within a lab), there are two options: #### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#using-unix-groups) Using UNIX Groups Create a new regular user (`cryosparcuser`) and install and run CryoSPARC as this user. Create a CryoSPARC project directory (on a shared file system) where project data will be stored, and create sub-directories for each lab member. If extra security is necessary, use UNIX group privileges to make each sub-directory read/writeable only by `cryosparcuser` and the appropriate lab member's UNIX account. Within the CryoSPARC command-line interface, create a CryoSPARC user account for each lab member, and have each lab member create their projects within their respective project directories. This method relies on the CryoSPARC web application for security to limit each user to see only their own projects. This is not guaranteed security, and malicious users who try hard enough will be able to modify the system to be able to see the projects and results of other users. #### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#using-separate-cryosparc-instances) Using Separate CryoSPARC Instances If each user must be guaranteed complete isolation and security of their projects, each user must install CryoSPARC independently within their own home directories. Projects can be kept private within user home directories as well, using UNIX permissions. Multiple single-user CryoSPARC master processes can be run on the same master node, and they can all submit jobs to the same cluster scheduler system. This method relies on the UNIX system for security and is more tedious to manage but provides stronger access restrictions. Each user will need to have their own CryoSPARC license ID in this case. ### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#deploying-cryosparc-on-aws) Deploying CryoSPARC on AWS CryoSPARC can be deployed on-premises or in the cloud. See below for a guide on deploying CryoSPARC on AWS resources. [Deploying CryoSPARC on AWS](https://guide.cryosparc.com/setup-configuration-and-management/cryosparc-on-aws) ### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#database-and-command-api-security) Database and Command API Security CryoSPARC is designed to run within a trusted private network. CryoSPARC instances are not security-hardened against malicious actors on the network and should never be exposed to the Internet or hosted on an untrusted network. The information in this section, Database and Command API Security, applies to CryoSPARC v4.0+. CryoSPARC v4.0 introduces additional authentication to reduce the likelihood of accidental mis-use by actors on a large institution/multi-user shared network. #### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#database-security) Database Security CryoSPARC's MongoDB database runs with access control enabled: Requests to read or write from the database must be authenticated with a username and password. CryoSPARC sets this password automatically and uses it for internal master → master and worker → master requests. Note that communication between the database and other CryoSPARC services is not encrypted. Enable or disable MongoDB access control by setting `CRYOSPARC_DB_ENABLE_AUTH` variable in `cryosparc_master/config.sh` and `cryosparc_worker/config.sh` to `true`(default) or `false`. To connect to the database with access control, use `cryosparcm mongo` to access the Mongo shell, or use `cryosparcm icli` to access an interactive Python client. #### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#command-api-security) Command API Security CryoSPARC's command API server (executes actions triggered by the web application including creating and modifying projects and jobs) also requires authentication. The command server expects the CryoSPARC License ID in the`License-ID` header of incoming web requests. CryoSPARC includes this automatically in internal requests to the API. Requests with missing or incorrect license ID will be rejected. Note that communication between the command server and other CryoSPARC services is not encrypted. You provide the License ID during CryoSPARC master and worker package installation. The license is written in plain-text to `config.sh` in the installation directories. The license ID in the worker installation must match the master installation. #### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#optional-password-argument) Optional password argument CryoSPARC allows creating users and updating users from the command line. Rather than specifying a `--password` flag during these operations, you may omit it to instead run a secure password prompt. [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#example-systems) Example Systems ---------------------------------------------------------------------------------------------------------------------------------------- We **do not** currently partner with any specific hardware vendors to sell machines with CryoSPARC pre-installed. ### [](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#example-hardware-systems) Example Hardware Systems Below are details of example workstations that meet or exceed the minimum requirements specified above, including those we use internally for development and testing. Example 4-GPU Example 2-GPU Component Hardware Product CPU 32 Cores (base clock 2.8GHz+), e.g, AMD Threadripper 3975X Memory 256GB DDR4 @ 3200MHz Storage 4TB PCIe SSD (cache); 200TB RAID 6 storage server via 10Gbps link (raw movies) GPU 4x NVIDIA Quadro GV100, or 4x NVIDIA Tesla V100 or 4x NVIDIA RTX 8000 Component Hardware Product CPU 16 Cores (base clock 3.0GHz+) Memory 128GB DDR4 Storage 2TB PCIe SSD (cache); HDD storage server in RAID configuration (raw movies) GPU 2x NVIDIA RTX 3090 [PreviousNon-commercial license agreement](https://guide.cryosparc.com/licensing/non-commercial-license-agreement) [NextCryoSPARC Installation Prerequisites](https://guide.cryosparc.com/setup-configuration-and-management/cryosparc-installation-prerequisites) Last updated 19 days ago * [CryoSPARC System Architecture Overview](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#cryosparc-system-architecture-overview) * [Master-worker pattern](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#master-worker-pattern) * [Typical CryoSPARC System Setups](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#typical-cryosparc-system-setups) * [Single Workstation](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#single-workstation) * [Master-Worker](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#master-worker) * [Clusters](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#clusters) * [CryoSPARC System Requirements](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#cryosparc-system-requirements) * [CryoSPARC Master Node Requirements](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#cryosparc-master-node-requirements) * [Worker Node/Cluster Worker Minimum Requirements](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#worker-node-cluster-worker-minimum-requirements) * [Operating System](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#operating-system) * [Disks and compression](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#disks-and-compression) * [Solid State Storage (SSDs)](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#solid-state-storage-ssds) * [Graphical Processing Units (GPUs)](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#graphical-processing-units-gpus) * [Browser Requirements](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#browser-requirements) * [Additional Configuration Notes](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#additional-configuration-notes) * [Network Accessibility](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#network-accessibility) * [Root Access](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#root-access) * [Multi-user environment](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#multi-user-environment) * [Deploying CryoSPARC on AWS](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#deploying-cryosparc-on-aws) * [Database and Command API Security](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#database-and-command-api-security) * [Example Systems](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#example-systems) * [Example Hardware Systems](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#example-hardware-systems) --- # Management and Monitoring (≤v4.7) | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring-4.7.md) . This page refers to CryoSPARC ≤v4.7. For v5.0+, please see [https://github.com/cryoem-uoft/guide-beta/blob/master/setup-configuration-and-management/management-and-monitoring-v5.0](https://github.com/cryoem-uoft/guide-beta/blob/master/setup-configuration-and-management/management-and-monitoring-v5.0) [](https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring-4.7#environment-variables) Environment variables ------------------------------------------------------------------------------------------------------------------------------------------------- Specify additional environment variables in the configuration files to augment CryoSPARC's low-level behaviour. [Environment variables (≤v4.7)](https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring-4.7/environment-variables-v4.7) [](https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring-4.7#cryosparcm-cryosparcm-cli-and-cryosparcw-references) cryosparcm, cryosparcm cli and cryosparcw references -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Workstations or master nodes with a `cryosparc_master` installation have access to `cryosparcm`, CryoSPARC's built-in [command-line](https://en.wikipedia.org/wiki/Command-line_interface) utility for all administrative, management and advanced usage tasks. [cryosparcm reference (≤v4.7)](https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring-4.7/cryosparcm-4.7) The `cryosparcm cli` command provides an extensive API for programmatically controlling cryoSPARC from the command-line. [cryosparcm cli reference (≤v4.7)](https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring-4.7/cli-4.7) Workstations or worker nodes with a `cryosparc_worker` installation have access to `cryosparcw`, a utility similar to `cryosparcm` for managing worker installations. [cryosparcw reference (≤v4.7)](https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring-4.7/cryosparcw-4.7) [PreviousSoftware Updates and Patches](https://guide.cryosparc.com/setup-configuration-and-management/software-updates) [NextEnvironment variables (≤v4.7)](https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring-4.7/environment-variables-v4.7) Last updated 4 months ago * [Environment variables](https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring-4.7#environment-variables) * [cryosparcm, cryosparcm cli and cryosparcw references](https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring-4.7#cryosparcm-cryosparcm-cli-and-cryosparcw-references) --- # Flat vs Hierarchical Navigation | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/application-guide/flat-vs-hierarchical-navigation.md) . Traditionally CryoSPARC has relied exclusively on a hierarchical navigation system, solely permitting workspaces to be viewed inside of their containing projects, and jobs to be viewed inside their containing workspaces. This has been reimagined in CryoSPARC v4 to allow navigating to each level (or granularity) of projects, workspaces, sessions, and jobs independently without having an upper level selection set. For instance, you could navigate directly to the jobs tab without selecting a project or workspace in order to see all jobs across the instance. This is nice to get a higher level view of what jobs are running, or have been run, in any shared projects or just generally to see the composition of your workflows. From here, you may want to open the filter options and filter all jobs available by type to compare results across the instance. By doing this, jobs can be aggregated and compared using a wide range of filtering options, either by their containers, or by their attributes. Flattened views can be directly accessed using the “Spotlight” menu (accessible by clicking the magnifying glass icon in the navigation bar, the search bar button on the home page header, or by pressing the `command` + `k` keys together on the keyboard. From here you can simply begin typing “all” and options for viewing all projects, workspace, sessions, or jobs will appear below. Navigating to one of these options and pressing the `enter` key, or simply clicking on one, will navigate you to that granularity with no parent granularity selected (eg. all workspaces with no project selected, or all jobs with no project or workspace selected. Flattened views are also automatically used for certain linking functions throughout the app to aggregate data with relevant filters applied automatically. An example of this is the “Job History” page. This can be accessed by clicking the overflow menu button with three horizontal dots in the navigation bar. The containing menu has an item called Job History in the first section with a clock icon beside the title. Clicking on the item will navigate you to the jobs granularity table view with no project or workspace selected and a status filter with statuses of completed, failed, and killed automatically applied. This was an entirely separate page in previous versions of CryoSPARC, but can now be built simply by adding the appropriate filters to the flattened jobs browse page. Further filters can be applied to this view if you would like to narrow the results further (such as a lane filter, user filter, and/or job type filter). This new aggregated data view can also be saved by bookmarking it in your browser or copying and saving the link constructed in the address bar. A CSV of the results shown in the filtered view can also be downloaded using the “Download CSV” button in the footer for archival or sharing purposes. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FiZ36lZ1LJDlZHAV6sCkG%252Fv4-0-0-app-browse-system-flat-navigation-example.jpg%3Falt%3Dmedia%26token%3D96383818-7626-470f-8184-1c98aec19ace&width=768&dpr=3&quality=100&sign=396b0dae&sv=2) The Job History view in the flattened jobs view with applicable filters applied. [PreviousTags](https://guide.cryosparc.com/application-guide/tags) [NextFile Browser](https://guide.cryosparc.com/application-guide/file-browser) Last updated 1 month ago --- # File Browser | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/application-guide/file-browser.md) . The file browser supports [Python's glob syntax](https://docs.python.org/3/library/glob.html) for Unix style pathname pattern expansion. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FxMeAKYuBxJqZC3UMwahW%252Fv4_5_file_browser_full%25402x.png%3Falt%3Dmedia%26token%3D0a61da3b-c674-4c94-873a-2918b526c1f0&width=768&dpr=3&quality=100&sign=b33bc5a2&sv=2) [](https://guide.cryosparc.com/application-guide/file-browser#search-and-bookmarks) Search and Bookmarks ------------------------------------------------------------------------------------------------------------- In v4.5+, the file browser allows for bookmarking directories for common access. To create a bookmark, right click on a directory and select 'Bookmark path' from the context menu that displays. Alternatively, click the bookmark button within the control bar to bookmark the current directory. Each bookmark can have a unique title, description and colour. Bookmarks are scoped to a specific CryoSPARC user account. You can search and navigate to bookmarked directories and project directories from the sidebar. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FBUX6WcZsSXDKK91xf5Yw%252Fv4_5_file_browser%25402x.png%3Falt%3Dmedia%26token%3D782714b2-0eb0-4fb9-bd4d-4a57d27d3188&width=768&dpr=3&quality=100&sign=1966005d&sv=2) [PreviousFlat vs Hierarchical Navigation](https://guide.cryosparc.com/application-guide/flat-vs-hierarchical-navigation) [NextBlueprints](https://guide.cryosparc.com/application-guide/blueprints) Last updated 1 month ago --- # Keyboard Shortcuts | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/application-guide/keyboard-shortcuts.md) . Keyboard shortcuts allow you to perform a variety of common actions more efficiently across the application. The [Shortcuts Dialog](https://guide.cryosparc.com/application-guide/keyboard-shortcuts#shortcuts-dialog) shows all available keyboard shortcuts for the application inside of hierarchical sections in a single panel. It also includes a full “fuzzy” search to quickly locate a shortcut of interest. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FXaM1KzqpcUPKwkk30qjR%252Fv5-0-0-shortcuts-dialog.png%3Falt%3Dmedia%26token%3D433f1824-9c29-443d-9947-0f9a4cb4eaad&width=768&dpr=3&quality=100&sign=5d901f0e&sv=2) The shortcuts dialog can be opened by clicking the “keyboard” icon button located on the main navigation bar at the leftmost side of the window. The button is located at the very bottom of the bar just above the user initials. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FGGv79dVwg1wE7rPAe6wt%252Foutput-onlinepngtools.png%3Falt%3Dmedia%26token%3D5bed14fe-fa51-4879-bf33-70993e984837&width=768&dpr=3&quality=100&sign=d0a78f5c&sv=2) All of the keyboard shortcuts available across the application are included in this dialog. Shortcuts are located in sections based on the areas of the application where they are applicable. Sections are organized further into subsections to create clearer delineations of where the shortcut can be used. Each section can be collapsed to hide its contents, this will also collapse all of its subsections. The **search bar** at the top of the dialog is designed to filter by section title, shortcut description, and the shortcut itself. Typing a search term into the bar will surface whichever one of these items that has the highest priority and closest match. If no matches are found, the search will attempt to surface shortcuts that are the closest match, or have related metadata. This can help you “fuzzy” search for shortcut that you may not remember exactly. [PreviousAdmin Panel](https://guide.cryosparc.com/application-guide/admin-panel) [NextImage Formation](https://guide.cryosparc.com/cryo-em-foundations/image-formation) Last updated 1 month ago --- # CryoSPARC Tools | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/cryosparc-tools.md) . **cryosparc-tools** is an open-source Python library that enables powerful scripting access to CryoSPARC and is available for **CryoSPARC v4.1+.** ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F4eUlrhACmkWJbqxteb5t%252Fv4-1-0-cryosparc-tools-architecture.png%3Falt%3Dmedia%26token%3D58fb89a5-613a-4da5-8b52-9e0e78fa4991&width=768&dpr=3&quality=100&sign=5481e323&sv=2) Use cases: * Programmatically read and write exposure, particle and volume data * Easily perform advanced operations on metadata (alignments, CTF, etc) and programmatically insert modified data back into CryoSPARC * Access project, workspace and job data * Build and run jobs to orchestrate custom cryo-EM workflows * Extend CryoSPARC functionality with third-party software packages * **cryosparc-tools** is [on GitHub](https://github.com/cryoem-uoft/cryosparc-tools) and available via `pip` and can be used outside of the CryoSPARC environment in your own programs and tools The full documentation for **cryosparc-tools** is hosted at [https://tools.cryosparc.com/](https://tools.cryosparc.com/) **cryosparc-tools** source code is hosted [on Github](https://github.com/cryoem-uoft/cryosparc-tools) . If you are using CryoSPARC v4.0 or older, please see: [PreviousAutomated Workflows](https://guide.cryosparc.com/processing-data/automated-workflows) [NextData Processing Tutorials](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies) Last updated 1 month ago --- # How to Access CryoSPARC Live | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/live/how-to-access-cryosparc-live.md) . You must have first installed CryoSPARC: How to [Download, Install and Configure](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure) . CryoSPARC Live is installed alongside CryoSPARC without any extra steps, for version v3.0+. [](https://guide.cryosparc.com/live/how-to-access-cryosparc-live#access-cryosparc-live-v4.0) Access CryoSPARC Live (v4.0+) ------------------------------------------------------------------------------------------------------------------------------- As of CryoSPARC v4.0.0, CryoSPARC Live is integrated directly within the main interface and does not require access through a separate port. To access All Live Sessions, click on the CryoSPARC Live icon on the navigation bar. [](https://guide.cryosparc.com/live/how-to-access-cryosparc-live#access-cryosparc-live-v3.3) Access CryoSPARC Live (**≤**v3.3) ----------------------------------------------------------------------------------------------------------------------------------- Once you start the CryoSPARC instance, the CryoSPARC Live web application is hosted on `CRYOSPARC_BASE_PORT` + `6`. In order to find `CRYOSPARC_BASE_PORT`, run the command `cryosparcm status`. ### [](https://guide.cryosparc.com/live/how-to-access-cryosparc-live#cryosparcm-status) `cryosparcm` status ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiqpn-KyXzhpmHV2Jj%252F-MNirZgth6ggkRZFwRQT%252FCSL_ACCESS_1_csm_status_v3-4x.png%3Falt%3Dmedia%26token%3D98c2d1dd-59ea-4e2a-b28b-5cdbf04abd1c&width=768&dpr=3&quality=100&sign=1497784c&sv=2) In this example, the base port is `61000` and so from the master node, use a web browser to access CryoSPARC Live at `http://localhost:61006` From other machines on the network, use the hostname of the master node to access CryoSPARC Live, in this example at `http://uoft:61006` [PreviousPrerequisites and Compute Resources Setup](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup) [NextUI Overview](https://guide.cryosparc.com/live/ui-overview) Last updated 5 months ago * [Access CryoSPARC Live (v4.0+)](https://guide.cryosparc.com/live/how-to-access-cryosparc-live#access-cryosparc-live-v4.0) * [Access CryoSPARC Live (≤v3.3)](https://guide.cryosparc.com/live/how-to-access-cryosparc-live#access-cryosparc-live-v3.3) * [cryosparcm status](https://guide.cryosparc.com/live/how-to-access-cryosparc-live#cryosparcm-status) --- # Managing Data | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/application-guide/managing-data.md) . Do not remove from the filesystem any directory that is managed by an [attached CryoSPARC project](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/guide-data-management-in-cryosparc-v4.0#2.-attaching-detaching-archiving-and-unarchiving-projects). First delete unwanted projects using the _Delete Project_ GUI action or the [`delete_project()` method of the CryoSPARC CLI](https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring-4.7/cli-4.7#delete_project-project_uid-str-request_user_id-str-all_jobs_in_project-list-all_workspaces_in_projec) . Available in the management dialog, the Project Data and Session Data tabs allow you to view and perform actions related to managing data size. [](https://guide.cryosparc.com/application-guide/managing-data#project-data) Project Data ---------------------------------------------------------------------------------------------- The project data table is a comprehensive view of all available projects across the instance with the intention of allowing decisions to be made in regards to data size on disk. The table can be sorted by any of its fields and each project row houses a nested workspaces table. This nested table shows all workspaces within said project and pertinent information about them. The actions column makes available a “Refresh Project Stats” button for each row, intended to allow the fetching of the most up to date project sizes. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FqL0dJdbN8XpQzkMC9PZH%252Fv4-0-0-app-managing-data-1.png%3Falt%3Dmedia%26token%3D54095583-4b3c-41a7-954d-b0f71d9855bb&width=768&dpr=3&quality=100&sign=be90f184&sv=2) [](https://guide.cryosparc.com/application-guide/managing-data#session-data) Session Data ---------------------------------------------------------------------------------------------- The session data table is similar to the project data table in intention, with more scope in actionable options. The table shows all projects with available sessions as top level rows, and the contained sessions as collapsable sub-rows beneath. Sorting the table by any of its columns will sort both the projects and contained sessions by the attribute. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FWj0qNvWDufrbcYLOuDXc%252Fv4-0-0-app-managing-data-2.png%3Falt%3Dmedia%26token%3D56d16dec-3c75-45b1-831d-7e007852ed6f&width=768&dpr=3&quality=100&sign=5e12f904&sv=2) The actions column allows the user to refresh stale data on the project level, which refreshes the project total size and the size of all sessions, or by a single session, which refreshes solely that session and the project total updated with its new size. The download button will download a set of sessions stats, and the link button will take you to the single session live view, closing the dialog. Each cell in the session row for the columns Raw Data, Micrographs, Thumbnails, Particles, and Metadata, are interactive and will trigger a management menu when clicked. These menus give a variety of options for managing session data depending on the session status and row type. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FKmJtA2VI06XWmLlHHwKH%252Fv4-0-0-app-session-data-management-menu.jpg%3Falt%3Dmedia%26token%3D86c55e29-795e-45ff-8ca6-967f9cffcab2&width=768&dpr=3&quality=100&sign=732d2a25&sv=2) For more information about CryoSPARC Live Session Data Management, please see: [Guide: CryoSPARC Live Session Data Management (≤v4.7)](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/cryosparc-live-session-data-management-4.7) [PreviousUpload Local Files](https://guide.cryosparc.com/application-guide/upload-local-files) [NextDownloading and Exporting Data](https://guide.cryosparc.com/application-guide/downloading-and-exporting-data) Last updated 1 month ago * [Project Data](https://guide.cryosparc.com/application-guide/managing-data#project-data) * [Session Data](https://guide.cryosparc.com/application-guide/managing-data#session-data) --- # Instance Management | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/application-guide/instance-management.md) . The management dialog can be opened on any page by clicking the “adjustments” button on the navigation bar located on the lefthand side of the app window. Once open, different sections can be navigated to using the tabs located at the top of the dialog, which correspond to sections for the management of jobs, tags, project and session data, as well as instance-level information. Alternatively, manage dialog sections can be navigated to directly by opening the navigation bar “triple dot” overflow menu and clicking on an option in the first section of the menu. The manage dialog will appear above the current page content without navigating you away from the page. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FKTBMC8R1taV3E4OuJfmw%252Fv4-0-0-app-instance-management-access.jpg%3Falt%3Dmedia%26token%3Db2439e5a-f48e-4c10-b4b8-f667ce31747b&width=768&dpr=3&quality=100&sign=ef00332c&sv=2) ### [](https://guide.cryosparc.com/application-guide/instance-management#instance-tab) Instance Tab The instance tab provides a read-only view of all lanes and targets configured via the command line. For more information on how to configure CryoSPARC processing nodes, refer to the following page on the guide: [Connecting a Worker Node](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/downloading-and-installing-cryosparc#connecting-a-worker-node) . ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FqClKV9htjCpDfF3bBZbw%252Fv4-0-0-app-instance-management-instance-tab-1.png%3Falt%3Dmedia%26token%3D2805771f-990d-403d-8a21-b02e903c9503&width=768&dpr=3&quality=100&sign=d669d4a8&sv=2) Instance Tab in v4.0 v5.0 and later introduce a substantially redesigned _Instance_ tab that expands both visibility and operational control. The updated interface provides a structured table of GPU information for available targets, along with comprehensive search and filtering capabilities across lanes, targets, and GPUs. It also enables direct copying of filesystem paths and cluster submission scripts, streamlining workflow integration. Lanes and targets can be viewed as cards or in a table. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F3KWdcmW9SfTnbXP8Nk9G%252Fv5-0-0-instance-tab.png%3Falt%3Dmedia%26token%3D5d64e51c-a0fc-4c1d-ba64-d6a39862a428&width=768&dpr=3&quality=100&sign=adc8b121&sv=2) Instance Tab (Card View) in v5.0+ ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FOaazYav9nuzRG6syvAzk%252Fv5-0-0-instance-tab-table.png%3Falt%3Dmedia%26token%3Dbe1b1931-0c12-45ab-9487-433f19bd1b33&width=768&dpr=3&quality=100&sign=33111906&sv=2) Instance Tab (Table View) in v5.0+ ### [](https://guide.cryosparc.com/application-guide/instance-management#backups-tab) Backups Tab The backups tab displays a history of the most recent backups that were configured via the command line. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F63mh9gOqVWo4FdoBS6LX%252Fv4-0-0-app-instance-management-backups-tab-1.png%3Falt%3Dmedia%26token%3D7d5c24bc-5858-4595-9b18-ed7b54e899e9&width=768&dpr=3&quality=100&sign=e22cdc80&sv=2) ### [](https://guide.cryosparc.com/application-guide/instance-management#notifications-tab) Notifications Tab Notifications are presented in the CryoSPARC interface from local actions (such as queuing a job) and external events (such as the progress of a project export). In most cases these notifications are hidden when an event is resolved (i.e. the project export completes). In rare cases you may want to manually clear an active notification from displaying. This can be done by clicking on the ‘Clear’ button next to any active notification: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FvYmqsqgbwwYFsNIk8xaF%252Fv4-0-0-app-instance-management-notifications-tab-1.png%3Falt%3Dmedia%26token%3Da6a71326-3430-4a58-aad3-8150a2b279b4&width=768&dpr=3&quality=100&sign=8d40ba4e&sv=2) To view a history of inactive notifications, click on the ‘Inactive’ toggle: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FbOGdi3ytWiuQ9IX51SUN%252Fv4-0-0-app-instance-management-notifications-tab-2.png%3Falt%3Dmedia%26token%3D2064200b-6950-40cf-944a-29eea955b320&width=768&dpr=3&quality=100&sign=7c246649&sv=2) [PreviousDownloading and Exporting Data](https://guide.cryosparc.com/application-guide/downloading-and-exporting-data) [NextAdmin Panel](https://guide.cryosparc.com/application-guide/admin-panel) Last updated 1 month ago * [Instance Tab](https://guide.cryosparc.com/application-guide/instance-management#instance-tab) * [Backups Tab](https://guide.cryosparc.com/application-guide/instance-management#backups-tab) * [Notifications Tab](https://guide.cryosparc.com/application-guide/instance-management#notifications-tab) --- # CryoSPARC Live Tutorial Videos | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/live/tutorial-videos.md) . [](https://guide.cryosparc.com/live/tutorial-videos#cryosparc-live-walkthrough) CryoSPARC Live Walkthrough --------------------------------------------------------------------------------------------------------------- Join us while we process cryo-EM data (EMPIAR-10288) using cryoSPARC Live. We'll walk you through the interface and process this dataset from start to finish. CryoSPARC Live is available with the cryoSPARC software platform: [How to Access CryoSPARC Live](https://guide.cryosparc.com/live/how-to-access-cryosparc-live) For more information about starting your own Session in cryoSPARC Live, see: [New Live Session: Start to Finish Guide](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide) Download EMPIAR-10288, the dataset used in this walkthrough: [![Logo](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2Fwww.ebi.ac.uk%2Fem_static%2Fempiar%2Ffavicon%2Fapple-touch-icon.png&width=20&dpr=3&quality=100&sign=50e2f750&sv=2)EMPIAR-10288 Cryo electron microscopy of Cannabinoid Receptor 1-G Protein Complexwww.ebi.ac.uk](https://www.ebi.ac.uk/pdbe/emdb/empiar/entry/10288/) The gain reference file provided in this dataset is in a file format (`.dm4`) that cryoSPARC doesn't support. [Click this link to download a compatible version of the gain reference to use in cryoSPARC.](https://structura-assets.s3.amazonaws.com/EMPIAR-10288-gainreference/CountRef_CB1__00826_Feb19_14.19.25.mrc) #### [](https://guide.cryosparc.com/live/tutorial-videos#parameters-used-for-data-processing) Parameters Used For Data Processing: * Microscope/Camera * Raw pixel size (Å): 0.86 * Accelerating voltage (kV): 300 * Spherical aberration (mm): 2.7 * Total exposure dose (e/Å^2): 58 * Advanced: Flip gain ref in Y? : True * Blob Picker * Minimum particle diameter (Å): 100 * Maximum particle diameter (Å): 150 * Advanced: Use elliptical blob: True * Extraction box size (pix): 320 * Advanced: Fourier crop to box size (pix): 240 [](https://guide.cryosparc.com/live/tutorial-videos#webinar-real-time-cryo-em-analysis-for-all-cryosparc-live) Webinar - Real-time cryo-EM analysis for all: cryoSPARC Live -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Learn how cryoSPARC Live, a seamless real-time 2D and 3D processing system for single particle cryo-EM, accelerates time-to-structure and drives rapid insights into sample characteristics and data quality, enabling decision making while the sample is still in the microscope. CryoSPARC Live is not just for facilities; it is also the fastest, simplest way for beginners and experts alike to process cryo-EM data that has already been collected. We will cover use cases, performance considerations, real-time experimentation and practical workflows, and will be joined by two expert guest speakers, Giovanna Scapin (NIS) and Craig Yoshioka (PNCC), who will discuss how they use cryoSPARC Live in practice in both industry and academic settings. [](https://guide.cryosparc.com/live/tutorial-videos#cryosparc-live-10-minute-overview) cryoSPARC Live: 10-Minute Overview ------------------------------------------------------------------------------------------------------------------------------ For more information about starting your own Session in cryoSPARC Live, see: [New Live Session: Start to Finish Guide](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide) [PreviousNew Live Session: Start to Finish Guide](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide) [NextLive Jobs and Session-Level Functions](https://guide.cryosparc.com/live/live-jobs-and-session-level-functions) Last updated 3 years ago * [CryoSPARC Live Walkthrough](https://guide.cryosparc.com/live/tutorial-videos#cryosparc-live-walkthrough) * [Webinar - Real-time cryo-EM analysis for all: cryoSPARC Live](https://guide.cryosparc.com/live/tutorial-videos#webinar-real-time-cryo-em-analysis-for-all-cryosparc-live) * [cryoSPARC Live: 10-Minute Overview](https://guide.cryosparc.com/live/tutorial-videos#cryosparc-live-10-minute-overview) --- # A Tour of the CryoSPARC Interface | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/application-guide/a-tour-of-the-cryosparc-interface.md) . The information in this section applies to CryoSPARC v4.0+. For the Application Guide applicable to CryoSPARC ≤v3.3, please see: [v3 User Interface Guide](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide) [](https://guide.cryosparc.com/application-guide/a-tour-of-the-cryosparc-interface#overview) Overview ---------------------------------------------------------------------------------------------------------- CryoSPARC v4.0 Application Walkthrough. The CryoSPARC application is a web interface that makes it easy to quickly and efficiently process cryo-EM data from raw movies to a high resolution structure. Within the interface you can create projects and workspaces to organize data, queue and run jobs, view and share their results and outputs, and export information for record keeping and use in other software. CryoSPARC also features additional tools to organize, search, and view data in a variety of different ways. This section of the guide will provide an overview of all these features. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FVAQf9sC8XWKU3kVZGzk2%252Fv4-0-0-app-tour-1.png%3Falt%3Dmedia%26token%3D9e04e22d-5938-4cd4-a84a-82768a9100ba&width=768&dpr=3&quality=100&sign=96d0f008&sv=2) [](https://guide.cryosparc.com/application-guide/a-tour-of-the-cryosparc-interface#application-layout) Application Layout ------------------------------------------------------------------------------------------------------------------------------ ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FvCrXLpvaCdLTElOLHHLn%252Fv4-0-0-app-tour-2.png%3Falt%3Dmedia%26token%3Db6127b7c-119b-4123-8db3-7908cc9270e1&width=768&dpr=3&quality=100&sign=b4452af6&sv=2) The interface is comprised of five primary elements: 1. The navigation bar is located on the left side of the screen. It contains links to the homepage (Dashboard), the project browse view, and the session browse view. It also includes buttons to open various management dialogs, such as the 'current jobs' dialog and spotlight search dialog. 2. At the top of the page are the navigation controls, used to navigate to and switch between different view levels (projects, workspace, sessions, jobs). It is similar to the navigation controls in CryoSPARC v3 with added clarity and utility. 3. Below the navigation is a large content area that will adapt based on the page you're viewing. Every page has a centralized action bar with key elements such as filter controls. 4. Below the content area is a footer designed to provide quick access to current (queued or active) jobs. When browsing projects, workspaces, sessions, or jobs, the footer adapts to display a total count and various filter options. 5. To the right of the content area is a sidebar with three tabs: details, builder (for building and editing jobs) and cart (for filtering and creating jobs based on the outputs of completed jobs). Similar to CryoSPARC v3, you can select cards to view their details and perform actions. Additionally, you can now collapse the sidebar in order to view more of the main content area. [](https://guide.cryosparc.com/application-guide/a-tour-of-the-cryosparc-interface#navigation-bar) Navigation Bar ---------------------------------------------------------------------------------------------------------------------- ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FbPaW32h4YdCNeTpVrcHS%252Fv4-0-0-navigation-bar%25402x.png%3Falt%3Dmedia%26token%3Dd3116219-193e-4563-b689-51cb00b589f5&width=768&dpr=3&quality=100&sign=cb477279&sv=2) The navigation bar is located on the left side of the screen. It contains links to the: * Dashboard * Projects browser (all projects) * Session browser (all sessions) * Management dialog containing several tabs for managing jobs, data and instance information * An overflow menu with additional dialogs and actions [](https://guide.cryosparc.com/application-guide/a-tour-of-the-cryosparc-interface#dashboard) Dashboard ------------------------------------------------------------------------------------------------------------ ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fc1EykG6xRsVSNzCyBn58%252Fv4-0-0-app-dashboard.jpg%3Falt%3Dmedia%26token%3D5de9f2f8-4fca-42f6-96ef-67cb24f9d6f0&width=768&dpr=3&quality=100&sign=f567ae09&sv=2) The first page you'll see after logging in is the Dashboard. It displays an overview of your CryoSPARC instance with helpful modules such as a processing history heat-map and charts to get a sense of what has been run recently as well as a module to see all active jobs. You can filter these modules to see information about only your jobs or jobs across all users within the instance. The Dashboard also includes additional modules featuring external CryoSPARC and community resources such as links to tutorials on the CryoSPARC Guide, trending posts on the CryoSPARC Discussion Forum and the latest EMPIAR and EMDB uploads. [PreviousTroubleshooting](https://guide.cryosparc.com/setup-configuration-and-management/troubleshooting) [NextBrowsing the CryoSPARC Instance](https://guide.cryosparc.com/application-guide/browsing-the-cryosparc-instance) Last updated 1 month ago * [Overview](https://guide.cryosparc.com/application-guide/a-tour-of-the-cryosparc-interface#overview) * [Application Layout](https://guide.cryosparc.com/application-guide/a-tour-of-the-cryosparc-interface#application-layout) * [Navigation Bar](https://guide.cryosparc.com/application-guide/a-tour-of-the-cryosparc-interface#navigation-bar) * [Dashboard](https://guide.cryosparc.com/application-guide/a-tour-of-the-cryosparc-interface#dashboard) --- # Tags | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/application-guide/tags.md) . Tags enable a useful level of user-customizable organization in your CryoSPARC instance. Tags are shared among all users within the instance and can be customized with a title, description, and colour. There are two categories of tags, a 'general' type tag that can be applied to items of any type, and item-specific tags that only apply to either project, workspaces, sessions, or jobs. The main purpose of tags are to allow you to quickly find and view specific subsets of data either inside of a container (projects or workspaces) or across the entire instance. This gives a lot of flexibility for creating arbitrary groupings of items that you want to be able to find again quickly or compare and reference. [](https://guide.cryosparc.com/application-guide/tags#accessing-tags) **Accessing Tags** --------------------------------------------------------------------------------------------- The main access point for creating and navigating tags is the Quick Access Menu that can be expanded from the navigation bar on the left side of the app window. By clicking the “Tags” tab at the top of the menu you will be able to see all of the tags currently created in your instance. Each tag grouping is represented here in a collapsable drawer with the type of tag and total count of tags with that type shown on the drawer header. You can use the search bar at the top to filter the available tags and quickly find one that you are looking for. Beside the search bar there is a “+” button that will open the tag creation slide-over and allow you to create a new tag. Once created that new tag will be available to view and use. Each tag row in the menu shows the unique tag ID (e.g., T5, T20, etc.), the tag title, and a count of how many items have been given that tag (eg. a project tag of EMPIAR with a count of 10 has been assigned to 10 projects in the instance). Clicking on one of these rows will navigate you to the relevant view with a filter for that tag applied (in the case of general tags, a context menu will open when the row is clicked and allow you to select the type of item you want to view). ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FNghvrUQ9pFvXbD2BsqPM%252Fv4-0-0-app-browse-system-quick-access-tags.jpg%3Falt%3Dmedia%26token%3Da545c4d5-b970-461d-a181-1e3e50405441&width=768&dpr=3&quality=100&sign=18694bb8&sv=2) [](https://guide.cryosparc.com/application-guide/tags#applying-tags) **Applying Tags** ------------------------------------------------------------------------------------------- Once tag(s) exist in the instance, they can be applied to any relevant item. There are multiple ways to apply a tag depending on how you are interacting with the item. Let’s look at applying a tag to a project as an example. By navigating to the browse section we will see all of our projects in the cards view. We can choose a project to apply our tag to and use either the quick actions menu, or the sidebar to add it. Right clicking the card or clicking the triple dot menu in the header will open the quick actions menu, we can then navigate down to the “Edit tags” menu item and into the sub menu with a list of relevant tags. Project tags will appear first and general tags below. Each tag in the menu has the same information as the rows in the quick access menu, the ID, title, and a count of how many items that tag has been applied to. By clicking any tag in the menu (lets take our EMPIAR tag for example) that tag will be applied to the project. Opening the menu again and viewing the “Edit tags” submenu will show a checkmark on any tags that have been applied to the card (in this case EMPIAR). Clicking the tag row again will remove the tag from the project. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FmkfaREbmK0Dm4SB8pavK%252Fv4-0-0-app-browse-system-tagging-project.jpg%3Falt%3Dmedia%26token%3D913da833-4a0c-4bbd-882f-665392ff1ab6&width=768&dpr=3&quality=100&sign=dc134d33&sv=2) Tags can also be seen in the item sidebar, inside the “Details” panel (which is the first panel from the top). The tags row is just below the title row and displays all tags applied to the item. When this row is hovered, an edit button will appear, clicking this button will open the same “Edit tags” menu with the exact same functionality as the one available from the quick actions menu, discussed above. Tags can be applied to projects, workspaces, sessions, and jobs in exactly the same way. The only difference is the granularity specific tags available. [](https://guide.cryosparc.com/application-guide/tags#using-tags) **Using Tags** ------------------------------------------------------------------------------------- As mentioned above, the main use for tags is organizing data into subsets that either compliment the existing project and workspace demarcations, or acts as a wider aggregator around them. Tags are fundamentally custom filters, and as such operate functionally within the filter system. The control bar has a “Tags” quick filter button above the main content area, where tag filters can be set and removed. Tag filters are also available in the filter bar menu with all other applicable filters. Clicking the “Tags” quick filter button or entering the “Tags” filter submenu from the filter bar will open the same menu with identical functionality. Clicking on a tag in this menu will add a “Tags” filter group with the specific item applied. This will cause only items with this tag to be shown in the interface (eg. if you add the EMPIAR tag as a filter on the projects view, only projects with the EMPIAR tag applied to them will now show in the card grid). Adding additional tags is an additive filtering process and will show all items with any of the added tags applied to them. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F8IwPXx7Jp7KsgcbQPHbh%252Fv4-0-0-app-browse-system-tag-filtering-projects.jpg%3Falt%3Dmedia%26token%3D150aa575-ab13-4172-8fd6-2a2d73c45790&width=768&dpr=3&quality=100&sign=819ba34b&sv=2) [](https://guide.cryosparc.com/application-guide/tags#tag-management) Tag Management ----------------------------------------------------------------------------------------- ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FDFNiZQctpAc99eTTKvjn%252Fv5-0-0-tag-management-dialog.png%3Falt%3Dmedia%26token%3D1c831388-6b74-4042-a4da-2af2a700b24a&width=768&dpr=3&quality=100&sign=1aba9d7f&sv=2) Tags can be managed in detail from the **Tags** tab within the management dialog. This view presents all tags in a table, displaying their relevant metadata, including ID, title, description, type, author, and creation date. Each row represents a single tag and includes action controls at the end of the row for editing or deleting that tag. #### [](https://guide.cryosparc.com/application-guide/tags#filters) Filters Several filtering options are available to simplify tag management. These controls are located in the top bar above the table. * **Tag Type**: Filter the table to display only tags of a specific category (e.g., General, Projects, or Jobs). * **Mine / All Toggle**: Limit the view to tags created by the current user or display all tags available in the instance. * **Search**: Filter tags by direct text match. The search respects and combines with any other active filters. #### [](https://guide.cryosparc.com/application-guide/tags#editing) Editing Selecting **Edit** from the tag row’s context menu (accessible via the three-dot icon) opens the edit modal. From this modal, you can modify the tag’s title, update its description, and select a new colour. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FrfGiFL5O7IZpDN4zYzca%252Fv4-0-0-app-browse-system-edit-tag.jpg%3Falt%3Dmedia%26token%3D29239338-19e1-4a2c-ac97-b219c58c13c1&width=768&dpr=3&quality=100&sign=21b06af1&sv=2) #### [](https://guide.cryosparc.com/application-guide/tags#deleting) **Deleting** Deleting a tag permanently removes it from the instance and from all items to which it has been applied. **This action is irreversible.** #### [](https://guide.cryosparc.com/application-guide/tags#multi-actions) Multi-Actions The tag management table includes checkboxes in the first column, allowing multiple tags to be selected at once. Multi-selection respects all active filters and will only include tags currently visible in the filtered view. Once one or more tags are selected, a **Delete {X} Tags** button appears in the top bar. Selecting this option permanently deletes all selected tags from the instance and removes them from any associated items. As with single-tag deletion, this action cannot be undone. [](https://guide.cryosparc.com/application-guide/tags#tag-use-cases) **Tag Use Cases** ------------------------------------------------------------------------------------------- * Progression of a project by assigning various lifecycles (e.g., 'to-do', 'in-progress', 'done') * Denoting the type of microscope used to collect the data present in a project (e.g., 300KV, Krios) * Adding a demarcation of a quality result that can be referenced or returned to in the future (e.g., Good result) * Using a tag for reconstructions of a specific particle type that can be referenced together across the instance regardless of project (e.g., `CoV S Protein` tag) [PreviousView Options](https://guide.cryosparc.com/application-guide/view-options) [NextFlat vs Hierarchical Navigation](https://guide.cryosparc.com/application-guide/flat-vs-hierarchical-navigation) Last updated 1 month ago * [Accessing Tags](https://guide.cryosparc.com/application-guide/tags#accessing-tags) * [Applying Tags](https://guide.cryosparc.com/application-guide/tags#applying-tags) * [Using Tags](https://guide.cryosparc.com/application-guide/tags#using-tags) * [Tag Management](https://guide.cryosparc.com/application-guide/tags#tag-management) * [Tag Use Cases](https://guide.cryosparc.com/application-guide/tags#tag-use-cases) --- # Downloading and Exporting Data | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/application-guide/downloading-and-exporting-data.md) . [](https://guide.cryosparc.com/application-guide/downloading-and-exporting-data#downloading-lists-of-projects-workspaces-sessions-and-jobs-as-a-csv-file) Downloading Lists of Projects, Workspaces, Sessions, and Jobs as a CSV File ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ The browse system includes the capability to download a CSV of the data shown in any particular view. This download will respect all filters and any sorting options applied to the view. Initiating a CSV download can be accomplished by clicking the download button on the far right side of the application footer. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FkJIxzXHUqlnQYAHwZ3VG%252Fv4.2-csv-download-button.png%3Falt%3Dmedia%26token%3De6b16297-4501-4495-94c0-3f80fdc53469&width=768&dpr=3&quality=100&sign=5626b493&sv=2) CSV download button is located in the footer of all browse pages Pressing this button will open a dialog for customizing the CSV to suit your needs. Here you may update the CSV file name and select what information you would like to have appear in the CSV. The options shown in the “Table Columns” section represent columns in the CSV download and can be toggled for inclusion or exclusion. These options can also be dragged and dropped within the list to reorder the columns of the CSV table. Column options are mapped so that top to bottom in the download list corresponds to left to right in the CSV. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FhOcYFLGUtDoXvrCCb9cA%252Fv5-0-0-csv-dialog.png%3Falt%3Dmedia%26token%3D7aa1c5f3-4852-408f-bdf2-f89f42c87a35&width=768&dpr=3&quality=100&sign=abc5f7fd&sv=2) Dialog displaying options to customize the CSV file Clicking the blue “Download” button at the bottom of the dialog will download the formatted CSV to your device. In v5.0+ additional options have been introduced such as the inclusion of info tags and the ability to only download jobs that you have manually selected. You can download a list of all completed jobs across the entire instance this year by searching the spotlight (`command` + `k`) for ‘All jobs’ and adding a status filter of ‘completed’ and selecting a start and end date. [](https://guide.cryosparc.com/application-guide/downloading-and-exporting-data#downloading-job-results) Downloading Job Results ------------------------------------------------------------------------------------------------------------------------------------- From the browse view, you can select a job and inspect it to view all job output groups. Each contain a list of items (such as `.cs` file metadata or `.map` volumes) to download: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FGVgHW1mKrbFIumhDYmIO%252Fv4-0-0-app-downloading-2.png%3Falt%3Dmedia%26token%3D30f99c30-02be-4d7c-afa3-c6730f5c703f&width=768&dpr=3&quality=100&sign=7f63319a&sv=2) Viewing and downloading output results from the job inspection dialog. Alternatively, the list of output groups is available in the details sidebar of a job when selected under the “Outputs” panel: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FZNE96Rhi1zarN7WWwDwg%252Fv4-0-0-app-downloading-3.png%3Falt%3Dmedia%26token%3D260454db-391e-46c5-b020-232d3aa85411&width=768&dpr=3&quality=100&sign=600c4673&sv=2) Viewing and downloading outputs from the job details sidebar. Additional download options are available from the “Outputs” tab of the job inspection dialog. Here you can copy the file path or download individual low-level results as well as download results from a specific iteration: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FH5jIwebGi9Yocscs2mt2%252Fv4-0-0-app-downloading-4.png%3Falt%3Dmedia%26token%3Dea22c0d7-ac16-4978-ab10-9fc38477e600&width=768&dpr=3&quality=100&sign=1d5e7378&sv=2) Downloading the alignments3D low-level result from iteration 2 of this Homogeneous Refinement job. [](https://guide.cryosparc.com/application-guide/downloading-and-exporting-data#downloading-the-job-event-log) Downloading the Job Event Log ------------------------------------------------------------------------------------------------------------------------------------------------- Often it is helpful to download a standalone copy of the processing history of a job. In CryoSPARC v4.0 you can now generate a PDF job that contains a cover page of important metadata and the full event log including images. This makes it easy to archive and share the results of a job. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fip5MYSg5zKvFJZUVuytq%252Fv4-0-0-app-downloading-5.png%3Falt%3Dmedia%26token%3D14767ab6-8bb9-4116-ba21-f97f211e391d&width=768&dpr=3&quality=100&sign=dc28ec39&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fwfz4B6518sxmuP8MBwBf%252Fv4-0-0-app-downloading-6.png%3Falt%3Dmedia%26token%3D78f447ab-0ef4-4811-8120-5390abab0033&width=768&dpr=3&quality=100&sign=6cd31daf&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F4iQEmiVJTMLU2kJqaPfr%252Fv4-0-0-app-downloading-7.png%3Falt%3Dmedia%26token%3D2f506002-5751-4602-827d-1764efe00126&width=768&dpr=3&quality=100&sign=ccd4b7a5&sv=2) [](https://guide.cryosparc.com/application-guide/downloading-and-exporting-data#downloading-a-job-report) Downloading a Job Report --------------------------------------------------------------------------------------------------------------------------------------- In addition to downloading just the job event log, you can download the event log and a set of CryoSPARC system logs for the purposes of debugging. You can choose to include or exclude images in the event log PDF. The report is packaged in a compressed ZIP file. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FweRCwZPD3FzP3oSCTF7D%252Fv4-0-0-app-downloading-8.png%3Falt%3Dmedia%26token%3D2986d78d-2999-4740-95aa-7ac3183060d5&width=768&dpr=3&quality=100&sign=99508f8c&sv=2) [](https://guide.cryosparc.com/application-guide/downloading-and-exporting-data#exporting-jobs) Exporting Jobs ------------------------------------------------------------------------------------------------------------------- Refer to this guide on exporting jobs: [![Logo](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Ficon%252FCeHcWPT0EsCRkHxr4GZX%252Fstructura-profile-purple.png%3Falt%3Dmedia%26token%3D39f4a203-8992-4197-acd4-2191eec1b4fa&width=48&height=48&sign=66b523e0&sv=2)Guide: Data Management in CryoSPARC (≤v3.3) | CryoSPARC Guideguide.cryosparc.com](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/tutorial-data-management-in-cryosparc#use-case-share-a-particular-job-with-another-user) [](https://guide.cryosparc.com/application-guide/downloading-and-exporting-data#exporting-projects) Exporting Projects --------------------------------------------------------------------------------------------------------------------------- Refer to this guide on exporting projects: [![Logo](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Ficon%252FCeHcWPT0EsCRkHxr4GZX%252Fstructura-profile-purple.png%3Falt%3Dmedia%26token%3D39f4a203-8992-4197-acd4-2191eec1b4fa&width=48&height=48&sign=66b523e0&sv=2)Guide: Data Management in CryoSPARC (v4.0+) | CryoSPARC Guideguide.cryosparc.com](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/guide-data-management-in-cryosparc-v4.0+) [PreviousManaging Data](https://guide.cryosparc.com/application-guide/managing-data) [NextInstance Management](https://guide.cryosparc.com/application-guide/instance-management) Last updated 1 month ago * [Downloading Lists of Projects, Workspaces, Sessions, and Jobs as a CSV File](https://guide.cryosparc.com/application-guide/downloading-and-exporting-data#downloading-lists-of-projects-workspaces-sessions-and-jobs-as-a-csv-file) * [Downloading Job Results](https://guide.cryosparc.com/application-guide/downloading-and-exporting-data#downloading-job-results) * [Downloading the Job Event Log](https://guide.cryosparc.com/application-guide/downloading-and-exporting-data#downloading-the-job-event-log) * [Downloading a Job Report](https://guide.cryosparc.com/application-guide/downloading-and-exporting-data#downloading-a-job-report) * [Exporting Jobs](https://guide.cryosparc.com/application-guide/downloading-and-exporting-data#exporting-jobs) * [Exporting Projects](https://guide.cryosparc.com/application-guide/downloading-and-exporting-data#exporting-projects) --- # View and Download Results | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/view-and-download-results.md) . [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/view-and-download-results#view-and-download-results) View and Download Results --------------------------------------------------------------------------------------------------------------------------------------------------------------- As soon as a job starts, it will create output groups that can be viewed on the right hand side of the job stream log. Hovering over an output group (shown as a single draggable tile) will display a list of the available outputs within that group (e.g., map). Select an output from the drop-down to download files directly. This is especially helpful for downloading volumes for inspection using tools like Chimera. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNpKl4WOHqlu0z8dhRm%252F-MNpL-gPOCh3LIqmUSSQ%252FUI_RM_download_output_1.png%3Falt%3Dmedia%26token%3D46256478-6954-4260-959d-c77e6baebdc4&width=768&dpr=3&quality=100&sign=a182d15b&sv=2) [PreviousQueue Job, Inspect Job and Other Job Actions](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions) [NextJob Relationships](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/job-relationships) Last updated 1 year ago --- # Admin Panel | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/application-guide/admin-panel.md) . The admin panel can be accessed by clicking on the “key” icon located on the navigation bar on the lefthand side of the app window. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FJvwD0XPgmKn5qcbIKU8j%252Fv4-0-0-app-admin-panel-button.jpg%3Falt%3Dmedia%26token%3D7813155c-8741-46c9-bfc6-7381a31e9b4b&width=768&dpr=3&quality=100&sign=6c5f69c1&sv=2) [](https://guide.cryosparc.com/application-guide/admin-panel#instance-settings) Instance Settings ------------------------------------------------------------------------------------------------------ ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F7WUPnGIL80a9TlVhGD5k%252Fv4-0-0-app-instance-settings.jpg%3Falt%3Dmedia%26token%3D68baa2fd-cf71-41f6-abf2-ae2396846232&width=768&dpr=3&quality=100&sign=76c3e555&sv=2) The instance settings tab allows you to set general instance wide defaults. Two configurable fields exist here, the instance name, and the default instance job priority. * **Instance Name:** The instance name is simply a name to differentiate your instances visually. * **Default Instance Job Priority:** This is the default priority that jobs run within the instance will be given when launched. The job priority differentiates the importance between different jobs launched in an instance and allows more important jobs to run before less important jobs. The priority can be set on a job by job basis, and on a user default basis. By setting the default instance job priority, users can be given priority above or below this mark, and jobs can similarly be launched above or below this mark. If a “Lane Queue Override” has been set, an indicator will also appear here. This override allows for import jobs to be queued on lanes outside of the default. [](https://guide.cryosparc.com/application-guide/admin-panel#instance-logs) Instance Logs ---------------------------------------------------------------------------------------------- ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FHEkMAZuuuzPLPVDCfOa4%252Fv4-0-0-app-instance-logs.jpg%3Falt%3Dmedia%26token%3D42487dc1-1c9f-4a90-8ed0-3a44065bedb3&width=768&dpr=3&quality=100&sign=a2fd3205&sv=2) These are collections of the logged messages and errors from the instance. Selection menus on the top bar allow you to choose between a variety of different logs including: database, app, app\_api, app\_legacy, command\_core, command\_vis, and command\_rtp. The number of lines can be toggled to 50, 100, 200, or 500. The download report button will download a zipped folder of all of the available log files as well as browser and runtime diagnostics files. This information allows you to quickly diagnose any problems or behaviours your instance may be experiencing, and can be invaluable for debugging purposes. [](https://guide.cryosparc.com/application-guide/admin-panel#user-management) User Management -------------------------------------------------------------------------------------------------- ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FtdLxaqrixjnyCDnYNlSZ%252Fv4-0-0-admin-panel-user-management-1.png%3Falt%3Dmedia%26token%3Dd6a97170-84d6-48d2-8f89-68f39f024f0d&width=768&dpr=3&quality=100&sign=165c3cdd&sv=2) The user management tab is the hub for curating users across an instance. User details are displayed and a variety of general actions are contained in applicable columns. * **Email:** The user email can be edited by hovering the email cell and clicking the pencil icon. This will transition the email to an editable input. To exit without making any changes, simply click the “X” button, to confirm and update changes click the “check” button. (Clicking outside of the input will also close it without making any changes). * **Role:** The user’s role can be changed between “User” and “Admin”. The admin privilege will allow that user to perform any admin actions across the instance, and will also allow them to see and manage other user’s work. This operation cannot be performed on the person currently editing, and so the selection button is disabled for the current user. * **Tokens:** These are the reset and register tokens made available to allow a user to either reset their password, or register their account for the first time. An admin must deliver these tokens to the user so that they can use them on the register and reset pages. * **Live Data Management:** This toggles the users ability to perform actions on specific sessions within the “Session Management Table” in the “Manage” dialog. If the user has this permission, they can perform action on the sessions such as deleting and archiving session data. * **Job Priority Management:** This toggles the users ability to set a priority on a per job basis. If toggled on, the user will see the priority value field in the job builder, and can set it higher or lower than the instance or user priority value. * **Default Job Priority:** This value represents the default job priority on a user to user basis. It will show the same score as the instance default if not modified, and can be set above or below the instance default to override that score on a per user basis. * **Delete User:** Using this button will delete the user and all of their information from the instance. This is a non recoverable action, the deletion is immediate and permanent. ### [](https://guide.cryosparc.com/application-guide/admin-panel#create-new-user) Create New User This form allows you to create a new user within the instance. The user will be immediately added to the instance and appear in the User Management table. They will need to be given the registration token code available in the table in order to set a password and gain access to their account. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fxkg9ZgNFgmJTxFe0dc2M%252Fv4-0-0-app-new-user.jpg%3Falt%3Dmedia%26token%3Dc453fad6-cdc5-4396-b0ac-fb2e770cc910&width=768&dpr=3&quality=100&sign=9bbeab45&sv=2) ### [](https://guide.cryosparc.com/application-guide/admin-panel#cluster-configuration) Cluster Configuration The Cluster Configuration page allows you to set custom variables for cluster job submission scripts at an instance-wide and per-target level. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FIou2lLN2QFZ5vm7rOWfu%252Fv4_1_cluster_configuration.png%3Falt%3Dmedia%26token%3Da90ea456-9cb4-4431-99db-8b902922424d&width=768&dpr=3&quality=100&sign=9c929832&sv=2) For more information, see: [Guide: Configuring Custom Variables for Cluster Job Submission Scripts](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/guide-configuring-custom-variables-for-cluster-job-submission-scripts) ### [](https://guide.cryosparc.com/application-guide/admin-panel#user-lane-restrictions) User Lane Restrictions Every user in CryoSPARC can be restricted to use only a subset of the lanes available in the instance. This page allows you to modify a user’s lane assignments. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fe848gguqLb4IvJxBp0Ji%252Fv4_1_lane_restrictions.png%3Falt%3Dmedia%26token%3D456c00c2-ecd0-4f6a-a5a4-545b0f6912eb&width=768&dpr=3&quality=100&sign=9e4f8d3&sv=2) For more information, see: [Guide: Lane Assignments and Restrictions](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/guide-lane-assignments-and-restrictions) [PreviousInstance Management](https://guide.cryosparc.com/application-guide/instance-management) [NextKeyboard Shortcuts](https://guide.cryosparc.com/application-guide/keyboard-shortcuts) Last updated 1 month ago * [Instance Settings](https://guide.cryosparc.com/application-guide/admin-panel#instance-settings) * [Instance Logs](https://guide.cryosparc.com/application-guide/admin-panel#instance-logs) * [User Management](https://guide.cryosparc.com/application-guide/admin-panel#user-management) * [Create New User](https://guide.cryosparc.com/application-guide/admin-panel#create-new-user) * [Cluster Configuration](https://guide.cryosparc.com/application-guide/admin-panel#cluster-configuration) * [User Lane Restrictions](https://guide.cryosparc.com/application-guide/admin-panel#user-lane-restrictions) --- # Dashboard | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/dashboard.md) . The cryoSPARC Dashboard provides at-a-glance information on your Projects, Workspaces and status of recent Jobs, as well as a change log documenting updates in new versions of cryoSPARC and links to helpful resources. The header and footer contain links to Projects, Workspaces, the Resource Manager and the identity of the current user. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNo_aA0KMM-5Njfj_Lg%252F-MNobWqGIIM0z08r3JRB%252FUI_dashboard_final_3.png%3Falt%3Dmedia%26token%3D3ac992d6-7790-4d98-a665-812ac43ed015&width=768&dpr=3&quality=100&sign=667b8655&sv=2) [Previousv3 User Interface Guide](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide) [NextProject and Workspace Management](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/project-and-workspace-management) Last updated 1 year ago --- # Waves as Vectors | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/cryo-em-foundations/image-formation/waves-as-vectors.md) . **Summary:** Waves can be represented by vectors. The length of the vector represents the wave's amplitude and the direction the vector points represents the wave's phase. The wave's oscillation can be represented by rotating the vector. [](https://guide.cryosparc.com/cryo-em-foundations/image-formation/waves-as-vectors#waves-as-vectors) Waves as vectors --------------------------------------------------------------------------------------------------------------------------- A wave can be described by its amplitude, frequency, and phase. Phase describes how a wave evolves through time. As a quantity, it generally only makes sense as a phase shift relative to some other wave. For instance, these two waves are shifted by a quarter of their wavelength. ![A single cycle of two sine waves is shown. They are shfited by pi/2 radians, so the peak of one wave lines up with another wave crossing zero.](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FnPTeLJat8esSoxXv7YjX%252Fphase-shift.png%3Falt%3Dmedia%26token%3D836fa0ec-2652-4393-888c-386bccd98b2c&width=768&dpr=3&quality=100&sign=41bd85e3&sv=2) We would describe this as a phase shift of 90° or, more typically, π/2 in radians. At first, the notion of describing a phase shift (which in this graph looks like a movement of the wave left or right) as a rotation may be confusing. It can be helpful to imagine these waves as three-dimensional helices viewed from the side, rather than 2D waves: ![At the top, a sine wave oscillates up and down. A white line is drawn across the zero point of the wave, and a white arrow points up or down to the value of the wave at the furthest-right point. Bottom left: the same wave is displayed as a helix, viewed from an oblique angle. The arrow points to the tip of the helix. Bottom-right: the helix is viewed directly down its helical axis. The wave now looks like a circle, with the arrow rotating smoothly in a circle.](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FeX140lGkqP741N9BNLFS%252Fsingle-wave.avif%3Falt%3Dmedia%26token%3Db2015970-7b28-4a3f-897e-15706fb1ed8d&width=768&dpr=3&quality=100&sign=af4a4bf8&sv=2) Here, we see that a sine wave can be modeled as the rotation of a vector (blue arrow, right of the animation) with a length equal to the wave’s amplitude at a rotational velocity of ω\=2πf\\omega = 2\\pi{}fω\=2πf, where fff is the wave’s frequency. Put another way, the speed of rotation represents frequency — a vector which spins faster traces out a wave which oscillates more frequently. Now, consider what happens when we rotate the vector by π/2: ![This animation is similar to the previous one, except a pink wave has been added with a pi/2 phase shift. In the bottom-right, the arrows make a right angle as they rotate in a circle.](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FcX8IKiwvKXGdfLwAFmjS%252Fupdated-two-phase.avif%3Falt%3Dmedia%26token%3Db0e92064-130a-4a70-8790-3e6ac903b461&width=768&dpr=3&quality=100&sign=f23f894d&sv=2) The pink wave appears to be shifted forward in time compared to the blue wave by a quarter of the wavelength. ### [](https://guide.cryosparc.com/cryo-em-foundations/image-formation/waves-as-vectors#adding-waves) Adding waves **Summary:** Adding two waves together can change their amplitude, phase, or both. The vector representation is especially useful when we begin to consider sums of waves rather than individual sine waves. For instance, it is an intuitive result that when we add two sine waves with the same frequency and phase together we get another wave of the same frequency and phase, but with a greater amplitude: ![Left: two sine waves of the same frequency and phase but differing amplitudes. Right: the waves added together have the same frequency but a greater amplitude.](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fel8oDXkVMC5CYFlm8UC3%252Fadding_simple_waves.png%3Falt%3Dmedia%26token%3D961b04e1-c109-45fe-bcbf-e49b79cfecbc&width=768&dpr=3&quality=100&sign=38e25d78&sv=2) Using the vector notation for this simple example, we observe the same behavior: ![An animation of the addition above. A short blue arrow and a short red arrow are rotating around their bases. When the waves are added, they form a longer line which is also rotating around its base.](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FfCAXSJvut2Mn443acyzj%252Fadding_two_sine.avif%3Falt%3Dmedia%26token%3D890222ad-5f62-4822-85c3-31b90c0489c3&width=768&dpr=3&quality=100&sign=9bff83f6&sv=2) Adding the two vectors together produces a vector pointing in the same direction, rotating at the same speed, but with a longer total length. In this case, the utility of thinking about adding waves this way may be unclear, but consider the following surprising result: ![Adding a large wave and a small wave with a pi/2 phase shift results in a wave which looks like a phase-shifted copy of the first wave.](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Ft9beRVNwCO9gMjgKAcdD%252Fadding_makes_phase_shift.png%3Falt%3Dmedia%26token%3D0685c923-434d-44c1-8f79-91aa40afbc91&width=768&dpr=3&quality=100&sign=3cf05b6a&sv=2) In this example, the red wave has the same frequency as the blue wave, but a much smaller amplitude and a π/2 phase shift. When we add together these two waves, we get the purple wave as a result — it looks like a phase-shifted version of the blue wave! This result is less surprising if we represent the waves as vectors instead: ![Adding a small wave with a pi/2 phase shift results in a vector which follows the hypotenuse of the triangle formed by the two waves. When the phase-shifted wave is small, this resulting wave has essentially the same magnitude as the original wave.](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F8sHfvyWfOvITbyVPJCLS%252Fadding_makes_phase_shift.avif%3Falt%3Dmedia%26token%3D65920941-8e1d-45a0-834b-6e3ce8a0e263&width=768&dpr=3&quality=100&sign=537221e&sv=2) Because the red vector is always pointing perpendicular to the blue vector, adding the two has the effect of rotating the blue vector with only a very modest effect on the final magnitude. Recalling that a rotation is equivalent to a phase shift, we have arrived at the same result as directly adding each point of the wave. The animation of vector rotation is helpful for developing a sense of what these vectors represent, but makes the figures cumbersome. For the rest of this guide, we will only draw the waves at some static position — implicitly, the vectors rotate as time passes or, equivalently, as we move through space. For instance, the above animation would be drawn like so: ![A right angle formed by three vectors, labeled Wave 1, Wave 2, and Result (hypotenuse).](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F1xFh8IVNsbdDNGw9eLYa%252Fsimplified-notation.png%3Falt%3Dmedia%26token%3Dd01e826b-cd59-4d1f-95c1-e28e223fd079&width=768&dpr=3&quality=100&sign=80c5d96a&sv=2) Using this method, it is clear that the resulting wave has approximately the same amplitude as wave 1 (precisely ∣Wave 1∣2+∣Wave 2∣2\\sqrt{|Wave\\ 1|^2 + |Wave\\ 2|^2}∣Wave 1∣2+∣Wave 2∣2​, where ∣Wave 1∣|Wave\\ 1|∣Wave 1∣ is the magnitude of Wave 1’s vector), but has a phase shift of arctan⁡∣Wave 2∣∣Wave 1∣\\arctan{\\frac{|Wave\\ 2|}{|Wave\\ 1|}}arctan∣Wave 1∣∣Wave 2∣​. If ∣Wave 2∣≪∣Wave 1∣|Wave\\ 2| \\ll |Wave\\ 1|∣Wave 2∣≪∣Wave 1∣, we can approximate the Result wave by shifting the phase of Wave 1 by the magnitude of Wave 2. This approximation is closely related to the Weak Phase Object approximation, covered in [Contrast in Cryo-EM](https://guide.cryosparc.com/cryo-em-foundations/image-formation/contrast-in-cryo-em) . [PreviousContrast in Cryo-EM](https://guide.cryosparc.com/cryo-em-foundations/image-formation/contrast-in-cryo-em) [NextAliasing](https://guide.cryosparc.com/cryo-em-foundations/image-formation/aliasing) Last updated 1 month ago * [Waves as vectors](https://guide.cryosparc.com/cryo-em-foundations/image-formation/waves-as-vectors#waves-as-vectors) * [Adding waves](https://guide.cryosparc.com/cryo-em-foundations/image-formation/waves-as-vectors#adding-waves) --- # Blueprints | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/application-guide/blueprints.md) . [](https://guide.cryosparc.com/application-guide/blueprints#introduction) Introduction ------------------------------------------------------------------------------------------- Blueprints allow you to customize parameter settings of individual job types in CryoSPARC, and then save those customizations as templates to be used later in any project or workspace. The saved customizations, called blueprints, become available right from the CryoSPARC job builder and quick actions menus and can be easily created, applied, edited, and exported/imported to another instance. As an example, if movies are frequently imported from the same microscope at the same magnification, an `Import Movies` blueprint could be created for that microscope to automatically set the Cs, pixel size, number of frames, electron dose, etc. At a high level, blueprints are meant to create an organizational space that can be accessed quickly in any project and workspace to deploy relevant jobs from a curated template. They remove the need to maintain workspaces full of separate template jobs and alleviate the act of searching across the instance for a job previously used to process similar data that yielded good results; these jobs can be scattered across projects and workspaces and searching for them can be difficult and time consuming. Instead, with blueprints, templates of jobs are only a click away in the job builder. [](https://guide.cryosparc.com/application-guide/blueprints#creating-a-blueprint) Creating a Blueprint ----------------------------------------------------------------------------------------------------------- A blueprint can be created either from a pre-existing job, or from a job type (eg. `Import Movies`, `3D Classification`, etc). When creating a blueprint from a pre-existing job the Create Blueprint dialog will have it’s custom parameters pre-populated with the custom parameters of that job. When creating from a job type, the dialog will have no pre-set custom parameters. ### [](https://guide.cryosparc.com/application-guide/blueprints#creating-from-a-pre-existing-job) Creating from a Pre-existing Job There are two ways to create a blueprint based on an existing job. First, by clicking the “Create Blueprint” option in the job’s quick access menu. Second, by selecting the job and clicking the “Create Blueprint” option in the job sidebar’s actions panel. Either of these options will open the Create Blueprint dialog. Similarly, with the job card selected, you can navigate to the job sidebar and open the actions panel by clicking the “Actions” button on the footer. You can select the “Create Blueprint” option in the panel to open the corresponding dialog. From here you can add a title - which will be the name of the blueprint shown in the job builder - and modify or add any parameters you would like to have applied when using your blueprint. Clicking the green “Create” button at the bottom of the dialog will create your new blueprint. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FLYUf7zIrf89wawHkXwyp%252Fv4-4-0-blueprints-create-1.gif%3Falt%3Dmedia%26token%3De9c15fe5-9d34-4c19-a95f-faffa5865bdb&width=768&dpr=3&quality=100&sign=ce35cc2e&sv=2) ### [](https://guide.cryosparc.com/application-guide/blueprints#creating-from-the-builder-sidebar-panel) Creating from the Builder Sidebar Panel You can also create a blueprint from scratch using the Job Builder. This will open the same creation dialog but with no custom parameters pre-populated. To do this, navigate to the Job Builder sidebar panel, find the job type from which you would like to create a blueprint, and use the “triple dot” overflow menu button to open the overflow menu; from here you can select the “Create Blueprint” option which will open the Create Dialog enabling you to build your blueprint. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FF3UXgG5sFBSKiTCxG48H%252Fv4-4-0-blueprints-create-2.gif%3Falt%3Dmedia%26token%3D5e27df96-8c86-4bf3-af00-118f0d7468af&width=768&dpr=3&quality=100&sign=5475f85b&sv=2) [](https://guide.cryosparc.com/application-guide/blueprints#applying-a-blueprint) Applying a Blueprint ----------------------------------------------------------------------------------------------------------- To apply a blueprint, simply navigate to any relevant workspace and either choose a blueprint you would like to use from the Job Builder, or select a currently building job and apply the blueprint to it. ### [](https://guide.cryosparc.com/application-guide/blueprints#applying-to-a-pre-existing-job) Applying to a Pre-existing Job Select the job you would like to apply the blueprint to, and using either the quick access menu or the job sidebar actions panel, select the “Apply Blueprint” option to open a submenu with relevant blueprint options for that job type. Selecting a blueprint will clear all of the pre-existing custom parameters (if any have been set) and apply all of the blueprint parameters. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FzNThVSw8qYyRR1WTQuiB%252Fv4-4-0-blueprints-apply-1.gif%3Falt%3Dmedia%26token%3D664ba9e0-be43-4a3f-86cb-b1d0577c446e&width=768&dpr=3&quality=100&sign=c0fd0dc0&sv=2) ### [](https://guide.cryosparc.com/application-guide/blueprints#applying-from-the-job-builder) Applying from the Job Builder Navigate into a workspace and then select the Job Builder sidebar panel. Find the job type that you would like to apply a blueprint from and open the blueprint drawer. The drawer can be opened by clicking on the down arrow beside the job type, or by navigating to it using the keyboard and pressing `shift` + `down arrow` to open the drawer. Select the blueprint you would like to apply. This will create a new job of the relevant type, and apply all of the blueprint parameters to it. From here it can have its inputs connected and be run as expected. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FMzNmXJhNS0C5ZooPI6JY%252Fv4-4-0-blueprints-apply-2.gif%3Falt%3Dmedia%26token%3De178e6f8-effa-4882-8de2-33e3e9e07826&width=768&dpr=3&quality=100&sign=7f418df1&sv=2) [](https://guide.cryosparc.com/application-guide/blueprints#modifying-a-blueprint) Modifying a Blueprint ------------------------------------------------------------------------------------------------------------- Blueprints cannot be edited or deleted unless you are the creator of the blueprint or an administrator. Modification options can be accessed in the Job Builder by navigating to a blueprint and clicking the “triple dot” button on the far right side to open the overflow menu. This will present options to edit or delete the blueprint. ### [](https://guide.cryosparc.com/application-guide/blueprints#editing-a-blueprint) Editing a Blueprint Selecting the “Edit” option from the overflow menu will open an Edit Blueprint dialog which, for all intents and purposes, is the same as the “Create Blueprint” dialog. Any and all parameters can be added, reset, and removed, and the blueprint will be permanently updated with the new parameters by pressing the “Update” button at the bottom of the dialog. ### [](https://guide.cryosparc.com/application-guide/blueprints#deleting-a-blueprint) Deleting a Blueprint Selecting the “delete” option from the overflow menu will open a confirmation popover which, upon confirming the deletion, will irreversibly remove the blueprint from the instance. [](https://guide.cryosparc.com/application-guide/blueprints#annotating-a-blueprint) Annotating a Blueprint --------------------------------------------------------------------------------------------------------------- Blueprints are meant to facilitate consistency and repeatability in creating jobs for a specific purpose. This often means that a Blueprint will have a variety of specific parameters set to particular values. In order to maintain intelligibility over the reasons for setting those parameter values, and even the parameters themselves, Blueprints includes a suite of annotation options. ### [](https://guide.cryosparc.com/application-guide/blueprints#job-details) **Job Details** Job details operate nearly identically to those found in the job builder. They allow you to apply annotations on a job by job basis by adding a title and/or description. When creating a Blueprint these fields will be pre-populated with any pre-existing titles or descriptions that the original job was given. An additional option to the far right of the field label allows you to check or uncheck an “Apply to Job” option. This option determines whether the Blueprint title and/or description will be added to jobs that the Blueprint is applied to. These options are checked by default. ### [](https://guide.cryosparc.com/application-guide/blueprints#parameter-details) **Parameter Details** Each parameter includes an option for annotation. This can be accessed by clicking the “Additional Options” toggle button to the far right of the parameter. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FXx2X2uYSeZK7TRguAMHc%252Fv4-4-0-blueprints-annotations-1.png%3Falt%3Dmedia%26token%3D44254ba5-ee6b-4e67-850a-b49a1a6dde84&width=768&dpr=3&quality=100&sign=e2c6a35a&sv=2) From here you can add any relevant notes about the parameter, for instance why it was set to the current value and/or in what situations it should be changed. The parameter note will be visible inside of the sidebar information tooltip for the relevant Blueprint when hovered. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fahp7HpnykAf8zFsO41YE%252Fv4-4-0-blueprints-annotation-2.png%3Falt%3Dmedia%26token%3Dea177e13-c5f9-4095-b794-269fb40321f7&width=768&dpr=3&quality=100&sign=c5949682&sv=2) [](https://guide.cryosparc.com/application-guide/blueprints#importing-exporting-a-blueprint) Importing / Exporting a Blueprint ----------------------------------------------------------------------------------------------------------------------------------- Blueprints, like workflows, are designed for portability. This means that they can be easily saved, stored, and shared between instances. Blueprints contain no identifying information of the instance that they were created in, and no references to the jobs that were used to create them. This allows blueprints to be a powerful tool for maintaining a catalog of your proprietary job templates, or cooperatively iterating on a processing approach agnostic of institution or instance. ### [](https://guide.cryosparc.com/application-guide/blueprints#exporting-a-blueprint) Exporting a Blueprint A blueprint can be exported by clicking on the the “triple dot” overflow button beside the individual blueprint you would like to export. Clicking on the “Export” option in the menu will download the blueprint to your device. The downloaded file is a `.json` (JavaScript Object Notation) file and can be easily inspected using any modern web browser. This compact file includes all of the necessary information to recreate a blueprint in any CryoSPARC instance (running on a version current to or greater than the introduction of the blueprints feature). ### [](https://guide.cryosparc.com/application-guide/blueprints#importing-a-blueprint) Importing a Blueprint Importing a blueprint can be done by clicking the “Import Blueprint” button on the footer of the Job Builder sidebar panel. This will open a device native file browser where you can find and upload a previously exported blueprint `.json` file. Once selected the file will be imported into your instance and will appear in the Job Builder sidebar panel like any other blueprint. The imported blueprint has no special properties outside of a `imported` attribute to demarcate it as created outside of the instance. The imported blueprint can be used, modified, and exported like any other blueprint. [](https://guide.cryosparc.com/application-guide/blueprints#summary) Summary --------------------------------------------------------------------------------- Blueprints are a template library for single jobs that allow you to consolidate useful template jobs in a single place where they can be easily created, applied, edited, and exported/imported agnostic of instance. They can be created from pre-existing jobs or from scratch using the Job Builder, and can be applied to pre-existing jobs or new ones. Blueprints can be modified by the creator or administrators, and can be exported and imported as `.json` files for portability. [PreviousFile Browser](https://guide.cryosparc.com/application-guide/file-browser) [NextWorkflows](https://guide.cryosparc.com/application-guide/workflows) Last updated 1 month ago * [Introduction](https://guide.cryosparc.com/application-guide/blueprints#introduction) * [Creating a Blueprint](https://guide.cryosparc.com/application-guide/blueprints#creating-a-blueprint) * [Creating from a Pre-existing Job](https://guide.cryosparc.com/application-guide/blueprints#creating-from-a-pre-existing-job) * [Creating from the Builder Sidebar Panel](https://guide.cryosparc.com/application-guide/blueprints#creating-from-the-builder-sidebar-panel) * [Applying a Blueprint](https://guide.cryosparc.com/application-guide/blueprints#applying-a-blueprint) * [Applying to a Pre-existing Job](https://guide.cryosparc.com/application-guide/blueprints#applying-to-a-pre-existing-job) * [Applying from the Job Builder](https://guide.cryosparc.com/application-guide/blueprints#applying-from-the-job-builder) * [Modifying a Blueprint](https://guide.cryosparc.com/application-guide/blueprints#modifying-a-blueprint) * [Editing a Blueprint](https://guide.cryosparc.com/application-guide/blueprints#editing-a-blueprint) * [Deleting a Blueprint](https://guide.cryosparc.com/application-guide/blueprints#deleting-a-blueprint) * [Annotating a Blueprint](https://guide.cryosparc.com/application-guide/blueprints#annotating-a-blueprint) * [Job Details](https://guide.cryosparc.com/application-guide/blueprints#job-details) * [Parameter Details](https://guide.cryosparc.com/application-guide/blueprints#parameter-details) * [Importing / Exporting a Blueprint](https://guide.cryosparc.com/application-guide/blueprints#importing-exporting-a-blueprint) * [Exporting a Blueprint](https://guide.cryosparc.com/application-guide/blueprints#exporting-a-blueprint) * [Importing a Blueprint](https://guide.cryosparc.com/application-guide/blueprints#importing-a-blueprint) * [Summary](https://guide.cryosparc.com/application-guide/blueprints#summary) --- # v3 User Interface Guide | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide.md) . [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide#dashboard) Dashboard ----------------------------------------------------------------------------------------------------- [Dashboard](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/dashboard) [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide#projects-and-workspaces) Projects and Workspaces --------------------------------------------------------------------------------------------------------------------------------- [Project and Workspace Management](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/project-and-workspace-management) [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide#jobs) Jobs ------------------------------------------------------------------------------------------- [Create and Build Jobs](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/create-and-build-jobs) [Queue Job, Inspect Job and Other Job Actions](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions) [View and Download Results](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/view-and-download-results) [Job Relationships](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/job-relationships) [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide#resource-manager) Resource Manager ------------------------------------------------------------------------------------------------------------------- [Resource Manager](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/resource-manager) [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide#user-management) User Management ----------------------------------------------------------------------------------------------------------------- [User Management](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/user-management) [PreviousFAQs and Troubleshooting](https://guide.cryosparc.com/live/faqs-and-troubleshooting) [NextDashboard](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/dashboard) Last updated 1 year ago * [Dashboard](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide#dashboard) * [Projects and Workspaces](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide#projects-and-workspaces) * [Jobs](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide#jobs) * [Resource Manager](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide#resource-manager) * [User Management](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide#user-management) --- # Managing a CryoSPARC Live Session from the CLI (v5.0+) | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli-v5.0.md) . In CryoSPARC version 5, CryoSPARC Live sessions can be created and controlled using [cryosparc-tools](https://tools.cryosparc.com/) . An [example](https://tools.cryosparc.com/examples/live-session.html) is provided on the _cryosparc-tools_ web site. [PreviousManaging a CryoSPARC Live Session from the CLI (≤v4.7)](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli) [NextFAQs and Troubleshooting](https://guide.cryosparc.com/live/faqs-and-troubleshooting) Last updated 1 month ago --- # Prerequisites and Compute Resources Setup | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup.md) . [](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup#hardware-configuration) Hardware Configuration --------------------------------------------------------------------------------------------------------------------------------- CryoSPARC Live is an independent application that is hosted alongside your main CryoSPARC instance. It works best when it has direct access to the files being written by the microscope that is actively recording images. Users can access the CryoSPARC Live interface directly from within the network via a web-browser, in the same way as interacting with the main CryoSPARC instance. See: [Access cryoSPARC Live](https://guide.cryosparc.com/live/how-to-access-cryosparc-live) . CryoSPARC Live automatically manages hardware resources available (GPUs) and deploys multiple preprocessing workers concurrently as well as dispatching reconstruction jobs to the main cryoSPARC instance transparently via the job scheduler. The compute workflow for CryoSPARC Live is detailed below. While the overall hardware and system requirements for CryoSPARC Live are similar to the prerequisites of the main CryoSPARC system (See: [Hardware and System Requirements](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements) ), **please read below on for detailed information on GPU requirements.** ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiq-1ak5jIBcarDGsP%252F-MNiqVrDHOmgTpzxao-t%252Fcsl_PR_1_1_cl_architecture.png%3Falt%3Dmedia%26token%3D93d41d85-35f0-4115-8c62-a9c9a00924b9&width=768&dpr=3&quality=100&sign=3a2bfc1c&sv=2) [](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup#system-requirements-and-compute-resources-terminology) System Requirements and Compute Resources Terminology ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **We recommend a minimum of 4 GPUs for a seamless experience and the ability to keep up with data collection.** There are three "task types" in CryoSPARC Live which correspond to the three "lane types", described below. On configuration of a particular Live session, you will need to tell Live which node(s) or "lanes" to use for each of these three types of tasks as well as how many GPUs to use in parallel for preprocessing. The compute resources allocated to these different task types can be configured on a per-session basis, can be adjusted during a session, and can be saved for future use via Configuration Profiles. 1. **Preprocessing jobs:** CryoSPARC Live Worker (motion correction, CTF estimation, thumbnail generation, particle picking and particle extraction all in one) 2. **Reconstruction jobs:** Streaming 2D Classification and Streaming 3D Refinement 3. **Auxiliary jobs:** Which are optional or transient (e.g., 2D Classification used for template creation for the template picker and Ab-Initio Reconstruction) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FOpxy6YDy1AXfR2pfKEJh%252Fimage.png%3Falt%3Dmedia%26token%3D36684fef-7b41-4650-8ea9-d460a3f6a124&width=768&dpr=3&quality=100&sign=868a893c&sv=2) Compute Resources section on the Configuration Tab ### [](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup#preprocessing-lane) Preprocessing Lane **Minimum requirement: 1 GPU** The `Number of Preprocessing GPU Workers` **(minimum one)** selected here will determine how many CryoSPARC Live workers will be spawned. Live workers perform preprocessing of incoming movies, including motion correction, CTF estimation, thumbnail generation, particle picking, and particle extraction. Each CryoSPARC Live worker will allocate one GPU to its main process and use this continuously unless the worker is killed (by way of pausing the Session). The number of preprocessing workers can be changed by modifying this value. The new number of workers will be spawned once the Session is paused, then started again. **CPU Memory Bandwidth** CryoSPARC Live's preprocessing step required fast copies of movie data from disk to the GPU. To achieve this, configure your system with a high memory bandwidth CPU **(>100 GB/s)**. Slower CPUs will still work but may not achieve optimal throughput. #### [](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup#session-priority) Session Priority The `Session Priority` set here will determine the priority of spawned CryoSPARC Live Workers in the job queue. See [Guide: Priority Job Queuing](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/tutorial-priority-job-queuing) for more information abut job priority. ### [](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup#reconstruction-lane) Reconstruction Lane **Minimum requirement: 2 GPUs** CryoSPARC Live will automatically schedule a Streaming 2D Classification and Streaming Refinement job when requested on the lane selected here. Each reconstruction job (Streaming 2D Classification, Streaming 3D Refinement) will allocate one GPU to its main process and use this continuously as new particles arrive. Therefore, the minimum required GPUs for this lane is **two**. In the case where the lane selected here does **not** have more than one GPU available when the job is queued, the CryoSPARC scheduler will still automatically schedule reconstruction jobs to the lane as resources become available. Since these jobs hold resources indefinitely, any other `queued` jobs that are waiting for resources in this lane may remain indefinitely in the queue until the Session is paused or the reconstruction job is manually stopped. The GPUs in this lane are only used once the streaming 2D/3D jobs are started. You can opt, during a Live collection session, to not start these streaming jobs until some time into data collection. During this time, the GPUs are free, and can be used for other phases (eg., Preprocessing or Auxiliary jobs - see below). ### [](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup#auxiliary-jobs-lane) Auxiliary Jobs Lane **Minimum: 1 GPU** Auxiliary jobs in CryoSPARC Live include 2D Classification jobs (for generating templates for the template picker), Create Template jobs (for generating templates from volumes for the template picker) and Ab-Initio jobs (for initial model creation). Each job allocates one GPU to its main process. The minimum required GPUs for this lane is **one**. You can use the same lane for auxiliary jobs as for reconstruction even if there are only 2 GPUs available in the lane, as long as you are careful that the streaming reconstruction jobs are not already started when auxiliary jobs need to run (otherwise these jobs will be indefinitely queued). Most of the time, this is not an issue as template generation generally precedes streaming 2D classification and ab-initio reconstruction precedes streaming 3D refinement. ### [](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup#how-much-gpu-memory-do-i-need) How much GPU memory do I need? **The** _**recommended**_ **amount of GPU memory required is at minimum 11GB per GPU** to be able to process most types of data successfully in CryoSPARC. For minimum GPU requirements per job type/data type, see the table below. #### [](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup#minimum-gpu-memory-requirements) Minimum GPU Memory Requirements Preprocessing Reconstruction Auxiliary Gatan K2 Images, TFS Falcon 3 Images: **8GB+** Gatan K2 Super Resolution, Gatan K3, Gatan K3 Super Resolution, TFS Falcon 4 Images TFS Falcon 4 EER Images: **11GB+** 2D Classification: **4GB+** 3D Refinement [(heavily dependent on box size)](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements#graphical-processing-units-gpus) : **11GB+** 2D Classification: **4GB+** Ab-Initio Reconstruction: **8GB+** [](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup#fewer-than-4-gpus) Fewer than 4 GPUs? ------------------------------------------------------------------------------------------------------------------------ **We recommend a minimum of 4 GPUs** so that you can allocate two to preprocessing and two to reconstruction or auxiliary tasks for a seamless experience and the ability to truly keep up with a data collection session - i.e., be able to generate 3D structures while still collecting data. If 4 GPUs are not available or you are using CryoSPARC Live on previously collected data (i.e., not "live" during a data collection session), it is possible to reduce the compute requirement somewhat. **Preprocessing** can use 1 or more GPUs, scaling linearly in terms of throughput. **Reconstruction** uses one GPU for 2D classification and one GPU for 3D refinement, which run indefinitely in streaming mode, continuously taking in new particles and updating the 2D/3D results. Both 2D streaming classification and 3D streaming refinement can be started part-way into the session (e.g., an Ab-Initio model needs to be first created before refinement can begin). **Three GPUS** will also be able to perform all the functions, but depending on your data collection speed, having only one GPU for preprocessing may not be sufficient to keep up with your camera. With more GPUs, the preprocessing can become correspondingly faster. **With two GPUs**, you can still achieve many of the benefits of Live processing but you will need to manually switch between using 2 GPUs for preprocessing **/** 1 GPU for preprocessing + 1 GPU for 2D classification **/** 1 GPU for preprocessing + 1 GPU for 3D refinement. Therefore, the results will not be updating in a Live streaming fashion, but only periodically as these switches are made. [](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup#run-configuration) Run Configuration ----------------------------------------------------------------------------------------------------------------------- New in CryoSPARC v5.0+, the Run Configuration section allows setting parameters related to how the session will start, priority of processing, and whether the session will auto pause. For complete details, see [this section](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#run-configuration) . ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FyVTUn3r6cnYQJH8T2X0r%252Fimage.png%3Falt%3Dmedia%26token%3D13a46006-78c6-4283-a2dc-ed7e9dd3cb26&width=768&dpr=3&quality=100&sign=e4d1c6c8&sv=2) Run Configuration section on the Configuration tab [](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup#using-a-cluster-considerations) Using a Cluster - Considerations --------------------------------------------------------------------------------------------------------------------------------------------------- Users can select a cluster lane for any of the three processing lanes in CryoSPARC Live. Some considerations to keep in mind when using a cluster include resource availability and quality of service limitations. When you start a Live session, preprocessing workers are immediately dispatched to the queuing system to be launched. If you are using a cluster, the scheduling and launching of jobs is taken care of by the cluster scheduler, whose timings may be unpredictable due to availability of resources at queue time. This increases the latency of starting your Live session, and we believe it impacts the experience of CryoSPARC Live. Some cluster management systems, especially those serving multiple users, **for example in a university,** also have QOS’s (quality of service policies) in place, which limit how long a single process can use resources. Due to the nature of CryoSPARC Live, the preprocessing, streaming 2D classification and streaming 3D refinement workers run indefinitely until the session is “paused”. See: [Session Functions](https://guide.cryosparc.com/live/live-jobs-and-session-level-functions) . If your cluster has these policies, ensure you are keeping track of your running jobs by either using the built-in resource manager in CryoSPARC, or the cluster scheduler’s CLI. You may also choose to circumvent these policies if your cluster supports dedicating entire worker nodes. If this is the case, you may be able to use a dedicated node to run CryoSPARC Live jobs by configuring it as a normal worker node in CryoSPARC. More details on how add worker nodes can be found in the general installation instructions ('Worker Node'): [Downloading and Installing cryoSPARC](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/downloading-and-installing-cryosparc) [PreviousAbout CryoSPARC Live](https://guide.cryosparc.com/live/about-cryosparc-live) [NextHow to Access CryoSPARC Live](https://guide.cryosparc.com/live/how-to-access-cryosparc-live) Last updated 5 months ago * [Hardware Configuration](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup#hardware-configuration) * [System Requirements and Compute Resources Terminology](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup#system-requirements-and-compute-resources-terminology) * [Preprocessing Lane](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup#preprocessing-lane) * [Reconstruction Lane](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup#reconstruction-lane) * [Auxiliary Jobs Lane](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup#auxiliary-jobs-lane) * [How much GPU memory do I need?](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup#how-much-gpu-memory-do-i-need) * [Fewer than 4 GPUs?](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup#fewer-than-4-gpus) * [Run Configuration](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup#run-configuration) * [Using a Cluster - Considerations](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup#using-a-cluster-considerations) --- # Job Relationships | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/job-relationships.md) . [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/job-relationships#view-job-relationships) View job relationships ------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/job-relationships#card-view) Card view You can easily see how jobs relate to each other by clicking on the job of interest (e.g., J6) in the card view. The parent(s) of the selected job, J5 (i.e., the job which provided inputs) are highlighted with a dashed red line, while the child J7 (i.e., the job which used inputs from J6) is highlighted with a dashed green line. Other un-related jobs fade away until you click somewhere else on the Workspace canvas. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNp8K5OFfZFAubyFm-9%252F-MNpEJbiEbX8m79MpAUd%252FUI_parent_child_card_view_1.png%3Falt%3Dmedia%26token%3D9ce15180-c1e5-4800-b8cc-b4eac38138f6&width=768&dpr=3&quality=100&sign=f3d01ba1&sv=2) ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/job-relationships#tree-view) Tree view You can also view an interactive tree of all the jobs in a Project or Workspace. This makes it extremely easy to visualize your processing journey and understand how outputs of one job flow into the inputs of another. Switch between the cards view and the tree view using the buttons at the top of any Project or Workspace. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNp8K5OFfZFAubyFm-9%252F-MNpEscNuHJP2kSN-ifT%252FUI_tree_view_button_1.png%3Falt%3Dmedia%26token%3D9341a3a0-040a-4895-a1ac-1e9033d81a22&width=768&dpr=3&quality=100&sign=f2161f6d&sv=2) The tree view displays all jobs and their connections. Click and drag to move the tree around. Hold `ALT` and scroll to zoom in and out of the tree. You can also scroll up and down or left and right using a trackpad or mouse wheel. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNp8K5OFfZFAubyFm-9%252F-MNpFEWlCWR4EAxDimFj%252FUI_tree_view_zoom_1.png%3Falt%3Dmedia%26token%3Df0058d20-566c-4976-ac22-3ee02e0e04c5&width=768&dpr=3&quality=100&sign=1b2ce556&sv=2) All jobs are connected with each other, except for jobs in 'Building' status (purple) that have not been connected to anything, which will remain beside the tree on the right side until they are connected by dragging inputs. Clicking on a particular job card in Tree View will highlight its parent and children jobs, and fade out other jobs in tree, until you double-click elsewhere on the workspace, or click `ESC` to de-select the job: ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/job-relationships#interact-with-jobs) Interact with jobs You can interact with jobs in the card view and the tree view in exactly the same manner: * Click on the job number to open the job card (inspect view). Alternatively, press the `SPACEBAR` while the job of interest is selected. Exit inspect view by clicking the `x` or pressing the `SPACEBAR` again. * Toggle between active and inactive Building states by pressing `b` while a Building job is selected, or clicking on the Building button on the job card * Open the Job Builder to drag and drop inputs from job cards [PreviousView and Download Results](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/view-and-download-results) [NextResource Manager](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/resource-manager) Last updated 1 year ago * [View job relationships](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/job-relationships#view-job-relationships) * [Card view](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/job-relationships#card-view) * [Tree view](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/job-relationships#tree-view) * [Interact with jobs](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/job-relationships#interact-with-jobs) --- # Job: Simulate Data (Legacy) | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-legacy.md) . [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-legacy#description) Description -------------------------------------------------------------------------------------------------------------------------------------------- This job generates simulated particles from an input volume by projecting, CTF corrupting, and adding noise. This is an advanced job type that is not intended for common workflows. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-legacy#input) Input -------------------------------------------------------------------------------------------------------------------------------- * a volume with `map` result [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-legacy#output) Output ---------------------------------------------------------------------------------------------------------------------------------- * particles with `blob`, `ctf`, and `alignments3D` [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-legacy#undefined) ----------------------------------------------------------------------------------------------------------------------------- [PreviousJob: Simulate Data (GPU)](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-gpu) [NextAutomated Workflows](https://guide.cryosparc.com/processing-data/automated-workflows) Last updated 1 month ago * [Description](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-legacy#description) * [Input](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-legacy#input) * [Output](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-legacy#output) * [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-legacy#undefined) --- # Simulations | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations.md) . [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations#simulation-jobs) Simulation Jobs --------------------------------------------------------------------------------------------------------------------------- [Job: Simulate Data (GPU)](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-gpu) [Job: Simulate Data (Legacy)](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-legacy) [PreviousJob: Orientation Diagnostics](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-orientation-diagnostics) [NextJob: Simulate Data (GPU)](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-gpu) Last updated 1 month ago --- # Queue Job, Inspect Job and Other Job Actions | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions.md) . [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#queue-job) Queue Job ------------------------------------------------------------------------------------------------------------------------------------------------- Once you have built a job and are ready to run the job, click 'Queue' on the Job Builder. This will bring up the Queue Modal. 1\. Select the lane where the job should run. By default, new jobs you create will run in the current active workspace. If you wish to run the job and link it into a different workspace, you can also specify this under 'Run Job in'. 2\. Optionally, you can set the Job Priority. See: Priority Queuing. 3\. Click 'Create' to run the job. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNp-GhMnk7S_4i012N3%252F-MNp-u7Itn17iRoJX-F1%252FUI_queue_modal_1.png%3Falt%3Dmedia%26token%3Da430e57e-2981-4b57-ba93-24c03e875836&width=768&dpr=3&quality=100&sign=a23af70b&sv=2) ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#queue-a-chain-of-jobs-that-will-run-automatically) Queue a chain of jobs that will run automatically You can queue up a chain of jobs, which will each commence as soon as their respective required inputs (i.e., the outputs from jobs earlier in the series) become available. For example, while an Import Movies job is running, open the job card, and drag and drop the `imported-movies` output into a new Patch Motion Correction job in the Job Builder. Queue the Patch Motion Correction job and it will appear in "Queued - waiting because inputs are not ready" state until the imported movies become available, at which point it will start running automatically. See also, [Management and Monitoring](https://github.com/cryoem-uoft/guide-beta/blob/master/setup-configuration-and-management/management-and-monitoring-4.7) if you wish to create and launch jobs through the command line interface. [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#view-job-status) View Job Status ------------------------------------------------------------------------------------------------------------------------------------------------------------- You can easily determine the status of jobs within the workspace view looking at the coloured dots on the job card. Purple = building, teal = queued, gray = started, blue = running, green = completed, orange = killed and red = failed. Hovering over a job's status circle will indicate the status as a tool tip. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNp3PaKV_kHNiPD8uih%252F-MNp3wCgTFroCGJFT602%252FUI_job_status_completed_1.png%3Falt%3Dmedia%26token%3Dfe3a487d-1741-4526-a438-670aa1b35257&width=768&dpr=3&quality=100&sign=75473604&sv=2) [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#inspect-job) Inspect Job ----------------------------------------------------------------------------------------------------------------------------------------------------- Once the job starts to run, you can view its progress and intermediate outputs at any time from the inspect view. ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#open-inspect-view) Open inspect view To open the inspect view, click on the job number (at the top right of the job card), or select the job and press `SPACEBAR`. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNp-GhMnk7S_4i012N3%252F-MNp12M73peknvR4WUdJ%252FUI_running_job_1.png%3Falt%3Dmedia%26token%3Df75fcefb-be55-4dc1-98a1-260ef4dd5cf1&width=768&dpr=3&quality=100&sign=3cfb9a2c&sv=2) The job inspect view has several tabs, including an 'Overview' or stream log, 'Inputs and Parameters', 'Outputs' and 'Metadata'. To navigate easily through the stream log without having to scroll, select a checkpoint from the top, or use the filters. Clicking on "Follow latest" will cause the streamlog to tail the end of the job's output as it runs in real time. You can also filter by image, plots, etc. Press `SPACEBAR` again or use the `x` on the top right of the inspect view, to close it. ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#view-parameters-that-were-modified) View parameters that were modified To quickly view which parameters you adjusted for a given job, open the inspect view and click on the 'Inputs and Parameters' tab. Parameters which were unchanged from their default values display with a blue **D**. Parameters you specified are outlined in green and display with a green **S**. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNoYrg6xIQPUH7z0qeM%252F-MNoZ2-gLJhmC5Dpbz5t%252FJB_5_v2_viewspecparams.png%3Falt%3Dmedia%26token%3D3ebaa5f2-e4a9-47ad-9afa-b55ed35bcfc8&width=768&dpr=3&quality=100&sign=90b75efb&sv=2) ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#interact-with-outputs-and-result-groups) Interact with outputs and result groups Each output in cryoSPARC jobs contains "low level" result groups that may be useful for advanced processing strategies. For a detailed overview of outputs and passthrough results, please see: [Inputs and Outputs in cryoSPARC](https://guide.cryosparc.com/processing-data/general-tutorials/job-builder-tutorial#inputs-and-outputs-in-cryosparc) [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#other-job-actions) Other Job Actions ----------------------------------------------------------------------------------------------------------------------------------------------------------------- ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNp4hKxX7_Bqn2GEs90%252F-MNp6_MTNG_Zq2BRUQze%252FUI_job_actions_all_1.png%3Falt%3Dmedia%26token%3D139afee0-5a04-416c-96af-1a9164176162&width=768&dpr=3&quality=100&sign=abd62f37&sv=2) ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#link-job) Link Job To avoid redundant processing, it's possible to link jobs in multiple workspaces to continue processing in a new or existing workspace. Select a job and click on 'Link Job' the Job Details panel to select another Workspace in the same Project where you would like to link the job. Any changes you make to the job in one workspace will be reflected in the other, as both workspaces will be displaying the exact same job. If a linked job is deleted from any workspace, all other instances of it in other workspaces will also be deleted. You can also unlink a job from a workspace if needed. ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#move-job) Move Job When re-organizing your workspaces, you may find it necessary to move a job from one workspace to another. To do so, select the job and navigate to the 'Details' tab. Here, you can click the Move Job dropdown to select which workspace you would like to move the job to. The job will disappear from its original workspace and move to the new one. ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#kill-job) Kill Job You can kill a job while it is running. A pop-up message will allow you to confirm. Killed jobs will display as orange and will remain part of the workspace unless you delete them. Killed jobs will not have their intermediate results expunged until they are cleared. ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#clear-job) Clear Job When a job has been killed, has failed, or has been completed, it can be cleared, meaning that its results and outputs are all erased from the file system, but its connections to other jobs and parameters are retained. This makes it easy to correct a mistakenly set parameter, or to re-run a failed job. Clearing also removes queued jobs from the queue and resets jobs to Building status. ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#clear-intermediate-results) Clear Intermediate Results Unused intermediate results can be cleared from iterative jobs to save space. This action can be executed at a job level (by clicking the "Clear Intermediate Results" button on the Job Details panel) or at a project level for every job (by clicking the "Clear Intermediate Results" button on the Project Details panel). This function will remove all unused outputs created by iterative jobs that save raw data at every iteration. Final results for every result slot will be retained, whether they have been used elsewhere or not. ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#export-job) Export Job Any individual job can be exported, for sharing, manipulation, or archiving. Jobs must be exported manually in order to create a "consolidated" exported-job directory which is then importable. For more details, please see: [Exporting a job](https://guide.cryosparc.com/processing-data/general-tutorials/tutorial-data-management-in-cryosparc#5-ability-to-export-and-import-individual-jobs) ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#clone-job) Clone Job Cloning is particularly useful when you wish to process subsequent jobs in a new workspace for better organization of your experiment, or if you wish to quickly replicate a job (to try another setting of parameters) without dragging and dropping inputs into the Job Builder. Select the job you wish to clone, and from the 'Details' tab, click 'Clone job'. By default, the job will be cloned in the current workspace. Click 'Queue' to launch the job. A modal will appear asking whether you wish to run the job in the current workspace or in a new workspace, and you can select as appropriate. Note that you can always edit the parameters of a new cloned job before Queuing. You can also change inputs by removing connected inputs and dragging in new inputs. ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#mark-job-as-complete) Mark Job as Complete This option allows you to mark failed or killed jobs in cryoSPARC as `completed` in order to allow their latest outputs to be used for further processing. For more details, please see: [Data Management in cryoSPARC](https://guide.cryosparc.com/guides-for-v3/tutorial-data-management-in-cryosparc) ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#delete-job) Delete Job While selected, you can delete a job from the 'Details' tab. A pop-up message will ask you to confirm the delete. Once deleted, the job will disappear from the workspace (except in the tree view, if it has children that need to be displayed). Note that job numbers within a workspace are unique, so a new job in the same workspace will not be assigned the same number as a previously run or previously deleted job. Deleted jobs are first cleared before deleting, meaning that their intermediate and final results are erased from disk. [PreviousCreate and Build Jobs](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/create-and-build-jobs) [NextView and Download Results](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/view-and-download-results) Last updated 1 year ago * [Queue Job](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#queue-job) * [Queue a chain of jobs that will run automatically](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#queue-a-chain-of-jobs-that-will-run-automatically) * [View Job Status](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#view-job-status) * [Inspect Job](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#inspect-job) * [Open inspect view](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#open-inspect-view) * [View parameters that were modified](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#view-parameters-that-were-modified) * [Interact with outputs and result groups](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#interact-with-outputs-and-result-groups) * [Other Job Actions](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#other-job-actions) * [Link Job](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#link-job) * [Move Job](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#move-job) * [Kill Job](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#kill-job) * [Clear Job](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#clear-job) * [Clear Intermediate Results](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#clear-intermediate-results) * [Export Job](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#export-job) * [Clone Job](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#clone-job) * [Mark Job as Complete](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#mark-job-as-complete) * [Delete Job](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions#delete-job) --- # UI Overview | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/live/ui-overview.md) . [](https://guide.cryosparc.com/live/ui-overview#browse-sessions-page) Browse Sessions Page ----------------------------------------------------------------------------------------------- From this page, you can view details of existing Sessions including current processing status and create new Sessions. The Browse Sessions page contains four components: 1. The **header** contains the main menu, toggle between the browse view and data management view, and button to create a new session 2. The **projects** list on the left side of the screen shows a list of all projects that contain Live sessions 3. The **session list** is shown in the middle of the screen and shows a card for each session. Clicking on a session will show more information in the sidebar 4. The **sidebar** on the right side of the screen populates with detailed information of the selected session There are many options available to filter sessions, such as selecting a status, what project it belongs to, and who created the session. To open an existing session, click ‘View Session’. To create a new session, please see: [New Live Session: Start to Finish Guide](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#1.-create-new-session) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FH0Xp7igNVZL3gzpKebvw%252Fv4-0-0-app-browse-system-session-view.png%3Falt%3Dmedia%26token%3D67d8cceb-f895-405e-bf40-0746582e8a1e&width=768&dpr=3&quality=100&sign=a8178f21&sv=2) [](https://guide.cryosparc.com/live/ui-overview#live-session-layout) Live Session Layout --------------------------------------------------------------------------------------------- ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNzXvuSVaxFb4QqotKP%252F-MNz_seLLuIWcC6mfd9f%252FCSL_UI_2.png%3Falt%3Dmedia%26token%3Db7da2191-4619-4ea5-9565-dbd9ea05a6e1&width=768&dpr=3&quality=100&sign=f7e1888b&sv=2) CryoSPARC Live User Interface ### [](https://guide.cryosparc.com/live/ui-overview#id-1.-header) 1\. Header ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNzXvuSVaxFb4QqotKP%252F-MNz_vhKS3Gi5m0yrniL%252FCSL_UI_3.png%3Falt%3Dmedia%26token%3D7f0b5c8f-dcf2-480e-8c81-fe2db05fca75&width=768&dpr=3&quality=100&sign=88b511e1&sv=2) The Header contains information about the session status, session and project title, as well as session-level functions (Start, Pause, Mark as Complete, etc) and the elapsed time. The Main Menu on the top left contains options for navigating back to the Browse Sessions page, as well as session settings and logout. ### [](https://guide.cryosparc.com/live/ui-overview#id-2.-sidebar) 2\. Sidebar The Sidebar is the main way to navigate through a Live session from start to finish and also contains detailed statistics for each processing stage, e.g., the number of exposures found and processed, the number of particles extracted, etc. Click on a sidebar item to navigate to the corresponding tab. Sidebar items can be collapsed or expanded. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNzXvuSVaxFb4QqotKP%252F-MNz_xu-maOgStX2cZNy%252FCSL_UI_4.png%3Falt%3Dmedia%26token%3D82e55cb3-ee40-4e93-b722-f341740b8a18&width=768&dpr=3&quality=100&sign=d1feaa29&sv=2) You can collapse various tabs to show or hide certain statistics ### [](https://guide.cryosparc.com/live/ui-overview#id-3.-exposure-feed) 3\. Exposure Feed As exposures are captured, detected, and loaded into CryoSPARC Live, a visualization/thumbnail for each appears sequentially from left to right in the top bar. Near the bottom of each thumbnail, a blue progress bar indicates the status of pre-processing (motion correction, CTF estimation, picking, extraction, etc). Rejected exposures are indicated with a red “X”. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNzXvuSVaxFb4QqotKP%252F-MNza2SEcOx386NXSdUT%252FCSL_UI_5.png%3Falt%3Dmedia%26token%3D036a8397-6110-4bc4-ad92-c279823a403f&width=768&dpr=3&quality=100&sign=9207f178&sv=2) ### [](https://guide.cryosparc.com/live/ui-overview#id-4.-exposure-navigation) 4\. Exposure Navigation ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNzXvuSVaxFb4QqotKP%252F-MNza5CrVKOo1HTxgmGj%252FCSL_UI_6.png%3Falt%3Dmedia%26token%3D939cc1f3-a6dc-4869-a48d-50f898f6cb62&width=768&dpr=3&quality=100&sign=de323cd&sv=2) The arrows allow for traversing through exposures including the ability to follow the latest exposure, or navigate to a specific exposure using its ID. ### [](https://guide.cryosparc.com/live/ui-overview#id-5.-data-processing-tabs-e.g.-individual-exposure-overview-picking) 5\. Data Processing Tabs (e.g., Individual Exposure, Overview, Picking) Each tab allows for configuration or inspection of a data processing stage. The Configuration tab contains session information, notes, hardware selections and parameters, with the option to trigger re-processing of previously processed data if parameters are changed. The Picking tab contains the ability to choose between the three available pickers, set, test and apply parameters and thresholds. More details can be found in: How to Start to Finish. ### [](https://guide.cryosparc.com/live/ui-overview#id-6.-exposure-volume-viewer) 6\. Exposure / Volume Viewer ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNzXvuSVaxFb4QqotKP%252F-MNzaUbgN-881Moq5r68%252FCSL_UI_7.png%3Falt%3Dmedia%26token%3Dfc9f0a54-98f7-4c8d-a0da-f91b0a387db4&width=768&dpr=3&quality=100&sign=ae756499&sv=2) The selected exposure (highlighted with a green border in the Exposure Feed) is expanded in the Exposure / Volume Viewer, allowing zooming/panning and filtering of the micrograph, display of motion trajectories, and CTF plots/Thon ring diagrams associated with the exposure. Particle picks associated with the exposure are shown (colours correspond to the different picking tools available in cryoSPARC Live) and manual picking can be done directly on the micrograph. Once a 3D Volume is available (following Ab-Initio Reconstruction or Streaming 3D Refinement), the volume can be examined in the same viewer. You can set a custom threshold value using the slider or input on the top-right of the viewer. You can also download the currently displayed volume using the button on the bottom-right of the viewer. There are several controls available to manipulate the volume: zoom by using the scroll wheel on a mouse or trackpad, rotate via holding the left mouse button and dragging across the canvas, pan via holding the right mouse button and dragging across the canvas. You can pause or play the rotation animation via the button on the bottom-right of the viewer. #### [](https://guide.cryosparc.com/live/ui-overview#exposure-viewer-ruler-and-scale-bar) Exposure Viewer Ruler and Scale Bar ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FLZRJsiB0nv40JsvsLj3i%252Fexposure-viewer-ruler-scale.gif%3Falt%3Dmedia%26token%3D7870f50b-2edf-4a96-9c08-5d00fd6ee91b&width=768&dpr=3&quality=100&sign=578a321&sv=2) The ruler and scale bar allow for easy size reference of an exposure and particles. The scale bar appears in the top right of the exposure viewer and shows a size reference in nanometers relative to the current zoom level of the exposure. It reactively scales while zooming and will change colour to maintain contrast over the exposure. The ruler tool can be selected in the exposure controls menu on the bottom right of the exposure viewer. When clicking and dragging on the exposure with this tool selected, a circle will be drawn emanating from the initially selected point with both its diameter in pixels and angstroms relative to the current scale displayed. ### [](https://guide.cryosparc.com/live/ui-overview#id-7.-footer) 7\. Footer ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNzXvuSVaxFb4QqotKP%252F-MNzaXZtpujNfQt_3osV%252FCSL_UI_8.png%3Falt%3Dmedia%26token%3D4ffaf8a8-da31-4ba3-afa8-d743ebf89caa&width=768&dpr=3&quality=100&sign=76459b9e&sv=2) The Footer contains information and links to active Live session jobs. Clicking on a job will open its event log via the Live interface. [](https://guide.cryosparc.com/live/ui-overview#navigating-the-session) Navigating the Session --------------------------------------------------------------------------------------------------- There are nine tabs within a CryoSPARC Live session that contain all the controls needed, from configuration to inspecting and filtering exposures, particles and 2D classes and attaining 3D reconstructions from selected particles. The currently selected tab is highlighted in blue and displays an arrow to the right. To view a particular tab, simply click on the name within the navigation sidebar. Underneath the title of each tab are statistics computed by CryoSPARC such as the number of processed exposures, particles extracted and selected for classification, and results of the reconstruction stages. You can choose to show or hide the statistics of each tab by clicking on the expand or collapse button at the right of each tab. Choose to hide or show all details by holding shift and clicking the toggle button for any tab. ### [](https://guide.cryosparc.com/live/ui-overview#details-tab) **Details Tab** ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNzXvuSVaxFb4QqotKP%252F-MNza_6ClQhXYSthf_bN%252FCSL_UI_9.png%3Falt%3Dmedia%26token%3Da3081752-b43d-4697-88eb-0a1a748e870c&width=768&dpr=3&quality=100&sign=f10fd61&sv=2) The details tab shows a high-level overview of the session including a complete processing history. Various actions are available, including the ability to export exposures or particles. You can also provide notes for the session, which is particularly helpful when looking back at a later date. ### [](https://guide.cryosparc.com/live/ui-overview#configuration-tab) **Configuration Tab** ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNzXvuSVaxFb4QqotKP%252F-MNzacDYzGqdzACIVonY%252FCSL_UI_10.png%3Falt%3Dmedia%26token%3D21e1f139-ed2c-4030-9666-14f6af6ade2f&width=768&dpr=3&quality=100&sign=6942c71c&sv=2) The configuration tab automatically loads when a session is first created, but you can navigate back to it at any time. It contains three panels: 1. The first panel shows an overview of the compute resources selected to run the session. Live runs on the same processing lanes configured for CryoSPARC. You can select what compute resources to use for preprocessing and reconstruction stage, along with any auxiliary processing that Live performs. It's possible to adjust the number of GPU workers allocated for the preprocessing stage throughout the lifecycle of the session. For example, at the start of the session, you can allocate four GPUs to quickly extract particles from exposures, then lower the number of workers to two or one when resources are needed for the reconstruction stage 2. The second panel contains all parameters available to configure. There are a few required parameters that must be provided before a session can start, such as the microscope and camera details. You can view advanced parameters via a toggle and use the text input to quickly filter through the list. Once you change one or more parameters, you have the chance to apply the changes to all exposures in the session or only future exposures. This is helpful when deciding to change parameters while the session is currently processing exposures. Upon clicking apply, Live will automatically determine the best way to reprocess exposures based on the new configuration 3. The third panel within the configuration tab is used to configure exposure groups. Exposure groups specify where to find files that you'd like to process. You can provide additional options such as a gain reference file for each exposure group. Multiple exposure groups can be added in a session, and they can be used for a variety of purposes. For example, you can choose to configure one exposure group per grid, or an exposure group for each beam-shift position in a template. After enabling an exposure group, you can choose to continuously listen to the specified directory and filename wildcard filter for new exposures, or ignore the group if you no longer want to include a set of exposures for processing. To speed up the configuration a new Live session, you can save all or some of the parameters of an existing session into a configuration profile. These profiles are available for all users to apply to current or future sessions. To save your current configuration, click on the 'Profiles' button at the top of the first panel. The profile section that appears in the first panel is also where you can view and apply existing profiles to the current session. ### [](https://guide.cryosparc.com/live/ui-overview#individual-tab) **Individual Tab** ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNzXvuSVaxFb4QqotKP%252F-MNzagJc1CJKJnw9lcP6%252FCSL_UI_11.png%3Falt%3Dmedia%26token%3D1423692a-18d3-4c05-9698-c7d1dd2c15f8&width=768&dpr=3&quality=100&sign=4938bbd0&sv=2) The individual tab contains details of the currently selected exposure, including diagnostic plots on the left side of the screen, and an interactive exposure viewer on the right, complete with a manual particle picker and particle inspection controls for manual, blob and template picks. Under the individual tab button within the sidebar, properties that are common across all exposures within the session are displayed including the exposure dimensions, pixel size and number of frames. By default, the selected exposure is the most recent exposure that has been fully processed. You can use the exposure feed to scroll across all of the exposures within the session and click to select one. You can also right click an exposure to view even more details about it. Additionally, you can use the exposure navigation controls in the header to navigate to the latest available exposure, first exposure, and anywhere in between. When set to show the latest exposure, the interface will automatically load and display new data as it becomes available. Hovering over the exposure viewer will display a set of controls such as changing the zoom level and setting the lowpass filter value. Additional details regarding the selected exposure are shown above the exposure viewer, including the status of the exposure and what exposure group it is contained in. Clicking on the exposure details will reveal further options, such as the ability to reject it from further processing, option to reprocess it, or view and copy the path of the exposure. To the right of the exposure details are the exposure pick details. Each type of pick is shown as a different colour: manual picks in green, blob picks in yellow and template picks in red. The display of each picker type on the exposure viewer can be toggled by clicking on the corresponding picker type button. The number of picks shown on the selected exposure is listed first, and if particles have been extracted, the number of extracted particles from the selected exposure is shown to the right of the arrow. More picking statistics and display options are available by clicking on the picking menu dropdown. Each type of pick can be displayed as a point, circular outline of the particle diameter, or extraction box size. ### [](https://guide.cryosparc.com/live/ui-overview#overview-tab) **Overview Tab** ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNzXvuSVaxFb4QqotKP%252F-MNzajz_DkDbnmygYVdx%252FCSL_UI_12.png%3Falt%3Dmedia%26token%3D05d9642b-bc7f-41ca-bc53-691a09065187&width=768&dpr=3&quality=100&sign=e0d960e&sv=2) The overview tab is a powerful view in cryoSPARC Live that allows for inspection of 20 different attributes across all exposures in the session. Exposure-level details such as time of processing, motion statistics, CTF fit, relative ice thickness and picking statistics are shown as scatter plots, with new exposures automatically added when they are processed. When you hover over a point in any plot, a preview of the exposure will display on the left side of the exposure viewer. Clicking on a point in any plot will select it and show it in the exposure viewer. At any time while a session is running, you can choose to set session-level thresholds based on any attribute value using the input boxes, slider or dragging a vertical region on the plot. Once confirmed, all current and future exposures that are contained within the range of the threshold will be noted as accepted. Exposures outside the threshold range will be marked as rejected and excluded from further processing. Exposures that are marked as accepted will be displayed in blue and rejected exposures are displayed in red. Thresholds are applied in an additive manner, and can be cleared at any time. Applying thresholds is a great way to filter exposures based on their quality. For example, you may choose to exclude any exposure that has a calculated CTF Fit (Å) value greater than 4 angstroms. Under the overview tab button within the sidebar, any attribute with a threshold applied will display with the set minimum and maximum value. ### [](https://guide.cryosparc.com/live/ui-overview#browse-tab) **Browse Tab** ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNzXvuSVaxFb4QqotKP%252F-MNzao9bo0aHrKJaAoZf%252FCSL_UI_13.png%3Falt%3Dmedia%26token%3Db7989322-12a6-4261-ab65-86e09d20211e&width=768&dpr=3&quality=100&sign=4866785a&sv=2) The browse tab is another great tool for exploring exposure data across the session. A scatter plot with a configurable X and Y axis is displayed on the top-half of the tab, and table of all exposures processed is displayed on the bottom-half of the tab. Just as the overview tab, accepted exposures are shown in blue and rejected in red and any exposure can be selected by clicking on a point. Hovering over a point will reveal a preview of that exposure on the left side of the exposure viewer. The dropdown menu can be used to filter the exposure table, and the download button can be used to save exposures shown in the table as a CSV file, which respects the applied filters and sort options at time of download. ### [](https://guide.cryosparc.com/live/ui-overview#picking-tab) **Picking Tab** ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNzXvuSVaxFb4QqotKP%252F-MNzarT-h3quJaEG2fZ5%252FCSL_UI_14.png%3Falt%3Dmedia%26token%3D164156ad-9adb-4451-887b-3b771d25161f&width=768&dpr=3&quality=100&sign=e6b81e7&sv=2) The picking tab contains all the controls needed to activate and filter either manual, blob or template picks. By default, the blob picker is activated at the start of every session. The active picker is always displayed in the sidebar under the details tab button. To view options for another picker type, click on corresponding button at the top of the tab. #### [](https://guide.cryosparc.com/live/ui-overview#manual-picker) **Manual Picker** The manual picks table will automatically update when new manual picks are added. To add or remove manual picks, hover over the exposure viewer and select the add or remove actions at the bottom right. When the 'add' picks mode is active, left click to add a pick and right click to remove a pick. When the 'remove' picks mode is active, left click to remove a pick. To extract manual picks, click the 'Start Manual Extraction' button at the bottom of the tab. If there are any manually picked particles in the session, the total number of picks and number of exposures with manual picks will be displayed under the picking tab button within the sidebar. #### [](https://guide.cryosparc.com/live/ui-overview#blob-picker) **Blob Picker** Within the blob picking section, you have the ability to adjust any blob picker parameters and apply that to all exposures or any future exposures. Additionally, you can 'test' a single exposure with different parameters to see the result in the exposure viewer before committing to applying those parameters. Test exposures will display in the feed with a the letter 'T' in a purple tag, and any test exposures can be reset via the exposure menu. Blob picks can be filtered in the section below, allowing you to adjust the NCC and power thresholds and confirm those thresholds. Once confirmed, you can choose to extract particles using the new thresholds for all or only future exposures. As you adjust the thresholds, helpful messages will appear below stating an estimate of how many picks will remain if the new thresholds are set. Additionally, a description of what picker is currently active and using what settings will also be displayed below the thresholds. #### [](https://guide.cryosparc.com/live/ui-overview#template-picker) **Template Picker** The template picker can be activated by creating or loading templates for the template picker to use. You can generate templates using existing manual, blob or template picks from the current session or load existing templates from another CryoSPARC job. Additionally, you can load templates from a completed streaming 2D classification job within the session if one has completed. Once templates are loaded, you can select one or more templates for the template picker to use. The template picking tab also contains similar sections to adjust parameters and filter and extract template picks as displayed in the blob picker. The template picker can be activated when at least one template is selected for the picker to use and all parameters are configured. Under the picking tab button within the sidebar, blob or template picking details such as the extraction box size and picking diameter will be shown for the active picker. ### [](https://guide.cryosparc.com/live/ui-overview#id-2d-classification-tab) **2D Classification Tab** ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNzXvuSVaxFb4QqotKP%252F-MNzavaJ5tfHhOjqJD6w%252FCSL_UI_15.png%3Falt%3Dmedia%26token%3Da22ebdff-de3b-42fc-9f71-d4b86e4ec690&width=768&dpr=3&quality=100&sign=5523e42f&sv=2) The 2D classification tab allows for the classification of particles extracted within the session. Once you are happy with the number of picks available in the session, a classification job can be started. Within CryoSPARC Live, 2D classification operates in a steaming fashion and will automatically classify new particles as they become available. At any time, you can restart the classification job or choose to start a new job and configure custom parameters via the CryoSPARC interface by using the purple build button. The interface will automatically update to an interactive mode when 2D classes are ready for selection. All particles within selected classes will continue to the reconstruction stage. Under the 2D classification tab button within the sidebar, details regarding classes and particles are shown, along with progress messages that update in real-time. Within the sidebar or at the top of the tab, you can click on the project and job ID of the streaming 2D classification job to view detailed logs and diagnostic plots for the job. Click on the close button at the top right of the screen or press spacebar to return to the main interface. ### [](https://guide.cryosparc.com/live/ui-overview#ab-initio-tab) **Ab-initio Tab** ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNzXvuSVaxFb4QqotKP%252F-MNzazNac-djy0rLFULq%252FCSL_UI-16.png%3Falt%3Dmedia%26token%3De9e034a0-36e4-4d8f-86bc-99a9db1d3344&width=768&dpr=3&quality=100&sign=aab28cd2&sv=2) The ab-initio tab allows you to configure and view an ab-initio model to be used for the refinement stage. You can either load an existing volume from a CryoSPARC job within the current project or run an ab-initio job based on particles from the session. An ab-initio job can be run once at least one 2D class is selected. Once in progress, based on the number of classes, one or more volumes will appear in the tab. Each volume can be selected and its 3D preview will update in real time as the job progresses. Similarly to the 2D classification tab, details of the ab-initio job will be displayed in the navigation sidebar. You can inspect the log output of the job by clicking on the job identifier within the sidebar or at the top of the tab. ### [](https://guide.cryosparc.com/live/ui-overview#refinement-tab) **Refinement Tab** ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNzXvuSVaxFb4QqotKP%252F-MNzb1a0E_4is0micCW-%252FCSL_UI_17.png%3Falt%3Dmedia%26token%3D560169c2-3938-42d4-a5b5-15909c5bc3b4&width=768&dpr=3&quality=100&sign=b1f0e4ff&sv=2) The final tab and last stage of Live processing is the refinement tab, which will run a homogeneous refinement on all selected particles in the session. You can configure a refinement job by ensuring an ab-initio volume has been loaded into the session and specifying a symmetry value. Additionally, you can choose to configure custom refinement parameters via the CryoSPARC interface using the purple build icon. Within CryoSPARC Live, the refinement operates in a steaming fashion and will automatically update when new particles become available. Once a refinement job has started to process, its volume will automatically load and continue to update as the job progresses. Diagnostic plots will appear and update over time. You can use the volume viewer to interact with the map, adjust the contour threshold, and download the map. Similarly to the ab-initio tab, details of the streaming refinement job will be displayed in the navigation sidebar. You can inspect the log output of the job by clicking on the job identifier within the sidebar or at the top of the tab. [PreviousHow to Access CryoSPARC Live](https://guide.cryosparc.com/live/how-to-access-cryosparc-live) [NextNew Live Session: Start to Finish Guide](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide) Last updated 9 months ago * [Browse Sessions Page](https://guide.cryosparc.com/live/ui-overview#browse-sessions-page) * [Live Session Layout](https://guide.cryosparc.com/live/ui-overview#live-session-layout) * [1\. Header](https://guide.cryosparc.com/live/ui-overview#id-1.-header) * [2\. Sidebar](https://guide.cryosparc.com/live/ui-overview#id-2.-sidebar) * [3\. Exposure Feed](https://guide.cryosparc.com/live/ui-overview#id-3.-exposure-feed) * [4\. Exposure Navigation](https://guide.cryosparc.com/live/ui-overview#id-4.-exposure-navigation) * [5\. Data Processing Tabs (e.g., Individual Exposure, Overview, Picking)](https://guide.cryosparc.com/live/ui-overview#id-5.-data-processing-tabs-e.g.-individual-exposure-overview-picking) * [6\. Exposure / Volume Viewer](https://guide.cryosparc.com/live/ui-overview#id-6.-exposure-volume-viewer) * [7\. Footer](https://guide.cryosparc.com/live/ui-overview#id-7.-footer) * [Navigating the Session](https://guide.cryosparc.com/live/ui-overview#navigating-the-session) * [Details Tab](https://guide.cryosparc.com/live/ui-overview#details-tab) * [Configuration Tab](https://guide.cryosparc.com/live/ui-overview#configuration-tab) * [Individual Tab](https://guide.cryosparc.com/live/ui-overview#individual-tab) * [Overview Tab](https://guide.cryosparc.com/live/ui-overview#overview-tab) * [Browse Tab](https://guide.cryosparc.com/live/ui-overview#browse-tab) * [Picking Tab](https://guide.cryosparc.com/live/ui-overview#picking-tab) * [2D Classification Tab](https://guide.cryosparc.com/live/ui-overview#id-2d-classification-tab) * [Ab-initio Tab](https://guide.cryosparc.com/live/ui-overview#ab-initio-tab) * [Refinement Tab](https://guide.cryosparc.com/live/ui-overview#refinement-tab) --- # Live Jobs and Session-Level Functions | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/live/live-jobs-and-session-level-functions.md) . [](https://guide.cryosparc.com/live/live-jobs-and-session-level-functions#cryosparc-live-architecture-and-reprocessing) CryoSPARC Live Architecture and Reprocessing ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- **CryoSPARC Live is built on the idea that one should be able to experiment with parameters on the fly and that the software should be able to efficiently reprocess/redo the steps that are necessary in order to test or effect parameter changes, while maintaining the overall progress of the Live session.** CryoSPARC Live tracks and manages exposures, particles, results, volumes, etc in a new architecture that allows for consistent management of session state, reprocessing only the necessary work when parameters are changed, tracking acceptance/rejection of images, and live streaming of particles through various stages of processing. This new underlying system supports many new features, and also requires some job types and session-level functions unique to CryoSPARC Live (as compared with the main CryoSPARC application). ### [](https://guide.cryosparc.com/live/live-jobs-and-session-level-functions#live-session-terminology) Live session terminology * **Session:** The processing workflow for doing real-time preprocessing and 2D/3D processing in CryoSPARC Live. New Sessions (`SX`) can be configured from the CryoSPARC Live Application directly and they are housed within an existing CryoSPARC Project. Live Sessions are the equivalent of Workspaces (`WX`) in the regular CryoSPARC Application. Therefore, there each session maps to one workspace. * **Exposure:** An individual image file that CryoSPARC Live preprocesses. The "Exposure" terminology is used to refer to both movies and micrographs interchangeably. * **Exposure Group:** A collection of exposures with the same optical parameters. Mainly includes information about where to find the exposures on disk, and how CryoSPARC can find them. A Live Session can have multiple exposure groups, which can be used to separate data collected from different data collection sessions, grids, or beam tilt groups. Exposures and their extracted particles can be ignored from downstream processing by ignoring the exposure groups from which they came via the Configuration tab. ### [](https://guide.cryosparc.com/live/live-jobs-and-session-level-functions#job-types-unique-to-cryosparc-live) Job types unique to cryoSPARC Live * **CryoSPARC Live Session Job:** The job created when a Live Session is created. This job, visible in the CryoSPARC interface, is the parent to all jobs later created by a running Live Session. * **CryoSPARC Live Worker:** The live preprocessing worker that executes motion correction, CTF estimation, thumbnail generation, particle picking and particle extraction for each exposure it encounters. * **Streaming 2D Classification:** An adapted version of 2D Classification that can classify new particles as they are extracted by the CryoSPARC Live Worker in a streaming fashion. * **Streaming Refinement:** An adapted version of Homogeneous Refinement that can refine a volume with new particles as they are classified by the Streaming 2D Classification job in a streaming fashion. * **CryoSPARC Live Exposure Export:** Exports all valid exposures from a Live Session to its CryoSPARC workspace, for use in further processing. * **CryoSPARC Live Particle Export:** Exports all valid particles from a Live Session to its CryoSPARC workspace, for use in further processing. ### [](https://guide.cryosparc.com/live/live-jobs-and-session-level-functions#session-level-functions) Session-Level Functions There are a few high-level Session Functions that can be performed. **Start Session:** This will cause the Preprocessing GPU Worker(s) to start reading in movies and perform preprocessing steps (motion correction, CTF estimation, particle picking and extraction). You **must** have filled out and saved all required parameters in the Configuration Tab in order to Start a Session. **Pause Session:** This will cause any currently running and queued CryoSPARC Live jobs (Preprocessing GPU Workers(s), Streaming 2D Classification, Ab-Initio Reconstruction and Streaming Refinement) to be killed and marked as completed. Once paused, the session can be started again to continue processing. In order to edit any Configuration or Compute Resources parameters from the Configuration Tab, e.g., changing the Number of Preprocessing GPU Workers during a running session, you will need to Pause Session and then Start Session again for them to take effect. **Mark as Completed:** This will set the status of the session to "Completed". Doing this allows you to organize your sessions, as well as manage the data created by the session via the "Manage Data" tab. **Clear Session:** This will delete all data created by the session, and remove all intermediate results created by the session. Only the parameters that were used to set up the session will remain, to allow for easy re-starting of the session. **Modify Exposure Processing Priority:** There are four modes available that modify which exposures are prioritized to be processed by the Preprocessing GPU Worker(s). This is available as of CryoSPARC `v3.3.2+220824` and later. * `normal`: Exposures are processed in ascending UID order, unless exposures that need to be reprocessed are available. In that case, exposures will be processed in descending UID order. This is the default priority mode. * `oldest`: The oldest found exposure that needs to be processed (or reprocessed) will be prioritized. * `latest`: The latest found exposure that needs to be processed (or reprocessed) will be prioritized. * `alternate`: Alternate between `oldest` and `latest` priority modes. Test exposures and exposures manually set to be reprocessed will always be processed first, no matter the priority mode. The exposure processing priority of a session can be modified at any time. To modify it in CryoSPARC v5+, use `cryosparcm cli` to run the following command: `cryosparcm cli "api.sessions.set_session_exposure_processing_priority('', '', exposure_processing_priority='')"` For example: `cryosparcm cli "api.sessions.set_session_exposure_processing_priority('P54', 'S1', exposure_processing_priority='latest')"` For older versions of CryoSPARC, use `cryosparcm rtpcli` to run the following command: `cryosparcm rtpcli "set_session_exposure_processing_priority('', '', '')"` For example: `cryosparcm rtpcli "set_session_exposure_processing_priority('P3', 'S1', 'latest')"` [PreviousCryoSPARC Live Tutorial Videos](https://guide.cryosparc.com/live/tutorial-videos) [NextPerformance Metrics](https://guide.cryosparc.com/live/performance-metrics) Last updated 5 months ago * [CryoSPARC Live Architecture and Reprocessing](https://guide.cryosparc.com/live/live-jobs-and-session-level-functions#cryosparc-live-architecture-and-reprocessing) * [Live session terminology](https://guide.cryosparc.com/live/live-jobs-and-session-level-functions#live-session-terminology) * [Job types unique to cryoSPARC Live](https://guide.cryosparc.com/live/live-jobs-and-session-level-functions#job-types-unique-to-cryosparc-live) * [Session-Level Functions](https://guide.cryosparc.com/live/live-jobs-and-session-level-functions#session-level-functions) --- # Create and Build Jobs | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/create-and-build-jobs.md) . [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/create-and-build-jobs#job-builder) Job Builder ------------------------------------------------------------------------------------------------------------------------------- From within a workspace, use the Job Builder to search for and create new jobs. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNoqDWujyGYLEHAKVGs%252F-MNorOr8QooxrB2pZst2%252FUI_job_builder_2.png%3Falt%3Dmedia%26token%3D8a6d25ef-e05b-4748-83d0-55547fd47bbc&width=768&dpr=3&quality=100&sign=b8dbb856&sv=2) Once you choose a job type, click on it to transition into the 'Building' view of the job builder. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNoqDWujyGYLEHAKVGs%252F-MNore-pd-HvVd9dxA5u%252FUI_job_builder_building_3.png%3Falt%3Dmedia%26token%3D538f040b-50db-4f77-aa3f-d77af51cd51b&width=768&dpr=3&quality=100&sign=872a255d&sv=2) ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/create-and-build-jobs#tutorial-job-builder) Tutorial: Job Builder For a detailed overview of how to use the Job Builder in cryoSPARC, please see: [Tutorial: Job Builder](https://guide.cryosparc.com/guides-for-v3/job-builder-tutorial) ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/create-and-build-jobs#inputs-and-parameters) Inputs and parameters Enter any required parameters, and connect any required inputs. Inputs to a job are the data or files that will be processed by the job. For example, the input to a Patch CTF Estimation job are micrographs. Inputs for a job can come from the Workspace in which you are actively working, or any other Workspace within your current Project. See [All Job Types in cryoSPARC](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc) for detailed information about the inputs, parameters and outputs for each job type. ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/create-and-build-jobs#drag-and-drop-to-connect) Drag and drop to connect Connect the required input(s) by dragging and dropping the output(s) from another job. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNoTUDBL-SG-vnN-ZXe%252F-MNoYCQiOMBv4KZI9ZOt%252FJB_1_cryoSPARC_dragndrop_optimized.gif%3Falt%3Dmedia%26token%3D31e3cf5f-3918-424e-9576-9fc487ef0ba6&width=768&dpr=3&quality=100&sign=7cfeb09f&sv=2) If you have successfully connected the output(s), you will see the job number appear below the input, as well as an option to Remove if you have accidentally connected the wrong output: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNoTUDBL-SG-vnN-ZXe%252F-MNoYIksz-YoJ_aiQ9Ov%252FJB_2_v2_inputsdropped.png%3Falt%3Dmedia%26token%3D96cd397c-9fc8-4258-ba8e-c230f32af0ce&width=768&dpr=3&quality=100&sign=fe62698b&sv=2) ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/create-and-build-jobs#default-and-spec-parameters) Default and spec parameters CryoSPARC v2 populates default parameters for most jobs where necessary. To show additional parameters, toggle to Advanced Mode at the top right hand corner of the Job Builder: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNoTUDBL-SG-vnN-ZXe%252F-MNoYS6yXeFV-dRz7aBO%252FJB_3_v2_advancedmode.png%3Falt%3Dmedia%26token%3Dd4fe4d72-25c7-4084-8122-2afe1f971b4b&width=768&dpr=3&quality=100&sign=8601e144&sv=2) All default parameters are indicated with a blue **D**. If you modify a parameter, it will be indicated with a green **S**. To restore the default value, click on the **x** to the right of the parameter you wish to restore. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNoTUDBL-SG-vnN-ZXe%252F-MNoYWvF2BBuuMlWKEA6%252FJB_4_v2_defaultparams.png%3Falt%3Dmedia%26token%3De8e748c1-f951-41b9-ae27-97b0d77ae60a&width=768&dpr=3&quality=100&sign=b93f58c8&sv=2) Once satisfied that you have successfully dragged the required input(s) and modified any applicable parameters, click' Queue' to launch the job. [Queue Job, Inspect Job and Other Job Actions](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions) ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/create-and-build-jobs#cancel-a-building-job) Cancel a building Job You can cancel building a job by clicking 'Cancel' at the bottom of the Job Builder at any time. Doing so will make the job inactive as shown below. To resume building the job, click on the 'Building' button in the middle of the job card, or, select the job card and press `B`. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNosa1V8TnTbP-uPLs_%252F-MNp-3twidXqlFJWnOJM%252FUI_job_builder_cancelled_1.png%3Falt%3Dmedia%26token%3D8154d261-75e8-4c4e-aebe-68342470f1fb&width=768&dpr=3&quality=100&sign=363d269a&sv=2) [PreviousProject and Workspace Management](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/project-and-workspace-management) [NextQueue Job, Inspect Job and Other Job Actions](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions) Last updated 1 year ago * [Job Builder](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/create-and-build-jobs#job-builder) * [Tutorial: Job Builder](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/create-and-build-jobs#tutorial-job-builder) * [Inputs and parameters](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/create-and-build-jobs#inputs-and-parameters) * [Drag and drop to connect](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/create-and-build-jobs#drag-and-drop-to-connect) * [Default and spec parameters](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/create-and-build-jobs#default-and-spec-parameters) * [Cancel a building Job](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/create-and-build-jobs#cancel-a-building-job) --- # Project and Workspace Management | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/project-and-workspace-management.md) . [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/project-and-workspace-management#projects-and-workspaces-in-cryosparc) Projects and Workspaces in cryoSPARC -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Projects in cryoSPARC are high level containers corresponding with a project directory on the filesystem, which house all associated Jobs. Each Project in cryoSPARC is entirely contained within a filesystem directory. All the jobs and their respective intermediate and output data created within a Project will be stored within the project directory. Projects are strict divisions. Files and jobs from different projects are stored in dedicated project directories and jobs cannot be connected from one project to another. A project directory is self-contained, meaning that if you have an intact project directory, you can import that Project into any cryoSPARC instance at any time. Workspaces on the other hand are logical groupings, like labels, that are created by the user to separate portions of a workflow for ease. A job can be added or removed from multiple workspaces. Workspaces do not have any particular directory on the filesystem. ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/project-and-workspace-management#when-to-create-a-new-project) When to create a new Project **Recommendation: Create a new Project for each new unrelated sample on which you are collecting data.** Additional recommendations for creating new Projects and Workspaces: * **Collecting new data for the first time on a new target molecule:** Create a new Project and a new Workspace within it. Import the movies/micrographs/particle stacks into the new Workspace. * **Collecting data a second or subsequent time on the same sample/target** (potentially the same or different grid from the same batch, potentially on a different day): Use the existing Project and existing Workspace where you processed the first set of images. Import the movies/micrographs/particle stack into the existing Workspace or a new Workspace, in the same Project. * **Collecting data on a new sample/grid/preparation of the same target molecule:** Use the existing Project, but create a new Workspace. This allows easy re-use of 3D volumes, 2D templates, and easy combining of particle images downstream. You can create multiple Workspaces within a Project, for example if collecting/processing new data from a similar sample. See similar considerations for creating Projects and Sessions for cryoSPARC Live in the New Live Session: Start to Finish Guide ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/project-and-workspace-management#creating-your-first-project) Creating your first Project 1.Navigate to the Projects view by clicking on the drawer icon on the header, or from the Projects button in the footer. ​​ ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNo_aA0KMM-5Njfj_Lg%252F-MNogDUFbLAvVNfL6F9q%252FUI_click_projects_view_2.png%3Falt%3Dmedia%26token%3D8826c3e4-6e4f-48f9-89df-9ac1762f027e&width=768&dpr=3&quality=100&sign=142eda63&sv=2) 2\. To create a project, press `N` or click on "+ Add" on the header, which will bring up a modal window for the New Project details. ​​ ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNo_aA0KMM-5Njfj_Lg%252F-MNog1aWN8-Pkh1ubOSt%252FUI_create_New_Project_1.png%3Falt%3Dmedia%26token%3Dfc47894e-ff0e-4148-8876-0fa099e16743&width=768&dpr=3&quality=100&sign=13e91238&sv=2) 3\. Enter a project Title and select a location for the associated project directory from the File Browser. The project directory you select should already exist, and it will be populated with job directories as you create jobs. All files associated with the project will be stored inside the selected project directory. You may also wish to enter a Description for your project. 4\. Your new project now appears on the Projects page. ​​ 5\. To open the Project, click on the "Px" button on the top left hand side of a Project card: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNogF-yXQGczl7hYiGa%252F-MNogedMSlUe6mDieD2B%252FUI_open_project_1.png%3Falt%3Dmedia%26token%3Df897b873-6bf3-403e-a650-2a3ecd6a9c15&width=768&dpr=3&quality=100&sign=3e0fdd05&sv=2) 6\. You can view Project details and actions from the Project Details panel: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNojqx2rYy5Tz0vSdA-%252F-MNokVt8VeQnhzD5PCaC%252FUI_project_details_1.png%3Falt%3Dmedia%26token%3De21c9b48-346e-48a9-9523-34da5ddec2eb&width=768&dpr=3&quality=100&sign=7ba241d4&sv=2) ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/project-and-workspace-management#creating-your-first-workspace) Creating your first Workspace Before you can start processing data inside of a new project, you will need to create at least one workspace. 1\. Click the project number (e.g., P44) to open the Project within which you want to create a Workspace. 2\. Once inside your selected Project, create a New Workspace using the "+ Add" button on the header or `N` on your keyboard. Or, you can click `New Workspace` from the Project Details panel on the right. Workspace titles can be changed later, and descriptions can be added any time. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNogF-yXQGczl7hYiGa%252F-MNoi8cTaTVmFpW-vvU9%252FUI_new_workspace_1.png%3Falt%3Dmedia%26token%3Db163995c-b54c-46e5-8ae5-b69dc46692e8&width=768&dpr=3&quality=100&sign=52261717&sv=2) 3\. You can open a Workspace either by clicking on the Px - Wx button for a particular Workspace (pictured immediately above), or you can locate it using the Workspace search bar in the header. 4\. You can enter notes or details about the Workspace and find available actions from the Workspace Details panel: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNogF-yXQGczl7hYiGa%252F-MNoj0V-Q3ElwMUYDYKb%252FUI_workspace_details.png%3Falt%3Dmedia%26token%3D6689ddd0-dd30-4286-85d5-b1fd32b6680c&width=768&dpr=3&quality=100&sign=5ec40eb3&sv=2) Once you have created at least one Workspace within a Project, you can start creating jobs: [Create and Build Jobs](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/create-and-build-jobs) [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/project-and-workspace-management#additional-project-and-workspace-actions) Additional Project and Workspace Actions ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/project-and-workspace-management#share-a-project) Share a Project Only users who own a particular Project or have a Project shared with them can see Workspaces, cryoSPARC Live Sessions and jobs within those Projects. To add a user to a Project, navigate to the Project Details Panel for a particular project (owner should be the logged in user) and click `Share With Users` to select the user you wish to give access. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MQSa2wI5zv_gNGKi40g%252F-MQSaEitf42SiY8BV6fK%252Fimage.png%3Falt%3Dmedia%26token%3D738f25b4-7c2a-49c5-9312-0312a4e9103f&width=768&dpr=3&quality=100&sign=943056e3&sv=2) ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/project-and-workspace-management#delete-a-project) Delete a Project Navigate to the Workspace and locate the 'Delete' button on the Workspace Details Tab. A pop-up message will ask you to confirm the delete. Once you confirm, another popup will show you a list of jobs and workspaces that will also be deleted and ask you to confirm again. Deleted jobs are first cleared before deleting, meaning that their intermediate and final results are erased from disk. Note that unlike deleting jobs, deleting a project doesn't remove the project folder from the disk. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNojqx2rYy5Tz0vSdA-%252F-MNokksnIviotzfnEmnb%252FUI_confirm_delete_jobs_1.png%3Falt%3Dmedia%26token%3Db6f5f293-0c2c-4fd1-a688-237cd1af9cbf&width=768&dpr=3&quality=100&sign=cfcc0495&sv=2) ### [](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/project-and-workspace-management#delete-a-workspace) Delete a Workspace Navigate to the Workspace and locate the 'Delete' button on the Workspace Details Tab. A pop-up message will ask you to confirm the delete. Once you confirm, another popup will show you two lists: A. jobs that reside only inside the selected workspace, and that will be fully deleted, and B. jobs that exist in the selected workspaces, and other workspaces (i.e., because they were [linked](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions) ). Jobs that exist in multiple workspaces (list B) will not be fully deleted. They will simply be disassociated from the workspace to be deleted. Note that deleted jobs are first cleared before deleting, meaning that their intermediate and final results are erased from disk. [PreviousDashboard](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/dashboard) [NextCreate and Build Jobs](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/create-and-build-jobs) Last updated 1 year ago * [Projects and Workspaces in cryoSPARC](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/project-and-workspace-management#projects-and-workspaces-in-cryosparc) * [When to create a new Project](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/project-and-workspace-management#when-to-create-a-new-project) * [Creating your first Project](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/project-and-workspace-management#creating-your-first-project) * [Creating your first Workspace](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/project-and-workspace-management#creating-your-first-workspace) * [Additional Project and Workspace Actions](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/project-and-workspace-management#additional-project-and-workspace-actions) * [Share a Project](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/project-and-workspace-management#share-a-project) * [Delete a Project](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/project-and-workspace-management#delete-a-project) * [Delete a Workspace](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/project-and-workspace-management#delete-a-workspace) --- # Local Refinement | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement.md) . Local Refinement and Particle Subtraction are two jobs useful in addressing the issue of sample flexibility, along with 3D Variability and Non-Uniform Refinement. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fk0pF7zzHHLULYjybuojM%252Floc-ref_rotation-pose.png%3Falt%3Dmedia%26token%3D09cf3709-4790-4924-914e-ddcf7c047778&width=768&dpr=3&quality=100&sign=5c84dcbb&sv=2) When aligning a particle with a flexible domain, a mask can be used to focus alignment on a particular region. This process is called Local Refinement. [**Local Refinement**](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-new-local-refinement-beta) is the process of iteratively refining a selected sub-volume within a larger volume, while also providing a characterization of the conformational distribution that is represented in the particle images. The primary use of Local Refinement is to account for and "undo" the relative motion between the masked sub-volume, and the rest of the molecule, which differentiates Local Refinement from other jobs like Non-Uniform Refinement. [**Particle Subtraction**](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta) is a useful job in preparing particles for the local refinement job. Particle Subtraction involves subtracting the signal, contributed to by the rest of the molecule, from the input particle images. This yields a set of particle images containing the signal only from the masked sub-volume. Often, using signal-subtracted particles instead of the raw particles may improve alignment during Local Refinement, and hence improve the quality of the refinement. These jobs are both used in the [**Case Study on Yeast U4/U6.U5 tri-snRNP**](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp) , in which we work step-by-step through the process for improving reconstruction quality for large, flexible domains. Also included in this section is a tutorial on [**Mask Creation**](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera) , which details how to take a consensus refinement and generate masks covering the region of interest (for Local Refinement), as well as the inverse region (for Particle Subtraction). [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement#local-refinement-jobs) Local Refinement Jobs -------------------------------------------------------------------------------------------------------------------------------------------- [Job: Local Refinement](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-new-local-refinement-beta) [Job: Particle Subtraction](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta) [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement#local-refinement-tutorials) Local Refinement Tutorials ------------------------------------------------------------------------------------------------------------------------------------------------------ [Tutorial: Mask Creation](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera) [Case Study: Yeast U4/U6.U5 tri-snRNP](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp) [PreviousJob: ThreeDFSC (Wrapper) (Legacy)](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-threedfsc-wrapper-legacy) [NextJob: Local Refinement](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-new-local-refinement-beta) Last updated 2 years ago * [Local Refinement Jobs](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement#local-refinement-jobs) * [Local Refinement Tutorials](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement#local-refinement-tutorials) --- # Job: Simulate Data (GPU) | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-gpu.md) . [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-gpu#description) Description ----------------------------------------------------------------------------------------------------------------------------------------- This job generates simulated particles from an input volume by projecting, CTF corrupting, and adding noise. This is an advanced job type that is not intended for common workflows. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-gpu#input) Input ----------------------------------------------------------------------------------------------------------------------------- * Volume with `map` result * Mask (optional) * Particles (optional) [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-gpu#common-parameters) Common Parameters ----------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-gpu#simulation-parameters) Simulation Parameters * `Number of particles to generate`: Specifies the number of particles to simulate. * `Noise Model`: Either "white", or "none". A white noise model will add Gaussian white noise to the simulated particles at the specified signal to noise ratio. Set to "none" for no noise. ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-gpu#ctf-parameters) CTF Parameters * `Read CTF from input particles`: If set to True and input particles are connected, the ctf information will be read from the particles and applied to the simulated images. ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-gpu#alignment-parameters) Alignment Parameters * `Read alignments from input particles`: If set to True and the input particles are connected, the 3D alignments will be read from the particles and used to generate the simulated images. * `Standard deviation of shift magnitude`: This specifies the standard deviation of the normal distribution that shifts are sampled from. The distribution is centered at the origin. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-gpu#output) Output ------------------------------------------------------------------------------------------------------------------------------- * Particles with `blob`, `ctf`, and `alignments3D` [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-gpu#example-images) Example Images ----------------------------------------------------------------------------------------------------------------------------------------------- ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fl5AWT3SjGesN285OSeWy%252Fv4-4-0-simulator-volume-example-0.png%3Falt%3Dmedia%26token%3D23e91700-8ce1-47f3-84fd-20ff54f7d1a0&width=768&dpr=3&quality=100&sign=8eccc2db&sv=2) Example input volume to the Simulator Job. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F1Cf6nlZcSwbkpJUPnHE9%252Fv4-4-0-simulator-example-0.png%3Falt%3Dmedia%26token%3Dba8c743e-53bc-4916-a66e-c6d51073067e&width=768&dpr=3&quality=100&sign=91f45b12&sv=2) Example simulated particle images with white noise. [PreviousSimulations](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations) [NextJob: Simulate Data (Legacy)](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-legacy) Last updated 1 month ago * [Description](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-gpu#description) * [Input](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-gpu#input) * [Common Parameters](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-gpu#common-parameters) * [Simulation Parameters](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-gpu#simulation-parameters) * [CTF Parameters](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-gpu#ctf-parameters) * [Alignment Parameters](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-gpu#alignment-parameters) * [Output](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-gpu#output) * [Example Images](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-gpu#example-images) --- # Job: Local Resolution Estimation | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-resolution-estimation.md) . [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-resolution-estimation#description) **Description** ----------------------------------------------------------------------------------------------------------------------------------------------------------- * Compute a local resolution map from the output of a refinement job. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-resolution-estimation#input) **Input** ----------------------------------------------------------------------------------------------------------------------------------------------- * Half maps of a refined volume, usually from a Homogenous/Heterogenous/NU-refinement job [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-resolution-estimation#common-parameters) **Common Parameters** ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- * `Annealing Factor` should be 0 for local resolution, increasing this to 1 will give the global resolution [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-resolution-estimation#output) **Output** ------------------------------------------------------------------------------------------------------------------------------------------------- * `map_locres` contains the resolution of each voxel of the original structure [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-resolution-estimation#notes) **Notes** ----------------------------------------------------------------------------------------------------------------------------------------------- * To accurately view `map_locres`, you can use the Surface Color tool in UCSF Chimera to colour the input map by the local resolution map. This is the best way to visualize the resolution at different parts of the structure. [PreviousJob: Validation (FSC)](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-validation-fsc) [NextJob: Local Filtering](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-filtering) Last updated 1 month ago * [Description](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-resolution-estimation#description) * [Input](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-resolution-estimation#input) * [Common Parameters](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-resolution-estimation#common-parameters) * [Output](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-resolution-estimation#output) * [Notes](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-resolution-estimation#notes) --- # Webinar Recordings | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/webinar-recordings.md) . [](https://guide.cryosparc.com/processing-data/webinar-recordings#webinar-methods-and-tools-for-processing-membrane-protein-cryo-em-data-ccemmp-seminar-series) Webinar: Methods and tools for processing membrane protein cryo-EM data (CCeMMP Seminar Series) -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- _July 2024_ Automation of single particle cryo-EM workflows can be very useful in a drug discovery context, especially when working on multiple structures within a similar class of targets. Using 8 publicly-available active state GPCR datasets, we walk through how to use new tools in CryoSPARC for one-click processing that in many cases can meet or exceed the resolution and map quality obtained from manual processing. New tools covered include [Workflows](https://guide.cryosparc.com/application-guide-v4.0+/workflows) (v4.4+) for automating the entire workflow, [Micrograph Denoiser](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/exposure-curation/job-micrograph-denoiser-beta) (v4.5+) and Junk Detector (pre-trained micrograph segmentation tool, in development) for improved particle picking, [Reference Based Auto Select 2D](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/job-reference-based-auto-select-2d-beta) for automated selection of good 2D classes, along with the use of decoy volumes for curation in 3D, and finally, [Reference Based Auto Select 3D](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-reference-based-auto-select-3d-beta) and [Align 3D maps](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-align-3d-maps) for selecting volumes and performing final refinements. [](https://guide.cryosparc.com/processing-data/webinar-recordings#webinar-cryosparc-live-the-advantages-of-on-the-fly-processing-in-cryo-em) Webinar: CryoSPARC Live: The advantages of on-the-fly processing in cryo-EM ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- _May 2024_ Modern cryo-EM datasets are typically several terabytes in size and often take over a day to collect. With traditional batch-based processing, users must wait for all files to be available on their local machine before beginning to process them. Moreover, processing must be performed entirely in serial, forcing later steps to wait until earlier steps (like motion correction) have completed. This talk covers the advantages of using CryoSPARC Live™, an on-the-fly processing system built in to CryoSPARC™. Movies are processed as they arrive, and later steps (like particle curation, 2D classification, and 3D refinement) can be performed in parallel with motion correction. This dramatically improves throughput, in some cases producing a high-quality 3D map even before all movies have been preprocessed. A version of this talk was originally presented at the Pacific Northwest Center for CryoEM (PNCC) in May 2024. [](https://guide.cryosparc.com/processing-data/webinar-recordings#webinar-real-time-cryo-em-analysis-for-all-cryosparc-live) Webinar: Real-time cryo-EM analysis for all: CryoSPARC Live --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- _June 2021_ Learn how CryoSPARC Live, a seamless real-time 2D and 3D processing system for single particle cryo-EM, accelerates time-to-structure and drives rapid insights into sample characteristics and data quality, enabling decision making while the sample is still in the microscope. CryoSPARC Live is not just for facilities; it is also the fastest, simplest way for beginners and experts alike to process cryo-EM data that has already been collected. We cover use cases, performance considerations, real-time experimentation and practical workflows, and are joined by two expert guest speakers, Giovanna Scapin (NIS) and Craig Yoshioka (PNCC), who discuss how they use CryoSPARC Live in practice in both industry and academic settings. [](https://guide.cryosparc.com/processing-data/webinar-recordings#webinar-resolving-flexibility-and-heterogeneity-with-3d-variability-analysis) Webinar: Resolving flexibility and heterogeneity with 3D Variability Analysis ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- _June 2020_ Learn how the new 3D Variability Analysis (3DVA) algorithm in CryoSPARC can reveal new functional and biological insight into the conformational and flexible motion dynamics of a protein molecule from single particle cryo-EM data. We will cover the new concepts introduced by the algorithm, interpretation of results, several case studies and examples, practical considerations to keep in mind when working on your own data, and live audience Q&A. 3DVA has already been used in many notable structural studies to shed light on protein dynamics, including GPCR structures and the SARS-CoV-2 spike protein. [https://doi.org/10.1016/j.jsb.2021.107702doi.org](https://doi.org/10.1016/j.jsb.2021.107702) [](https://guide.cryosparc.com/processing-data/webinar-recordings#webinar-cryoem-for-drug-discovery) Webinar: CryoEM for Drug Discovery -------------------------------------------------------------------------------------------------------------------------------------------- _May 2019_ Educational webinar hosted by Structura Biotechnology Inc., Merck and NVIDIA on how cryo-EM offers value for drug discovery and structure-based drug design, on targets like GPCRs and membrane proteins. Covers: why is cryo-EM useful for drug discovery; computational aspects involved in cryo-EM structure determination; future advancements. [PreviousTutorial: 3D Flex Mesh Preparation](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation) [NextAbout CryoSPARC Live](https://guide.cryosparc.com/live/about-cryosparc-live) Last updated 1 month ago * [Webinar: Methods and tools for processing membrane protein cryo-EM data (CCeMMP Seminar Series)](https://guide.cryosparc.com/processing-data/webinar-recordings#webinar-methods-and-tools-for-processing-membrane-protein-cryo-em-data-ccemmp-seminar-series) * [Webinar: CryoSPARC Live: The advantages of on-the-fly processing in cryo-EM](https://guide.cryosparc.com/processing-data/webinar-recordings#webinar-cryosparc-live-the-advantages-of-on-the-fly-processing-in-cryo-em) * [Webinar: Real-time cryo-EM analysis for all: CryoSPARC Live](https://guide.cryosparc.com/processing-data/webinar-recordings#webinar-real-time-cryo-em-analysis-for-all-cryosparc-live) * [Webinar: Resolving flexibility and heterogeneity with 3D Variability Analysis](https://guide.cryosparc.com/processing-data/webinar-recordings#webinar-resolving-flexibility-and-heterogeneity-with-3d-variability-analysis) * [Webinar: CryoEM for Drug Discovery](https://guide.cryosparc.com/processing-data/webinar-recordings#webinar-cryoem-for-drug-discovery) --- # Job: Local Filtering | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-filtering.md) . [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-filtering#description) **Description** ----------------------------------------------------------------------------------------------------------------------------------------------- Locally filter a refined map using a local resolution map. The Local Filtering job can: * filter an input volume using a spatially adaptive filter, either using a `lanczos` (default) or `gaussian` kernel * optionally apply global sharpening via a B-factor * Run on the GPU (default) or CPU * optionally filter only within a `mask_refine` input that is connected, to save time The job is designed to work with the output of the [Local Resolution Estimation](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-resolution-estimation) job. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-filtering#input) **Input** ----------------------------------------------------------------------------------------------------------------------------------- * a single volume input containing * `map_half_A` * `map_half_B` * `map_locres` * `mask_refine` (optional) [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-filtering#output) **Output** ------------------------------------------------------------------------------------------------------------------------------------- * a single volume output containing * `map_filtered` , the locally filtered, globally sharpened output map [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-filtering#common-parameters) **Common Parameters** ----------------------------------------------------------------------------------------------------------------------------------------------------------- * `B-factor for sharpening` should be set to a negative value to sharpen * `Maximum resolution for sharpening` should be set to a cutoff resolution to ensure that very high resolution Fourier components do not get over-sharpening resulting in a noisy output [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-filtering#common-next-steps) **Common Next Steps** ----------------------------------------------------------------------------------------------------------------------------------------------------------- [PreviousJob: Local Resolution Estimation](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-resolution-estimation) [NextJob: ResLog Analysis](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-reslog-analysis) Last updated 1 month ago * [Description](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-filtering#description) * [Input](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-filtering#input) * [Output](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-filtering#output) * [Common Parameters](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-filtering#common-parameters) * [Common Next Steps](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-filtering#common-next-steps) --- # Job: ResLog Analysis | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-reslog-analysis.md) . [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-reslog-analysis#description) **Description** ----------------------------------------------------------------------------------------------------------------------------------------------- This feature generates plots that show how the resolution of a given structure increases as more particles are added to the reconstruction. This insight may be useful in determining whether a higher-resolution result is possible with more particles, or if fewer particles are needed to achieve the same resolution. The implementation in CryoSPARC is based on [Stagg, S.M., Noble, A.J., Spilman, M. & Chapman, M.S. ResLog plots as an empirical metric of the quality of cryo-EM reconstructions. Journal of Structural Biology 185 (3), 418-426 (2014).](https://www.sciencedirect.com/science/article/pii/S1047847713003377?via%3Dihub) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FvfjD0iNG5ygRystAX9y7%252F33_ResLog_0143.png%3Falt%3Dmedia%26token%3D474607c6-a321-449f-bef7-a0f6de199fb0&width=768&dpr=3&quality=100&sign=dfe2e910&sv=2) Example ResLog output depicting number of particles vs. spacial frequency [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-reslog-analysis#input) **Input** ----------------------------------------------------------------------------------------------------------------------------------- * Particles [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-reslog-analysis#output) **Output** ------------------------------------------------------------------------------------------------------------------------------------- * Particles [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-reslog-analysis#common-parameters) **Common Parameters** ----------------------------------------------------------------------------------------------------------------------------------------------------------- * `Reconstruction box-size`: The volume size to use for refinement. If this is _None_, use the full image size. Otherwise images are automatically downsampled * `Symmetry`: Symmetry (e.g. C1, D7, etc.) of the dataset [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-reslog-analysis#common-next-steps) **Common Next Steps** ----------------------------------------------------------------------------------------------------------------------------------------------------------- * Additional refinements with tuned number of input particles based on the ResLog analysis [PreviousJob: Local Filtering](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-filtering) [NextJob: ThreeDFSC (Wrapper) (Legacy)](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-threedfsc-wrapper-legacy) Last updated 1 month ago * [Description](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-reslog-analysis#description) * [Input](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-reslog-analysis#input) * [Output](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-reslog-analysis#output) * [Common Parameters](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-reslog-analysis#common-parameters) * [Common Next Steps](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-reslog-analysis#common-next-steps) --- # Job: ThreeDFSC (Wrapper) (Legacy) | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-threedfsc-wrapper-legacy.md) . [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-threedfsc-wrapper-legacy#description) **Description** -------------------------------------------------------------------------------------------------------------------------------------------------------- This tool developed by the NYSBC, allows for visualizing directional FSCs of your structure. Please see the ThreeDFSC License Terms in the Notes. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-threedfsc-wrapper-legacy#output) **Output** ---------------------------------------------------------------------------------------------------------------------------------------------- * The histogram of FSCs will be outputted on the streamlog * Follow the instructions in the streamlog to access the full outputs of the ThreeDFSC software package through Chimera. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-threedfsc-wrapper-legacy#notes) **Notes** -------------------------------------------------------------------------------------------------------------------------------------------- Copy MIT License Copyright (c) 2017 New York Structural Biology Center, Salk Institute for Biological Studies Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-threedfsc-wrapper-legacy#limitations) **Limitations** -------------------------------------------------------------------------------------------------------------------------------------------------------- * The full functionality of ThreeDFSC is not currently implemented in this wrapper. The remaining functions will be added in future updates. [PreviousJob: ResLog Analysis](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-reslog-analysis) [NextLocal Refinement](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement) Last updated 1 month ago * [Description](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-threedfsc-wrapper-legacy#description) * [Output](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-threedfsc-wrapper-legacy#output) * [Notes](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-threedfsc-wrapper-legacy#notes) * [Limitations](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-threedfsc-wrapper-legacy#limitations) --- # Job: Particle Subtraction | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta.md) . [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta#docs-internal-guid-ed02a34a-7fff-5241-a950-69249ac4630d) At a Glance -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Subtract projections of a masked sub-volume from particle images. * Use input particles with a gold-standard half set * Ensure that the region to be subtracted is well-aligned * The mask must have a soft edge [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta#description) Description ------------------------------------------------------------------------------------------------------------------------------------------------------ A particle subtraction job is used to remove unwanted signal from particle images. The input volume is projected in the fitted pose for each input particle. This projection is then subtracted from the particle image, yielding an image of the particle as if it did not have the masked out region. Because the input volume is subtracted from the particle images, the final quality of the signal subtraction is highly dependent on the quality of the input volume. Often, performing a [Local Refinement](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-new-local-refinement-beta) of the sub-volume that is to be subtracted prior to performing Particle Subtraction improves the quality of this region, which therefore improves the quality of the final particle stack. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FQON6Pp556Fu9Xco5Miea%252Fpart-sub_particle-subtraction.png%3Falt%3Dmedia%26token%3D8046d63a-02db-4336-8a32-9df925213fab&width=768&dpr=3&quality=100&sign=b608a81e&sv=2) In a particle subtraction job, a sub-volume is projected in each particle image's pose. This projection is then subtracted from the image, resulting in an image which ideally contains only signal from the region outside the mask. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta#docs-internal-guid-07032c2a-7fff-89ef-1fa1-41a8f6c81924) Inputs --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta#particles) Particles Particles must be from a previous refinement job with half-set splits (i.e., homogeneous or non-uniform refinement). Results may improve if per-particle scale factors have been optimized. Why does Particle Subtraction require half-sets?[](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta#why-does-particle-subtraction-require-half-sets) In particle subtraction, a volume is subtracted from all particle images. If this volume was reconstructed from all particle images, the resulting subtracted particle stack would contain information from all particles, breaking the gold-standard assumption of half-set independence. Thus, each particle must only have the half-map from its respective half-set subtracted. The particle subtraction job will internally ensure that this is satisfied. ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta#volume) Volume Volume must include both half maps. The volume must be the same size as the particle images. ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta#mask) Mask The mask should cover the region of the volume that is to be **subtracted from the images**. Like other masks which “cut through” map density, the mask must have a soft edge. However, if your mask is too large it may incorporate noise from regions of the map with no appreciable signal. Thus, the size of the mask must be carefully tuned. We recommend a minimum soft padding width of 5×resolutionapix5 \\times{} \\frac{\\mathrm{resolution}}{\\mathrm{apix}}5×apixresolution​ where **resolution** is the volume’s GSFSC resolution in Å and **apix** is the volume’s pixel size in Å, but you may need to test several masks to find the optimal result. More advice on mask making is available in [Mask Creation](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera) . [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta#common-problems) Common Problems -------------------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta#windowing-and-scaling) Windowing and scaling It is essential that the windowing parameters in a Particle Subtraction job match those of the input refinement. The volume is multiplicatively scaled to match the contrast of each particle image, and using different windowing parameters may result in an incorrect scaling factor. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta#common-next-steps) Common Next Steps ------------------------------------------------------------------------------------------------------------------------------------------------------------------ We recommend performing a [Homogeneous Reconstruction Only](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-homogeneous-reconstruction-only) using the subtracted particle sets as a sanity check that the correct parts of the image have been subtracted. If the sub-volume that was subtracted was aligned prior to performing the subtraction, it can be helpful to use the low-level interface to replace the poses of the subtracted particles with those from an earlier refinement in which the remaining density was properly aligned. Particle subtracted images are often most useful in [Local Refinement](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-new-local-refinement-beta) . [PreviousJob: Local Refinement](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-new-local-refinement-beta) [NextJob: Local Refinement (Legacy)](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-local-refinement-beta) Last updated 2 years ago * [At a Glance](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta#docs-internal-guid-ed02a34a-7fff-5241-a950-69249ac4630d) * [Description](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta#description) * [Inputs](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta#docs-internal-guid-07032c2a-7fff-89ef-1fa1-41a8f6c81924) * [Particles](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta#particles) * [Volume](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta#volume) * [Mask](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta#mask) * [Common Problems](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta#common-problems) * [Windowing and scaling](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta#windowing-and-scaling) * [Common Next Steps](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta#common-next-steps) --- # FAQs and Troubleshooting | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/live/faqs-and-troubleshooting.md) . [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#faqs) FAQs ---------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#general) General #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#who-should-use-cryosparc-live) Who should use CryoSPARC Live? CryoSPARC Live is built for anyone with access to a microscope or cryo-EM single particle data: * Data collection facilities, cryo-EM cores and microscope operators who want to make the most of microscope time with real-time data quality assessment * Individual users who want to gain insights about data quality by performing 3D reconstructions in real-time with data collection * Users who are looking for the simplest, fastest way to process previously collected data in a seamless and "first cut" manner CryoSPARC Live is available free of charge for academic use. #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#how-can-i-access-cryosparc-live) How can I access CryoSPARC Live? To access the Live web application, please see: [How to Access CryoSPARC Live](https://guide.cryosparc.com/live/how-to-access-cryosparc-live) CryoSPARC Live jobs are housed within CryoSPARC projects, so there is full integration between the two applications and you can continue processing in CryoSPARC using the outputs of Live. #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#what-are-the-hardware-requirements-for-live) What are the hardware requirements for Live? Please see [Prerequisites and Compute Resources Setup](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup) #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#what-kind-of-data-can-i-process-in-cryosparc-live) What kind of data can I process in CryoSPARC Live? CryoSPARC Live supports reading in movie files (`.tif`, `.mrc`, `.mrc.bz2` and `.eer`). For more information about EER file support in CryoSPARC, please see: EER File Support #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#how-fast-is-processing-in-cryosparc-live) How fast is processing in CryoSPARC Live? CryoSPARC Live can use multiple GPUs concurrently for pre-processing and reconstruction jobs to maximize achievable throughput. With 4 GPUs assigned to preprocessing (and sufficiently fast storage), cryoSPARC Live can sustain a rate of up to 1 exposure per 1.4 seconds, or over 60,000 exposures per 24-hours. For more performance metrics, please see: [Performance Metrics](https://guide.cryosparc.com/live/performance-metrics) #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#how-do-i-view-keyboard-shortcuts) How do I view keyboard shortcuts? Click on the Main Menu in any CryoSPARC Live Session and select "Keyboard Shortcuts" for a list of all keyboard shortcuts available. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNj0ojXjKtkh-omp_Lj%252F-MNj3nOknfL9aPIrtFJr%252FFAQ_1_Screen%2520Shot%25202020-11-27%2520at%25206.34.57%2520PM.png%3Falt%3Dmedia%26token%3D2ca1a848-8aa5-408c-a6d9-1bbaf995feb7&width=768&dpr=3&quality=100&sign=8e798f22&sv=2) View keyboard shortcuts via the main menu. ### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#processing-data-in-live) Processing Data in Live #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#i-am-processing-a-brand-new-dataset-and-i-am-not-sure-which-values-to-enter-for-minimum-and-maximum) I am processing a brand new dataset and I am not sure which values to enter for minimum and maximum particle diameter. In this case, start with crude estimates of your particle diameter based on the molecular weight of the molecule. These diameters are only used for blob picking, and will not affect downstream results other than the quality of picks. Once the Live session is started, use the picking tab and the exposure viewer to display your picks as circles (open the dropdown beside the pick counts at the top of the exposure viewer to change the display mode from dots to circles). These circles will be drawn at the same size as the particle diameter that you had entered. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNj0ojXjKtkh-omp_Lj%252F-MNj3qU0JFnMrZ7uhpd4%252FFAQ_2_csl_faq_1_picks.png%3Falt%3Dmedia%26token%3Db8e40cc9-edad-4cb0-ad29-8de6c79009ea&width=768&dpr=3&quality=100&sign=cf9e359&sv=2) Based on this, choose new values for the minimum and maximum particle diameter, and use Test mode (click the 'Test' button on the picking tab) to reprocess the currently selected exposure using the new parameters that you provided. Once this is complete (you must wait for the selected micrograph to be picked up and processed by a Live worker) you can visualize the new picks, and adjust thresholds to determine the quality of the picks using these new parameters. You can repeat test mode as many times as needed to achieve good picking results.The same strategy can also be used with the template picker. Once you have picks that are reasonable for your data, be sure to also adjust the extraction box size on the configuration tab based on your new knowledge of the particle diameter, and "Apply to All' so that all exposures are re-extracted at the new box size. #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#i-enabled-an-exposure-group-but-later-realized-it-was-wrongly-configured.-how-can-i-change-this) I Enabled an exposure group but later realized it was wrongly configured. How can I change this? Click `New` to add a new Exposure Group. Configure this group as desired and click `Enable`. Then, `Remove` the incorrect group. CryoSPARC Live requires a minimum of one exposure group to be enabled at all times; you cannot remove an exposure group if it is the only one that exists. **Note:** exposure groups can only be removed if they do not have any exposures already found or processed within them. You can ignore all the exposures from a group that has already been established, or you can clear the session. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNj0ojXjKtkh-omp_Lj%252F-MNj3tTUO4HpqbbxbFfr%252FFAQ_3_Screen%2520Shot%25202020-11-27%2520at%25206.32.04%2520PM.png%3Falt%3Dmedia%26token%3Da4d92458-0392-443b-ad04-5156885c6da3&width=768&dpr=3&quality=100&sign=730c8605&sv=2) Press the "Remove" button to delete the exposure group. This button is only enabled if there are more than one exposure groups available. #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#where-can-i-see-job-status-progress) Where can I see job status / progress ? Jobs will update their status and relevant outputs (e.g., plots) in the CryoSPARC Live interface on their respective tabs. For more details, click into the job icon for any job to view its streamlog directly within the Live interface. Finally, you can always interact with Live jobs through your regular CryoSPARC interface: navigate to the Project where the Live Session is housed and click on the job number or press `SPACEBAR` to open the streamlog. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNj0ojXjKtkh-omp_Lj%252F-MNj3vqgJdxC6U27GVMK%252FFAQ_4_Screen%2520Shot%25202020-11-27%2520at%25206.29.07%2520PM.png%3Falt%3Dmedia%26token%3Dfd19bf81-3d63-467f-b806-6b4490edbe49&width=768&dpr=3&quality=100&sign=5b9aaf34&sv=2) Click on the "+" icon while in the Single Session page to view a job's streamlog directly. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNj0ojXjKtkh-omp_Lj%252F-MNj3xwfr16BsBZhgTGv%252FFAQ_5_Screen%2520Shot%25202020-11-27%2520at%25206.28.47%2520PM.png%3Falt%3Dmedia%26token%3D145c1130-fd81-42cd-97c5-5e83b9ca112b&width=768&dpr=3&quality=100&sign=8acf8217&sv=2) Click on the job icon to view its streamlog directly. #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#where-can-i-see-and-interact-with-job-outputs) Where can I see and interact with job outputs? You can interact with Live jobs through your regular CryoSPARC interface and download/use their outputs for further processing. Navigate to the Project/workspace where the Live Session is housed to view all Live jobs. 3D maps can be download directly from the Volume Viewer in the CryoSPARC Live interface. #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#the-sidebar-shows-that-that-a-few-of-my-exposures-have-failed.-how-can-i-find-which-ones-have-failed) The sidebar shows that that a few of my exposures have failed. How can I find which ones have failed? You can scroll through the exposure feed and look for the "rejected" icon on any thumbnails that have failed. Alternatively, navigate to the Browse Tab and filter by 'Failed\` exposures to view a list of all failed exposure UIDs. Click on any row to view that exposure. #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#i-forgot-to-flip-the-gain-reference-file-and-the-session-is-already-running.-how-can-i-fix-this) **I forgot to flip the gain reference file and the session is already running. How can I fix this?** Navigate to the Configuration Tab and ensure the "Show advanced" checkbox is selected. Under Microscope/Camera Parameter, adjust the toggles for flipping the gain reference in X or Y as required. Once you change the parameter, click `Apply to All` or `Apply to Future` to save the change and cause the CryoSPARC Live GPU Workers to re-process the exposures with the new parameter. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNj0ojXjKtkh-omp_Lj%252F-MNj4-b55SZJX0_JKk67%252FFAQ_6_Screen%2520Shot%25202020-11-27%2520at%25206.41.02%2520PM.png%3Falt%3Dmedia%26token%3D9b15ad8b-ffd8-4205-ba8c-ca4766ada0fd&width=768&dpr=3&quality=100&sign=8c6ddf42&sv=2) Use the search bar to quickly find the parameter you're looking for. #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#how-do-i-switch-from-blob-picking-to-template-picking-or-vice-versa-how-do-i-re-extract-particles-ba) How do I switch from Blob Picking to Template Picking or vice versa? How do I re-extract particles based on my new picking settings? The Blob Picker is the default active picker in CryoSPARC Live and is engaged as soon as exposures start to be processed. To engage the Template Picker at any time during the session, follow these instructions: [New Live Session: Start to Finish Guide](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#template-picker) #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#i-want-to-change-the-extraction-box-size) I want to change the extraction box size. During a running Session, you can navigate to the Configuration Tab at any time to change the extraction box size for any of the pickers. Once you change the parameter, click `Apply to All` to save the change and cause the CryoSPARC Live GPU Workers to re-extract all particles with the new box size. Note that you should not use `Apply for Future` in this case, as only the new box size particles will be used in streaming processing. #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#i-want-to-specify-custom-parameters-for-a-live-job) I want to specify custom parameters for a Live job. You can specify custom parameters for Streaming 2D Classification, Ab-Initio Reconstruction and Streaming Refinement jobs. To do so, navigate to the respective tab from the sidebar and click on the Gear icon. Click on the build icon (hammer) to Build with custom parameters. This will create a new job in building status. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNj0ojXjKtkh-omp_Lj%252F-MNj49VzzYkG8uSgZPRY%252FFAQ_7_Screen%2520Shot%25202020-11-27%2520at%25206.50.53%2520PM.png%3Falt%3Dmedia%26token%3D849534c1-0292-4919-ae9a-e63484e0e2bd&width=768&dpr=3&quality=100&sign=7df0817e&sv=2) Click on the build icon to create a new job that you can modify parameters for. Navigate to the CryoSPARC interface (to the Project where this Live Session is housed). Edit the parameters of the building job and then **return to the Live interface** to queue/start the job. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNj0ojXjKtkh-omp_Lj%252F-MNj4BiY3EFAHlvzMTou%252FFAQ_8_Screen%2520Shot%25202020-11-27%2520at%25206.51.21%2520PM.png%3Falt%3Dmedia%26token%3D6d799f69-deaf-43b1-ab2d-9641973f9118&width=768&dpr=3&quality=100&sign=2517980d&sv=2) Click "Queue" to schedule your job with custom parameters. #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#i-manually-rejected-an-exposure.-how-do-i-un-reject) I manually rejected an exposure. How do I un-reject? Select the manually rejected exposure and navigate to the **Individual Exposure** Tab. To un-reject the exposure individually, click on the dropdown menu above the Exposure Viewer and click **Un-reject Exposure,** OR use the keyboard shortcut **"R"**. To reset all manually rejected exposures, click on the dropdown menu in the header and click **Reset manual rejected exposures**. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNj0ojXjKtkh-omp_Lj%252F-MNj4E1rziGcZ8ohAFzX%252FFAQ_9_Screen%2520Shot%25202020-12-01%2520at%25203.50.00%2520PM.png%3Falt%3Dmedia%26token%3Dcb9ef6b5-141f-4472-b144-4ecb417b01a4&width=768&dpr=3&quality=100&sign=4e6358d8&sv=2) #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#what-does-test-adjustments-do) What does Test Adjustments do? Test Adjustments is useful for fine-tuning picking parameters and thresholds. If you are not sure how your parameter changes might affect processing or if you would like to experiment, you can use **'Test Adjustments'** which will cause only the currently selected exposure to be re-picked and re-extracted with the new picking parameter changes. Once this process is complete, the new pick locations will appear on the active micrograph. Exposures to which a **Test** parameter has been applied are indicated with a purple "**T"** on their respective thumbnails. You can apply '**Test Adjustments'** on as many individual exposures as you like. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNj0ojXjKtkh-omp_Lj%252F-MNj4GSvZHSlkrZBLtfk%252FFAQ_10_Screen%2520Shot%25202020-12-01%2520at%25204.11.33%2520PM.png%3Falt%3Dmedia%26token%3D9d218f16-bae9-47b9-8e4e-59995338f78a&width=768&dpr=3&quality=100&sign=750b0695&sv=2) An example Test exposure Once satisfied with the new picker settings, click **'Activate for All'** or '**Activate for Future'** as desired. This will trigger re-picking and re-extraction. Unless one of the **Activate** buttons is clicked following **Test Adjustments**, the exposure on which **Test Adjustments** was run will simply be excluded from any further processing (i.e., from Streaming 2D Classification and Streaming 3D Refinement). To undo **Test Adjustments** mode on a particular exposure, i.e., to reset it so that it can be included in further processing, click on the dropdown above the Exposure Viewer and click Reprocess exposure. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNj0ojXjKtkh-omp_Lj%252F-MNj4Ihlce4EcDAiPeAd%252FFAQ_11_Screen%2520Shot%25202020-12-01%2520at%25204.12.14%2520PM.png%3Falt%3Dmedia%26token%3D451066f8-c6f8-4cb4-b0d1-b58ae5a53e2d&width=768&dpr=3&quality=100&sign=51daa2da&sv=2) #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#when-and-how-often-will-my-streaming-2d-classes-update) When and how often will my Streaming 2D classes update? 2D classes update in two ways. First, when new particles are available, those particles are classified into the existing classes, and the number of particles in each class is updated. This allows the new particles to be filtered based on class selection, and selected particles flow on to 3D refinement. This can be ver fast, taking only seconds. Second, once the number of newly classified particles accumulates enough, the templates themselves are updated with the signal from the new particles, and all existing particles are re-classified based on the new templates. This can take a few minutes depending on how many particles are extracted. #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#when-and-how-often-will-my-streaming-3d-refinement-update) When and how often will my Streaming 3D Refinement update? 3D refinement updates with a new 3D structure when enough new particles have arrived from 2D classification. At this point, the 3D reference structure currently resolved is backtracked to a lower resolution, and all existing particles (including new particles) are used to refine the structure to high resolution until convergence. This can take several minutes depending on the number of particles, box size, symmetry, and convergence rate for the given dataset. #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#how-can-i-download-3d-volume-s-generated-in-live) How can I download 3D volume(s) generated in Live? You can download 3D volumes directly from the Volume Viewer in the Live interface. Alternatively, you can download them from within the CryoSPARC Project where the Live Session is housed. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNj0ojXjKtkh-omp_Lj%252F-MNj4LkuZrXEC-ZqnOur%252FFAQ_12_Screen%2520Shot%25202020-12-01%2520at%25204.16.21%2520PM.png%3Falt%3Dmedia%26token%3De0ed93e7-c568-41c6-a1ed-94c66637a4b7&width=768&dpr=3&quality=100&sign=4a922098&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNj0ojXjKtkh-omp_Lj%252F-MNj4NuyAgFZVjlq33DJ%252FFAQ_13_image.png%3Falt%3Dmedia%26token%3D59b99c8d-31d5-403a-bb33-0dfcbb76fff9&width=768&dpr=3&quality=100&sign=6cb0956c&sv=2) Download map directly from the Volume Viewer. ### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#compute-configuration) Compute Configuration #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#i-want-to-increase-or-decrease-the-number-of-gpus-being-used-for-preprocessing-in-my-live-session) I want to increase or decrease the number of GPUs being used for preprocessing in my Live Session. Pause the Session. This will cause any running CryoSPARC Live GPU Workers to complete their current task and spin down. Any auxiliary and reconstruction jobs (Generate Templates, Ab-Initio Reconstruction, Streaming 2D Classification and Streaming Refinement) will be killed and marked as completed, so their outputs can be used to potentially resume processing. Make any adjustments to the compute configuration page and then Start the session again. This will spin up new CryoSPARC Live GPU Worker jobs using the values set. Any Auxiliary or Reconstruction jobs will need to be manually restarted via their respective tabs. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNj0ojXjKtkh-omp_Lj%252F-MNj4RymPpaIMCsz0NoA%252FFAQ_14_Screen%2520Shot%25202020-12-01%2520at%25204.27.22%2520PM.png%3Falt%3Dmedia%26token%3Da15d0f7c-ed38-4f3c-97b3-59700cf8f97b&width=768&dpr=3&quality=100&sign=cb557a7f&sv=2) #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#if-i-pause-a-session-now-and-come-back-to-it-later-can-i-pick-up-where-i-left-off) **If I Pause a session now and come back to it later, can I pick up where I left off?** If you Pause a session, any active CryoSPARC Live GPU Workers will be spun down and any auxiliary and reconstruction jobs (Generate Templates, Ab-Initio Reconstruction, Streaming 2D Classification and Streaming Refinement) will be killed and marked as completed, so their outputs can be used to potentially resume processing. When you restart the session, remaining exposures will start to process where they left off, regardless of the stage they finished at. You will need to manually go to the 2D class and 3D refinement tabs to start/resume those processes. ### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#data-management) Data Management #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#how-can-i-safely-delete-cryosparc-live-jobs-or-outputs) How can I safely delete CryoSPARC Live jobs or outputs? Like other processing jobs, CryoSPARC Live jobs generate intermediate results and metadata. To free up space, users may wish to archive or permanently delete some of these outputs. CryoSPARC v3.0.0+ includes a new toolset for managing, archiving and deleting data from cryoSPARC Live sessions. Please see the detailed tutorial here: [CryoSPARC Live Session Data Management](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/cryosparc-live-session-data-management-4.7) #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#i-am-finished-with-the-live-workflow-and-i-want-to-continue-processing-in-cryosparc-or-elsewhere) I am finished with the Live workflow and I want to continue processing in CryoSPARC or elsewhere. Please see View/interact with Outputs and Perform Further Processing: [New Session: Start to Finish Guide](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide) [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#troubleshooting) Troubleshooting -------------------------------------------------------------------------------------------------- Please also see our CryoSPARC Troubleshooting page for general CryoSPARC issues: [Troubleshooting](https://guide.cryosparc.com/setup-configuration-and-management/troubleshooting) #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#my-live-session-is-running-but-one-or-more-of-my-jobs-failed.-how-can-i-troubleshoot-and-or-resume-p) My Live Session is running but one or more of my jobs failed. How can I troubleshoot and/or resume processing? Open the job streamlog by clicking on the job number within the CryoSPARC Live interface or navigate to the CryoSPARC interface to do the same. * `CryoSPARC Live GPU Workers`: If these fail, you can Pause the Session and then Start it again. Any exposure that haven't been preprocessed (i.e., are found but haven't had all of Motion Correction, CTF Estimation, Particle Picking or Extraction performed on them yet, or are new incoming exposures) will be picked up normally by the new CryoSPARC Live GPU Workers that are spun up, and will be processed accordingly. * `CryoSPARC Live Exposure Export` or `CryoSPARC Live Particle Export`: If these jobs fail, you can simply try to run them again without Pausing the session. * Auxiliary Jobs * Generate Templates: If this job fails, you will need to start it again by clicking "Generate Templates" once again * Ab-Initio Reconstruction: If this job fails, you will need to start it again by clicking "Restart" * Streaming Jobs (Streaming 2D Classification and Streaming Refinement): If these jobs fail, you will need to "Restart" or "Attempt Resume" #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#one-or-more-of-my-exposures-failed-to-process.-how-do-i-reset) One or more of my exposures failed to process. How do I reset? Select the failed exposure and navigate to the Individual Exposure Tab. You can view the traceback here. To reset the exposure individually, click on the dropdown menu above the Exposure Viewer and click "Reprocess Exposure". To reset all failed exposures, click on the dropdown menu in the header and click Reset Failed Exposures. #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#my-live-jobs-are-queued-with-gpu-not-available) My Live jobs are queued with 'GPU not available'. This is most likely because there are insufficient GPU resources allocated to each lane in the CryoSPARC Live compute configuration. Pause the session and navigate to the configuration tab to adjust compute settings, then click Start to continue processing. We recommend 4 GPUs be allocated for Live processing for the ability to perform preprocessing and real-time 2D/3D processing during a data collection session: [Prerequisites and Compute Resources Setup](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup) #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#how-can-i-view-queued-and-running-jobs) How can I view queued and running jobs? Click on the footer in CryoSPARC Live for a summary of all running and queued Live jobs. CryoSPARC Live jobs are also shown in the CryoSPARC Resource Manager**.** #### [](https://guide.cryosparc.com/live/faqs-and-troubleshooting#where-can-i-get-more-help-for-cryosparc-live) Where can I get more help for CryoSPARC Live? Please post on the [CryoSPARC Discussion Forum](http://discuss.cryosparc.com/) under the CryoSPARC Live category. [PreviousManaging a CryoSPARC Live Session from the CLI (v5.0+)](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli-v5.0) [Nextv3 User Interface Guide](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide) Last updated 1 year ago * [FAQs](https://guide.cryosparc.com/live/faqs-and-troubleshooting#faqs) * [General](https://guide.cryosparc.com/live/faqs-and-troubleshooting#general) * [Processing Data in Live](https://guide.cryosparc.com/live/faqs-and-troubleshooting#processing-data-in-live) * [Compute Configuration](https://guide.cryosparc.com/live/faqs-and-troubleshooting#compute-configuration) * [Data Management](https://guide.cryosparc.com/live/faqs-and-troubleshooting#data-management) * [Troubleshooting](https://guide.cryosparc.com/live/faqs-and-troubleshooting#troubleshooting) --- # About CryoSPARC Live | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/live/about-cryosparc-live.md) . [](https://guide.cryosparc.com/live/about-cryosparc-live#from-microscope-to-structure-in-minutes) From microscope to structure, in minutes ----------------------------------------------------------------------------------------------------------------------------------------------- CryoSPARC Live is a software platform that enables: * Real-time cryo-EM data quality assessment * Decision making based on 2D and 3D results during live data collection * An expedited, streamlined workflow for processing previously collected data * Direct seamless interoperation with CryoSPARC for advanced processing **CryoSPARC Live is built to enable experimentation with parameters on the fly, while the software efficiently manages the reprocessing that is necessary in order to test or effect parameter changes, and while maintaining the overall progress of the Live session.** Processing EMPIAR-10288 in cryoSPARC Live. [](https://guide.cryosparc.com/live/about-cryosparc-live#who-is-cryosparc-live-built-for) Who is CryoSPARC Live built for? ------------------------------------------------------------------------------------------------------------------------------- CryoSPARC Live is built for: * **Data collection facilities, cryo-EM cores and and microscope operators**, who want to make the most of microscope time with real-time data quality assessment and collection; * **Facilities and service providers**, who want to provide their users with expedited information about sample quality and/or delivery of 3D maps; and * **Individual users**, who wish to gain insights about data quality by performing 3D reconstructions in real-time with data collection, or on previously collected data in a seamless and "first cut" manner. [](https://guide.cryosparc.com/live/about-cryosparc-live#use-cases-for-cryosparc-live) Use cases for CryoSPARC Live ------------------------------------------------------------------------------------------------------------------------ ### [](https://guide.cryosparc.com/live/about-cryosparc-live#live-processing-at-microscope-during-data-collection) Live processing at microscope, during data collection CryoSPARC Live can be used by microscope operators during a collection session. We recommend the exposures are written to fast disks as I/O can become a bottleneck. ### [](https://guide.cryosparc.com/live/about-cryosparc-live#live-processing-offsite-during-data-collection) Live processing offsite, during data collection CryoSPARC Live can be engaged to read files that are being uploaded to the user, e.g., via AWS or other similar service. Live will read in new exposures as they are found. ### [](https://guide.cryosparc.com/live/about-cryosparc-live#seamless-first-cut-processing-of-previously-collected-data) Seamless first cut processing of previously-collected data Finally, we recommend using the Live workflow for expedited preprocessing (see below) and a first-cut look at data quality in 2D and 3D, for all datasets including those already collected. [](https://guide.cryosparc.com/live/about-cryosparc-live#what-does-cryosparc-live-enable) **What does CryoSPARC Live enable?** ----------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/live/about-cryosparc-live#continuous-import-and-expedited-preprocessing) Continuous import and expedited preprocessing ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNinkMOQaRg__Tep1YV%252F-MNiohSEbV2aYK0W8eC3%252FAboutCSL_1_2_preprocessing_thumbs.png%3Falt%3Dmedia%26token%3D4068274c-e062-4c91-b25d-13e42b985c45&width=768&dpr=3&quality=100&sign=6e21a5ee&sv=2) CryoSPARC Live watches specified directories for new files and processes them as they become available. CryoSPARC Live preprocessing includes four steps: motion correction, CTF estimation, particle picking and extraction. CryoSPARC Live can sustain a throughput of 450 or more exposures per hour, per GPU, for K3 data. On a 4-GPU machine, that can scale to 1800+ exposures per hour! For K2 or Falcon data, performance can be even higher, upwards of 650 exposures per hour per GPU. Particles from preprocessing are seamlessly transitioned into 2D classification and 3D reconstruction, and preprocessed exposures can also be exported for further processing in CryoSPARC or other software. For extensive benchmarks and throughput statistics, please see: [Performance Metrics](https://guide.cryosparc.com/live/performance-metrics) ### [](https://guide.cryosparc.com/live/about-cryosparc-live#adjustable-parameters-and-saveable-configurations) Adjustable parameters and saveable configurations * **Microscope/camera and job parameters:** Any of these can be adjusted over the course of a session, and combinations of parameters can be saved as "[Configuration Profiles](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#load-configuration-profile) " for use in future sessions. For example, microscope parameters such as the `pixel size`, `spherical aberration` and `accelerating voltage` are likely to be consistent for a given instrument and can be applied quickly using Configuration Profiles at the start of a new Session. If working on multiple samples of the same protein or complex, it may be useful to save picking and extraction parameters to save time. * **Exposure groups:** It is possible to add, remove, and ignore exposures from one or more Exposure Groups (collections of exposures with the same optical parameters). This makes it possible to process subsets of a larger dataset and make comparisons. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNinkMOQaRg__Tep1YV%252F-MNiojmcQ4QAF3-q5wE_%252FAboutCSL_2_csl_config_profiles_eg.png%3Falt%3Dmedia%26token%3Df30b2a09-271f-4970-9996-888b643c176c&width=768&dpr=3&quality=100&sign=acf8c44b&sv=2) Learn more about Configuration Profiles in the UI Overview: [UI Overview](https://guide.cryosparc.com/live/ui-overview) ### [](https://guide.cryosparc.com/live/about-cryosparc-live#streamlined-exposure-curation-threshold-based-and-manual) Streamlined exposure curation (threshold-based and manual) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNinkMOQaRg__Tep1YV%252F-MNiomLuMnzMYfaKKWVS%252FAboutCSL_3%293_overview_threshold_1.png%3Falt%3Dmedia%26token%3Db2c96e17-bae6-47ff-86fb-93abbc98b6ce&width=768&dpr=3&quality=100&sign=f3a2b620&sv=2) Users can review several useful statistics (e.g., CTF Fit, Defocus, Total Motion, etc) across a dataset and apply thresholds to automatically reject incoming and already-processed exposures that do not fall within the desired range for one or more parameters. Exposures can also be rejected manually. Rejected exposures are excluded from further processing unless thresholds are changed, which can be done at any time during a Live session. On export, exposures are split into different output groups (e.g., rejected and accepted) for further advanced processing outside of the Live workflow. ### [](https://guide.cryosparc.com/live/about-cryosparc-live#ability-to-test-and-refine-picking-strategies-while-collection-is-ongoing) Ability to test and refine picking strategies while collection is ongoing It is possible to have finalized a picking strategy within the first few hundred exposures, which can then be left to run on the remaining exposures as they come in. Blob-based picking is active by default at the start of any Live session and manual picking can be engaged at any time. Blob/manual picks can be curated and used to generate templates for template-based picking, or available templates can be loaded directly into the Session. Pick scores (NCC and Power Threshold) can be used to include or exclude picks. These and particle extraction parameters can be tested on a single or few exposures before being applied to the entire dataset, and can be updated as many times as necessary. Updated particle locations will be fed automatically into later stages of the pipeline (e.g., Streaming 2D Classification). ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNinkMOQaRg__Tep1YV%252F-MNiopWZ2UpPRV34aDP6%252FAboutCSL_4_csl_pick_adjust_1.png%3Falt%3Dmedia%26token%3D3fd16970-0db6-4120-9ec5-23a371cdeded&width=768&dpr=3&quality=100&sign=4e947886&sv=2) ### [](https://guide.cryosparc.com/live/about-cryosparc-live#make-go-no-go-decisions-about-a-sample-using-streaming-2d-classification) **Make go/no go decisions about a sample using Streaming 2D classification** **Real time streaming 2D classification enables assessing sample quality, preferred orientation issues and presence/absence of the expected target and/or ligands, as well as large conformational variability.** 2D Classification in CryoSPARC Live picks up newly extracted particles from incoming exposures and automatically updates 2D classes every few minutes using a new streaming method. This means that after starting a Streaming 2D Classification job, class averages will update automatically as new particles become available from upstream. Class averages can be selected as soon as the initial classification is complete, and these selections will be retained, enabling Streaming 3D Refinement to take in new particles from the selected classes and update the reconstruction in real time. Early feedback in 2D (and 3D, see below) can confirm whether a collection should continue, or whether upstream steps such as sample preparation may require improvement. Within a few hours of starting a Live session, it is possible to make a "go/no go" decision about the sample and assess issues that may result in a poor 3D reconstruction. Thus, it is possible to save on microscope time, or, to at least inform the user ahead of time about the quality of result they may be able to expect. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNinkMOQaRg__Tep1YV%252F-MNiosa-jqUFi5_bFS_9%252FAboutCSL_5_csl_strm_2d_1.png%3Falt%3Dmedia%26token%3Dbfed1716-0cc9-4913-93e4-aafac3708093&width=768&dpr=3&quality=100&sign=9c048f1e&sv=2) Learn more about CryoSPARC Live Jobs and Session-Level Functions: [Live Jobs and Session-Level Functions](https://guide.cryosparc.com/live/live-jobs-and-session-level-functions) ### [](https://guide.cryosparc.com/live/about-cryosparc-live#id-3d-reconstruction-and-streaming-3d-refinement-during-data-collection) **3D reconstruction and Streaming 3D refinement during data collection** The ability to generate a refinement during data collection is important as a diagnostic during collection, as a first-cut structure from which to continue further processing, and in many cases is comparable to the highest resolution structure(s) that can be generated after extensive advanced processing in CryoSPARC. Streaming 3D Refinement in CryoSPARC Live also will pick up newly available particles from Streaming 2D Classification so that over the course of a collection, the 3D structure updates every few minutes. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNinkMOQaRg__Tep1YV%252F-MNiovl-x6x9e-Tq7QT7%252FAboutCSL_6_csl_intro_3d_ref.png%3Falt%3Dmedia%26token%3D36bbf8cc-c7da-46f6-83b5-2f4e1495ad57&width=768&dpr=3&quality=100&sign=d5c5746b&sv=2) For a detailed walkthrough of setting up your own Live session, please see: [New Live Session: Start to Finish Guide](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide) ### [](https://guide.cryosparc.com/live/about-cryosparc-live#add-or-free-up-compute-resources-during-a-session) Add or free up compute resources during a Session Over the course of a CryoSPARC Live Session, it is possible to adjust the compute resources dedicated to processing. The `Number of Preprocessing GPU Workers` can be increased or decreased during a session in order to free up compute resources or add more parallelization capacity for preprocessing. Additionally, the compute lanes being used can be adjusted. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNj0ojXjKtkh-omp_Lj%252F-MNj4RymPpaIMCsz0NoA%252FFAQ_14_Screen%2520Shot%25202020-12-01%2520at%25204.27.22%2520PM.png%3Falt%3Dmedia%26token%3Da15d0f7c-ed38-4f3c-97b3-59700cf8f97b&width=768&dpr=3&quality=100&sign=cb557a7f&sv=2) Learn more about GPU requirements for Live: [Prerequisites and Compute Resources Setup](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup) ### [](https://guide.cryosparc.com/live/about-cryosparc-live#export-of-results-and-integration-with-cryosparc) Export of results and integration with cryoSPARC CryoSPARC Live is tightly integrated with CryoSPARC. Each CryoSPARC Live Session is housed within a CryoSPARC Project, so the results of live processing can always be used seamlessly for further advanced processing in CryoSPARC as well as for export. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNinkMOQaRg__Tep1YV%252F-MNip-oN7ap0HrHA4VjC%252FAboutCSL_8_csl_intro_export_integ.png%3Falt%3Dmedia%26token%3D4fdd3619-5fee-417f-9c8f-2e0677e29da7&width=768&dpr=3&quality=100&sign=e70ef9d1&sv=2) It is common to take the final map(s), exposures and particle stack(s) from CryoSPARC Live and hand them off directly to e.g., users of a microscope facility or lab, or to use the motion-corrected, CTF-estimated exposures directly for advanced processing without redoing preprocessing steps. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNinkMOQaRg__Tep1YV%252F-MNip1wVBWkl3G63CA4h%252FAboutCSL_9_csl_intro_download_map_1.png%3Falt%3Dmedia%26token%3Db13db80a-94b9-441e-b345-75b91ebb9cad&width=768&dpr=3&quality=100&sign=ef93daa2&sv=2) ### [](https://guide.cryosparc.com/live/about-cryosparc-live#programmatic-control-of-cryosparc-live) Programmatic control of CryoSPARC Live CryoSPARC Live Sessions and jobs can also be controlled via the command line: [Managing a CryoSPARC Live Session from the CLI (≤v4.7)](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli) [](https://guide.cryosparc.com/live/about-cryosparc-live#what-users-say) What users say -------------------------------------------------------------------------------------------- "CryoSPARC Live enables the Pacific Northwest Cryo-EM Center to keep up with data coming from all 5 of our microscopes, a data volume that has at times exceeded 15TB (30,000+ images) per day. We can monitor not only standard image pre-processing metrics, but also directly observe improvements to 3D maps concurrent with data collection! In many cases we’ve stopped data collection with a 2.2-3.5Å map already in hand. We can also quickly identify pathologies such as poor orientational distributions, or tune imaging conditions to improve alignment of difficult particles." **\- Craig Yoshioka, Center Co-Director, Pacific Northwest Center for Cryo-EM (PNCC)** In our facility, CryoSPARC Live has completely revolutionised the way we collect and process cryoEM data. Initially, we commissioned CryoSPARC Live with the primary goal of feedback on micrograph quality through on-the-fly motion correction and CTF estimation. But it has turned out to be so much more than we hoped for. We now use it for all initial processing of data coming through our facility. Our latest GPU workstation was specifically specced out for rapid CryoSPARC processing as it has become our primary in-house processing tool. Some of the aspects we love about CryoSPARC Live include: the GUI is intuitive, easy to use, and easy to learn for new users; processing is extremely fast; parameter changes are propagated throughout the workflow in a streaming fashion; easy linkages with ‘regular’ CryoSPARC make continued processing simple; picking tools are easy to use and work for almost all targets, and micrograph curation is powerful and easy to apply. CryoSPARC has become such an essential part of our data collection and analysis workflow; we could not imagine working without it. Highly valuable microscope time is now used much more efficiently and our microscope users are very satisfied. **\- Simon HJ Brown, PhD, Customer Solutions Expert, Cryo Electron Microscopy - Molecular Horizons, University of Wollongong** "CryoSPARC live may literally make your jaw drop. The speed of processing combined with the capability to make adjustments in real time are remarkable. It allows you to make go/no go decisions for each experiment using 2D and 3D results generated almost as fast as the images can be collected. Perhaps most impressive is the ability to go back and change almost any parameter on the fly, and it will immediately reprocess the images from that step, and key metrics such as motion and CTF fit can be examined in interactive plots where ranges can be set and immediately applied for image curation. Together these features take a large step toward the future in reducing both the time and resources that are needed to achieve a successful outcome for each data collection." **\- Jeff Speir, Director of Operations, NanoImaging Services Inc. (NIS)** "My group and I have been using CryoSPARC Live beta for a little over a year with great successes. We currently only run CryoSPARC Live for our in house projects and it has been of great help. It is easy to set up, very easy to use and very fast. There is no need to create scripts or any slightly complicated task. On most projects with very limited human interaction and no knowledge of the target of interest we can reach high resolution overnight before the run on the microscope is finished. The CryoSPARC-live features allow us to monitor the quality of the acquisition and make modifications in real time if necessary, whether it is a sample problem (e.g. ice thickness) or an issue with the alignment of the microscope. We are looking forward to extend the use to the core facility users allowing them to get real time feedback on their runs." **\- Eric Hanssen, Head, Advanced Microscopy Facility and Associate Professor, University of Melbourne** "CryoSPARC Live is a wonderful tool that not only gets researchers excited about their cryoEM experiments but lets microscope operators know they are acquiring high-quality processable data. With on-the-fly feedback users and staff are able to engage with each other to identify bottlenecks and modify data collection strategies in real-time to conduct optimized experiments for a sample. 3D feedback is critical because it ensures we are able to collect a full dataset of their macromolecule of interest, thereby allowing our users to accelerate their biomedical research." **\- Edward Eng, Manager, New York Structural Biology Center (NYSBC)** "Having used CryoSPARC Live at both New York Structural Biology Center (USA) and The Hospital for Sick Children (Canada), it has become an invaluable tool in real time assessment of samples for a frequent user like me. By having the movie frames aligned, CTF estimated, particles picked, 2D classification and ab initio done on the fly allows me to quickly judge which grid/sample is worth collecting on. As all the statistics can also be neatly presented as an overview, it is straightforward too to pick out trends and exceptions in the data: For instance I have noticed cases where a bunch of micrographs had poor estimated resolution, only to pinpoint the problem to a single grid square, allowing me to avoid similar grid squares subsequently. The final big advantage of CryoSPARC Live is that the resulting data can easily be passed onto the conventional CryoSPARC pipeline for further processing – saving both time and energy!" **\- Yong Zi Tan, Postdoctoral Fellow, The Hospital for Sick Children** [](https://guide.cryosparc.com/live/about-cryosparc-live#history-and-development) History and development -------------------------------------------------------------------------------------------------------------- CryoSPARC Live was first released as a private beta in May 2019. Based on extensive beta testing at dozens of facilities and labs globally, we have incorporated feedback and worked to improve the workflow, with several iterations already released. CryoSPARC Live will continue to evolve with advancements in data collection, user feedback and automation of the cryo-EM workflow. ### [](https://guide.cryosparc.com/live/about-cryosparc-live#embedded-cryosparc-live) Embedded CryoSPARC Live In 2022, we collaborated with Thermo Fisher Scientific Inc. to make available Embedded CryoSPARC Live, a version of CryoSPARC Live that is designed to seamlessly integrated with Thermo ScientificTM cryo-transmission electron microscope systems. To learn more about the collaboration, please visit: [https://cryosparc.com/embedded-cryosparc-live](https://cryosparc.com/embedded-cryosparc-live) [](https://guide.cryosparc.com/live/about-cryosparc-live#get-started) Get Started -------------------------------------------------------------------------------------- [Prerequisites and Compute Resources Setup](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup) [How to Access CryoSPARC Live](https://guide.cryosparc.com/live/how-to-access-cryosparc-live) [New Live Session: Start to Finish Guide](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide) [FAQs and Troubleshooting](https://guide.cryosparc.com/live/faqs-and-troubleshooting) [PreviousWebinar Recordings](https://guide.cryosparc.com/processing-data/webinar-recordings) [NextPrerequisites and Compute Resources Setup](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup) Last updated 4 months ago * [From microscope to structure, in minutes](https://guide.cryosparc.com/live/about-cryosparc-live#from-microscope-to-structure-in-minutes) * [Who is CryoSPARC Live built for?](https://guide.cryosparc.com/live/about-cryosparc-live#who-is-cryosparc-live-built-for) * [Use cases for CryoSPARC Live](https://guide.cryosparc.com/live/about-cryosparc-live#use-cases-for-cryosparc-live) * [Live processing at microscope, during data collection](https://guide.cryosparc.com/live/about-cryosparc-live#live-processing-at-microscope-during-data-collection) * [Live processing offsite, during data collection](https://guide.cryosparc.com/live/about-cryosparc-live#live-processing-offsite-during-data-collection) * [Seamless first cut processing of previously-collected data](https://guide.cryosparc.com/live/about-cryosparc-live#seamless-first-cut-processing-of-previously-collected-data) * [What does CryoSPARC Live enable?](https://guide.cryosparc.com/live/about-cryosparc-live#what-does-cryosparc-live-enable) * [Continuous import and expedited preprocessing](https://guide.cryosparc.com/live/about-cryosparc-live#continuous-import-and-expedited-preprocessing) * [Adjustable parameters and saveable configurations](https://guide.cryosparc.com/live/about-cryosparc-live#adjustable-parameters-and-saveable-configurations) * [Streamlined exposure curation (threshold-based and manual)](https://guide.cryosparc.com/live/about-cryosparc-live#streamlined-exposure-curation-threshold-based-and-manual) * [Ability to test and refine picking strategies while collection is ongoing](https://guide.cryosparc.com/live/about-cryosparc-live#ability-to-test-and-refine-picking-strategies-while-collection-is-ongoing) * [Make go/no go decisions about a sample using Streaming 2D classification](https://guide.cryosparc.com/live/about-cryosparc-live#make-go-no-go-decisions-about-a-sample-using-streaming-2d-classification) * [3D reconstruction and Streaming 3D refinement during data collection](https://guide.cryosparc.com/live/about-cryosparc-live#id-3d-reconstruction-and-streaming-3d-refinement-during-data-collection) * [Add or free up compute resources during a Session](https://guide.cryosparc.com/live/about-cryosparc-live#add-or-free-up-compute-resources-during-a-session) * [Export of results and integration with cryoSPARC](https://guide.cryosparc.com/live/about-cryosparc-live#export-of-results-and-integration-with-cryosparc) * [Programmatic control of CryoSPARC Live](https://guide.cryosparc.com/live/about-cryosparc-live#programmatic-control-of-cryosparc-live) * [What users say](https://guide.cryosparc.com/live/about-cryosparc-live#what-users-say) * [History and development](https://guide.cryosparc.com/live/about-cryosparc-live#history-and-development) * [Embedded CryoSPARC Live](https://guide.cryosparc.com/live/about-cryosparc-live#embedded-cryosparc-live) * [Get Started](https://guide.cryosparc.com/live/about-cryosparc-live#get-started) --- # Automated Workflows | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/automated-workflows.md) . ### [](https://guide.cryosparc.com/processing-data/automated-workflows#automated-repeat-target-structure-determination) Automated repeat-target structure determination Single particle cryo-EM is a highly valuable technique for life sciences and drug discovery. Currently, obtaining state of the art results from cryo-EM data analysis requires a human in the loop to analyze intermediate results and make image processing decisions. This bottleneck limits the achievable throughput of structural characterization, especially in high-throughput settings such as structure-based drug design. In the link below, we describe the development of an end-to-end automation strategy using new tools built in CryoSPARC for “repeat-target” structure determination. We demonstrate our automation strategy on 21 G protein-coupled receptor (GPCR) datasets including both active and inactive states. In nearly all cases, the automated workflow meets or exceeds the published resolution and map quality. Our results demonstrate that it is now possible to completely automate the data processing workflow for repeat-target scenarios, and to obtain a high quality particle stack, consensus reconstruction and local refinement in the ligand binding region that are suitable for model building. Using the Workflows tool in CryoSPARC and the strategy described here, users can replicate, adapt and extend the automated workflow for their own targets. **End-to-end automation of repeat-target cryo-EM structure determination in CryoSPARC:** [**bioRxiv**](https://www.biorxiv.org/content/10.1101/2025.10.17.682689v1) ### [](https://guide.cryosparc.com/processing-data/automated-workflows#download-workflow-json-and-inputs) Download Workflow JSON and Inputs Below we provide a link to download a .zip archive containing CryoSPARC Workflow `JSON` files and required inputs which can be imported into CryoSPARC v4.7.1+ and used to process GPCR datasets like the ones we have tested. Two versions of the Workflow file are provided: one that processes raw movies directly from import, and another that processes micrographs that are generated after pre-processing in CryoSPARC Live. Please refer to the [**paper**](https://www.biorxiv.org/content/10.1101/2025.10.17.682689v1) for details. [**GPCR\_automated\_workflow\_materials\_v1.zip**](https://structura-assets.s3.us-east-1.amazonaws.com/automated-workflows/GPCR_automated_workflow_materials_v1.zip) ### [](https://guide.cryosparc.com/processing-data/automated-workflows#using-the-gpcr-workflow-file-on-your-own-gpcr-datasets) Using the GPCR workflow file on your own GPCR datasets The instructions below outline how to take our GPCR Workflow file and use it on your own GPCR datasets (i.e., targets of the same “class”). 1. Take the volumes you downloaded above. Upload these volumes to your compute setup. They do not need to be located within the project. 2. Import workflow JSON file into CryoSPARC using the workflow panel in the sidebar. 3. Select the workflow from the list, set all parameters, and choose the compute node to utilize. Parameters that might need to be adjusted (on a per dataset basis) are below: Import movies * Movies data path * Gain reference path * Raw pixel size (Å) * Accelerating voltage (kV) * Spherical aberration (mm) * Total exposure dose (e/Å^2) * Flip gain ref and defect file in Y Import 3D Volumes * Path to volumes/masks for GPCR reference, junk volumes, and the receptor mask Patch Motion Correction * Output F-crop factor Exposure Group Utilities * Regular expression string Extract from micrographs (x2) * Extraction box size Note: if using our reference, this should be 310.6 divided by the pixel size of the motion corrected micrographs (rounded to the nearest even pixel - e.g. 361.18 —> 362). 4. Click on the green “Apply” button in the bottom of the workflow GUI. 5. Download and inspect your final volumes, and proceed with any advanced processing if desired. ### [](https://guide.cryosparc.com/processing-data/automated-workflows#adapting-the-gpcr-workflow-for-other-target-classes) Adapting the GPCR workflow for other target classes Performing automated processing for a new target class is straightforward in CryoSPARC. Our strategy can be applied to any type of target, including but not limited to, membrane proteins, soluble proteins, nucleic acid samples, nucleoprotein targets, small proteins and large complexes. As described in the [**paper**](https://www.biorxiv.org/content/10.1101/2025.10.17.682689v1) , it is necessary to gather and set the target-class level inputs (a low-resolution reference map and masks, junk volumes, particle diameter, and particle separation distance) and optionally the workflow level inputs (number of rounds of decoy classification, 2D and 3D selection thresholds). These can then remain fixed for each dataset in the class. Below is a simple protocol for setting up a new automation Workflow for a new target class, by manually processing an exemplar dataset and then saving the resulting processing steps as a new Workflow. Once this setup is done, the Workflow can be saved and re-used in a single click for new datasets. 1. Decide on a definition of the target class (e.g., GLP-1 receptor with Nb6 nanobody) 2. Choose an exemplar cryo-EM dataset from the target class to use for instantiating the workflow. 3. Find a reference density map for the target class, for example from a previous manual processing of the exemplar dataset, from EMDB, or from structure prediction tools. This reference should be as similar as possible to the target class, but only needs to be approximately 15Å in resolution. Care should be taken to ensure that box sizes are appropriate. 4. As a starting point, import the GPCR automation Workflow JSON file (available [here](https://guide.cryosparc.com/processing-data/automated-workflows#download-workflow-json-and-inputs) ) into CryoSPARC v4.7.1+. Apply the workflow in a new project, but do not queue all the jobs. 1. Modify the Import Movies job to import the exemplar dataset and associated microscope parameters including exposure groups. 2. Modify the Import Volumes jobs to import the reference density for this target class. 5. Run each job in the Workflow to process the exemplar dataset. At each of the following points, inspect results and re-run jobs to set parameters, as appropriate, before proceeding to the next job: 1. Blob picking: adjust particle diameter and particle separation distance for blob picking, and visualize picks using Inspect Particle Picks to confirm. 2. Reference Based Auto Select 2D: adjust selection thresholds if necessary so that only good classes are selected. 3. Template picking: update parameters using the values from Blob picking. 4. Decoy classification: import existing junk volumes from a similar target class, or produce new junk volumes by running Ab-initio Reconstruction and terminating it early or manually editing and importing volumes. Add additional rounds of decoy classification if a single round has not sufficiently curated particles. 5. Reference Based Auto Select 3D: adjust selection thresholds if necessary. 6. Local Refinement: produce a local mask around the region of interest using Volume Tools. Masks should have adequate dilation (3-5Å) and a very soft edge (3x dilation distance). 6. In addition to the above, if prior processing experience with the target class is available, any other processing parameters (e.g. in exposure curation, 2D classification, refinements, etc) can be modified to optimize for the target class. 7. Once the Workflow is working for the exemplar dataset, select all the jobs and save as a new Workflow. 8. The new Workflow can now be re-used in a single click for fully automated processing of new datasets from the target class. [](https://guide.cryosparc.com/processing-data/automated-workflows#practical-tips-uploading-files-and-using-workflows) Practical tips: uploading files and using Workflows ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- #### [](https://guide.cryosparc.com/processing-data/automated-workflows#uploading-volumes-and-masks-for-use-in-workflow) Uploading volumes and masks for use in workflow All volumes and masks need to be within your filesystem including reference volumes, junk volumes, and masks, but they do not necessarily need to be located within the project where you intend to launch the automation workflow. The easiest way to upload files from your local filesystem to your compute infrastructure for use within CryoSPARC is to drag the file (or multiple files) to any CryoSPARC browser window: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FDoCTaxHO8rxCuZQGY0pM%252Fv4-7-1_automated_workflows_volume_upload.gif%3Falt%3Dmedia%26token%3D0e43af99-1839-4cc9-a85e-41033a9a12ce&width=768&dpr=3&quality=100&sign=6b4f6264&sv=2) * Files uploaded to CryoSPARC through the browser are added to a directory named `uploads` in the selected CryoSPARC project directory. The upload dialog lists all files already in the uploads directory in the first panel. * Once the uploads have completed, the Upload Files dialog can be closed (by clicking the green Done button) or more files can be uploaded by dragging and dropping into the dialog. * The [Upload Local Files](https://guide.cryosparc.com/application-guide-v4.0+/upload-local-files) guide page has more tips and guidance on the CryoSPARC browser upload system. If these volumes will be used across multiple CryoSPARC projects, you can make a directory within your compute filesystem where these volumes can be uploaded. Use `scp` from the command line or an SFTP software to upload them to the appropriate directory. #### [](https://guide.cryosparc.com/processing-data/automated-workflows#importing-the-workflow-json-file) Importing the workflow JSON file A workflow can be imported by clicking the “Import Workflow” button (red arrow) on the footer of the Workflows sidebar. This will open a file browser where you can find and upload a workflow `.json` file from your local filesystem. Once selected the file will be imported into your instance and will appear in the Workflows sidebar like any other workflow. The imported template has no special properties outside of a `imported` attribute to demarcate it as created outside of the instance. The imported workflow can be used, modified, and exported like any other. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FUZggKgkT977AYFDbs2OK%252Fv4-7-1_automated_workflows_import_workflow.png%3Falt%3Dmedia%26token%3D6ad853f6-fd24-4bca-b5d0-36ef3a82dd50&width=768&dpr=3&quality=100&sign=99e3610c&sv=2) The automated processing workflows for GPCRs will be placed into a group titled `Automated Workflows`. #### [](https://guide.cryosparc.com/processing-data/automated-workflows#applying-the-workflow) Applying the workflow Navigate to the “Workflows” sidebar panel: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FLrh7wg2icFN1xUXa7U4V%252Fv4-7-1-automated_workflows_side_panel.png%3Falt%3Dmedia%26token%3D35b8a9d9-3859-43b8-945d-062554093064&width=768&dpr=3&quality=100&sign=2ce080d3&sv=2) From here, click on the appropriate workflow to open the Workflow Apply Dialog: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FkmphHDUPfd3jKnY62Lor%252Fv4-7-1_automated_workflows_apply%2520dialog.jpeg%3Falt%3Dmedia%26token%3D8b885d22-c271-4563-8428-99a62684760f&width=768&dpr=3&quality=100&sign=c39b3b45&sv=2) * The settings section contains a `Queue on Apply` option. This allows you to set all jobs to queue as soon as the template is applied. The `Queue to Lane` option allows you to choose the lane the jobs will queue onto during application if you toggled the `Queue on Apply` option. * The proceeding job panels include all of the parameters that were exposed during the workflow’s creation. * Jobs that had no parameters set to a custom value or made visible during creation will not be shown, and will be coloured grey in the tree view. * Locked parameters are read-only in this view, and are denoted with a lock icon. * Resetting a parameter in this view will set it back to the custom value defined during creation. * Parameters that are flagged for adjustment prior to running are highlighted in orange and can easily be navigated to using the ‘Flagged’ parameters menu. Before applying the `agpcr_workflow_live_exposures.json` workflow, ensure that the parent job (Live Exposure Export) is selected before opening the workflow from the Workflows panel. #### [](https://guide.cryosparc.com/processing-data/automated-workflows#adjusting-parameters-in-a-workflow) Adjusting parameters in a workflow The Workflows functionality allows for parameters to be flagged as a pseudo-requirement to running the workflow of which there are multiple flagged parameters in the provided workflows. On the dialog footer you will see a “Flagged Parameters” tracker that shows the total number of flagged parameters and number of updated flagged parameters. Clicking this button will reveal a menu checklist of flagged parameters, organized by job, with a check mark-circle indicating whether it has been updated or not. Clicking a parameter in the menu will navigate to it. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FdwR2PnLBHOZP8xhpq1SR%252Fv4-7-1_automated_workflows_flagged_parameters.jpeg%3Falt%3Dmedia%26token%3D2db8968f-04c0-496c-a962-5bb7b0044d99&width=768&dpr=3&quality=100&sign=b3c85ee3&sv=2) Changing the value of a flagged parameter will update its colour to green and mark it with a green check mark. In these workflows the parameters that will need to be updated include: 1. Import movies 1. Movies data path 2. Gain reference path 3. Raw pixel size (Å) 4. Accelerating voltage (kV) 5. Spherical aberration (mm) 6. Total exposure dose (e/Å^2) 7. Flip gain ref and defect file in Y 2. Import 3D Volumes 1. Path to volumes/masks for GPRC reference, junk volumes, and the receptor mask 3. Patch Motion Correction 1. Output F-crop factor 4. Exposure Group Utilities 1. Regular expression string 5. Extract from micrographs (x2) 1. Extraction box size It is important to note that flagged parameters are not hard requirements. A workflow can be deployed without its flagged parameters having been updated. This is to maintain flexibility and not lock a user into updating a parameter they do not wish to. #### [](https://guide.cryosparc.com/processing-data/automated-workflows#executing-the-workflow) Executing the workflow The footer includes an “Apply” button to deploy the workflow into the current workspace, and a “Repeat” Button, which allows you to deploy the workflow and then open an identically configured Apply Dialog for quickly repeating the pipeline. For this example, we will maintain our default parameter values, select the “Queue on Apply” option, and apply the workflow in the tree view by clicking the “Apply” button. All of the jobs included in the workflow will now be automatically created, connected together, and queued onto the selected lane. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Ff6LyL0s08cHedKEzfKb6%252Fv4-7-1_automated_workflows_queue_workflow.jpeg%3Falt%3Dmedia%26token%3D80b71848-d07b-4510-b2f4-b13bdecbafa3&width=768&dpr=3&quality=100&sign=4827d4b5&sv=2) More details about all things workflow related can be found on the [Workflows](https://guide.cryosparc.com/application-guide-v4.0+/workflows) guide page, including how to modify or rebuild a workflow for further customization. [PreviousJob: Simulate Data (Legacy)](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-legacy) [NextCryoSPARC Tools](https://guide.cryosparc.com/processing-data/cryosparc-tools) Last updated 1 month ago * [Automated repeat-target structure determination](https://guide.cryosparc.com/processing-data/automated-workflows#automated-repeat-target-structure-determination) * [Download Workflow JSON and Inputs](https://guide.cryosparc.com/processing-data/automated-workflows#download-workflow-json-and-inputs) * [Using the GPCR workflow file on your own GPCR datasets](https://guide.cryosparc.com/processing-data/automated-workflows#using-the-gpcr-workflow-file-on-your-own-gpcr-datasets) * [Adapting the GPCR workflow for other target classes](https://guide.cryosparc.com/processing-data/automated-workflows#adapting-the-gpcr-workflow-for-other-target-classes) * [Practical tips: uploading files and using Workflows](https://guide.cryosparc.com/processing-data/automated-workflows#practical-tips-uploading-files-and-using-workflows) --- # Helical Reconstruction | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta.md) . CryoSPARC provides several specialized jobs to aid with processing of helical assemblies. While many single particle analysis (SPA) methods carry over quite well to helical assemblies, particle picking and refinement have been modified to better take advantage of the symmetry present in the dataset, ultimately leading to higher quality reconstructions. This page summarizes the new job types, and links to various other educational pages on the topic of helical reconstruction. The [Filament Tracer (BETA)](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-picking/job-filament-tracer-beta) provides a new template-based particle picking tool for identification of individual filaments, which can be useful for many datasets. [Helical Refinement (BETA)](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-helical-refinement-beta) allows for the direct reconstruction and high-resolution refinement of helical assemblies, and for the simultaneous enforcement and refinement of helical symmetry parameters. The [Symmetry search utility (BETA)](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-symmetry-search-utility-beta) is a tool that can take a reconstructed volume and search for locally optimal symmetry parameters in a given search range. Prior to the use of any helical reconstruction jobs, the page detailing [helical symmetry in CryoSPARC](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/helical-symmetry-in-cryosparc) should be read, as it details important information about: * how helical symmetry is parameterized in CryoSPARC, * possible workflows for helical reconstruction, and * limitations in the case of unknown helical symmetry As well, for an overview of an example workflow involving all of the new jobs introduced, it is highly recommended to view the [EMPIAR-10031 case study](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs) . This case study outlines an example end-to-end workflow for helical reconstruction, that may be useful for reference. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta#background) Background --------------------------------------------------------------------------------------------------------------------------------- [Helical symmetry in CryoSPARC](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/helical-symmetry-in-cryosparc) [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta#helical-reconstruction-and-related-jobs) Helical Reconstruction and related Jobs ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [Job: Filament Tracer](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-picking/job-filament-tracer-beta) [Job: Helical Refinement](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-helical-refinement-beta) [Job: Symmetry search utility](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-symmetry-search-utility-beta) [Job: Average Power Spectra](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-average-power-spectra) [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta#helical-reconstruction-tutorials) Helical Reconstruction Tutorials ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [Tutorial: Helical Processing using EMPIAR-10031 (MAVS)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs) [PreviousJob: Local Refinement (Legacy)](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-local-refinement-beta) [NextHelical symmetry in CryoSPARC](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/helical-symmetry-in-cryosparc) Last updated 2 years ago * [Background](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta#background) * [Helical Reconstruction and related Jobs](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta#helical-reconstruction-and-related-jobs) * [Helical Reconstruction Tutorials](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta#helical-reconstruction-tutorials) --- # Job: Average Power Spectra | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-average-power-spectra.md) . [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-average-power-spectra#description) Description ------------------------------------------------------------------------------------------------------------------------------------------------------------- While processing helical samples, it is often helpful to examine the power spectra of particles associated with a given 2D class. These power spectra may be used to determine the helical symmetry present in the sample, via Fourier-space methods such as Fourier-Bessel analysis \[1, 2\]. External software tools such as [HELIXPLORER](https://rico.ibs.fr/helixplorer/helixplorer/?) \[3\] implement such methods, and may be used downstream on the results of this job. This job takes the outputs of a [Select 2D Classes](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/interactive-job-select-2d-classes) job, and generates _averaged power spectra_ for each desired class. Specifically, the power spectrum of each particle within the supplied 2D classes is computed. Then, the spectra from all particles belonging to a given 2D class are averaged together. Note that this is distinct from computing the power spectrum of a 2D class average, as in this job, the power spectra are computed _before_ class-averaging takes place. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-average-power-spectra#inputs) Inputs --------------------------------------------------------------------------------------------------------------------------------------------------- This job takes class averages and particles as inputs from a previous Select 2D Classes job. To avoid excessive computation, only the classes (and corresponding particles) for which averaged power spectra are desired should be passed to this job. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-average-power-spectra#parameters) Parameters ----------------------------------------------------------------------------------------------------------------------------------------------------------- * `Particle computational batch size`: This controls the computational batch size of particles used during the reconstruction of power spectra. If memory errors or excessive swapping are encountered, this parameter can be lowered. Note that this parameter does not affect the results. * `Interpolation order`: This controls the interpolation order of the spline interpolation used when averaging the particles’ power spectra together. By default, cubic spline interpolation is used. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-average-power-spectra#output) Output --------------------------------------------------------------------------------------------------------------------------------------------------- * Averaged Power Spectra (including a `mrc` file containing the power spectra themselves, and a `cs` file containing their metadata, both located within the job directory) * Note: Since these power spectra don’t currently have direct downstream uses within cryoSPARC, they are not registered as cryoSPARC outputs and hence are not visible within the UI. You can find the files in the job directory for further use. * In the job directory, `JX_power_spectra.cs` will contain the metadata associated with each power spectrum: this includes the `uid` of the class average that generated it, as well as the sample spacing in A˚−1Å^{-1}A˚−1, the path to the `mrc` file, and the index of the power spectrum in the `mrc` file [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-average-power-spectra#example-images) Example Images ------------------------------------------------------------------------------------------------------------------------------------------------------------------- ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fxw38ohQTdovFTcYscsrY%252Fv4-0-0-job-average-power-spectra-example-2dclass-spectra-0.png%3Falt%3Dmedia%26token%3Dd99ed0b0-def2-439a-b517-7d6dd433d636&width=768&dpr=3&quality=100&sign=537dcb75&sv=2) Above is an example of an image from the stream log. This shows four 2D classes of the Tobacco Mosaic Virus ([EMPIAR-10022](https://www.ebi.ac.uk/empiar/EMPIAR-10022/) ), as well as their associated average power spectra. Note the power spectra images can be found in the job directory, and the 2D class images can be found in the job directory of the preceding 2D Classification job. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-average-power-spectra#next-steps) Next Steps ----------------------------------------------------------------------------------------------------------------------------------------------------------- One may use these averaged power spectra to initiate Fourier-space investigation of helical symmetry. Note that the .mrc file output of this job may need to be converted to .jpg or .png format for use in other programs. One software tool that aids in this indexing procedure is [HELIXPLORER](http://rico.ibs.fr/helixplorer/) . HELIXPLORER performs a grid search over helical symmetry parameters, and compares theoretically expected power spectra from an ideal helix to the provided power spectrum. For each symmetry parameter, a score is computed based on how closely the two power spectra agree, which is useful in obtaining a set of candidate symmetry parameters. More information on the theory and operation behind HELIXPLORER is available via their [documentation](http://rico.ibs.fr/helixplorer/help.htm) . There are also more recent external indexing softwares available, such as: * [PyHI](https://pubmed.ncbi.nlm.nih.gov/34529294/) ([available on GitHub](https://github.com/xuewuzhang-UTSW/PyHI) , by Xuewu Zhang of the [Bai Lab at UT Southwestern](https://baistructurelab.org/) ) * [HI3D](https://jiang.bio.purdue.edu/hi3d/) , from the [Jiang lab at Purdue University](https://jiang.bio.purdue.edu/) [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-average-power-spectra#references) References ----------------------------------------------------------------------------------------------------------------------------------------------------------- \[1\] E. Egelman, "The iterative helical real space reconstruction method: Surmounting the problems posed by real polymers", Journal of Structural Biology, vol. 157, no. 1, pp. 83-94, 2007. Available: 10.1016/j.jsb.2006.05.015. \[2\] L. Gambelli, M. Isupov and B. Daum, "Escaping the symmetry trap in helical reconstruction", Faraday Discussions, 2022. Available: 10.1039/d2fd00051b. \[3\] L. Estrozi, A. Desfosses and G. Schoehn, "Helixplorer-1: Online Indexation Of Fibers And Helical Structures", [Rico.ibs.fr](http://rico.ibs.fr/) , 2018. Available:[http://rico.ibs.fr/helixplorer/help.htm](http://rico.ibs.fr/helixplorer/help.htm) . [PreviousJob: Symmetry search utility](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-symmetry-search-utility-beta) [NextUtilities](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities) Last updated 1 month ago * [Description](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-average-power-spectra#description) * [Inputs](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-average-power-spectra#inputs) * [Parameters](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-average-power-spectra#parameters) * [Output](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-average-power-spectra#output) * [Example Images](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-average-power-spectra#example-images) * [Next Steps](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-average-power-spectra#next-steps) * [References](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-average-power-spectra#references) --- # Job: Generate Micrograph Thumbnails | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-generate-micrograph-thumbnails.md) . [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-generate-micrograph-thumbnails#description) Description ---------------------------------------------------------------------------------------------------------------------------------------------------- This job generates preview thumbnails of input micrographs for downstream use inside or outside of CryoSPARC. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-generate-micrograph-thumbnails#input) Input ---------------------------------------------------------------------------------------------------------------------------------------- * Micrographs (exposures) [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-generate-micrograph-thumbnails#output) Output ------------------------------------------------------------------------------------------------------------------------------------------ * Micrographs with thumbnails (exposures) [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-generate-micrograph-thumbnails#common-next-steps) Common Next Steps ---------------------------------------------------------------------------------------------------------------------------------------------------------------- * Downstream CryoSPARC jobs will make use of the thumbnail previews for enhanced diagnostics * Preview images (available in two types, both 1x (regular DPI) and 2x (high DPI) formats) are available in the job directory: `/projects/P1/J1/thumbnails/*.png` [PreviousJob: Exposure Tools](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-exposure-tools) [NextJob: Cache Particles on SSD](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-cache-particles-on-ssd) Last updated 1 month ago * [Description](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-generate-micrograph-thumbnails#description) * [Input](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-generate-micrograph-thumbnails#input) * [Output](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-generate-micrograph-thumbnails#output) * [Common Next Steps](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-generate-micrograph-thumbnails#common-next-steps) --- # Job: Exposure Tools | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-exposure-tools.md) . [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-exposure-tools#description) **Description** ---------------------------------------------------------------------------------------------------------------------------------------- * Manually modify an exposure dataset's `mscope_params/neg_stain` and `mscope_params/phase_plate` values. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-exposure-tools#input) **Input** ---------------------------------------------------------------------------------------------------------------------------- * 1 exposure stack [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-exposure-tools#output) **Output** ------------------------------------------------------------------------------------------------------------------------------ * exposure stack with modified `mscope_params` [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-exposure-tools#common-parameters) **Common Parameters** ---------------------------------------------------------------------------------------------------------------------------------------------------- * `Negative Stain Data:` If Negative Stain Data is on, this indicates that there are light particles on dark background (-1). If it's off, this indicates the movies have dark particles on light background (cryo-em data, +1). * `Phase Plate Data:` Indicates if data was collected using a phase plate. [PreviousJob: Exposure Sets Tool](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-exposure-sets-tool) [NextJob: Generate Micrograph Thumbnails](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-generate-micrograph-thumbnails) Last updated 1 month ago * [Description](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-exposure-tools#description) * [Input](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-exposure-tools#input) * [Output](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-exposure-tools#output) * [Common Parameters](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-exposure-tools#common-parameters) --- # Job: Cache Particles on SSD | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-cache-particles-on-ssd.md) . [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-cache-particles-on-ssd#description) **Description** ------------------------------------------------------------------------------------------------------------------------------------------------ Cache particles on a workstation before running a job that requires the particles. This is useful when working with many particles, as it may take a long time to copy these over to the SSD. Pre-caching allows you to free up the GPU that would've normally been acquired during the caching process, allowing other jobs to use the GPU while caching completes. Read more about [SSD caching in CryoSPARC](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/tutorial-ssd-particle-caching-in-cryosparc) . ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FWvOltlz4qtZi0jLyEQZZ%252FScreen%2520Shot%25202021-11-29%2520at%25204.43.30%2520PM%2520%282%29.png%3Falt%3Dmedia%26token%3Db7d08054-2e2c-4d78-bfed-bbb6c57d3c10&width=768&dpr=3&quality=100&sign=889c9fbd&sv=2) A common "Cache Particles" workflow. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-cache-particles-on-ssd#input) Input -------------------------------------------------------------------------------------------------------------------------------- * Particles [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-cache-particles-on-ssd#output) Output ---------------------------------------------------------------------------------------------------------------------------------- * Particles [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-cache-particles-on-ssd#common-next-steps) Common Next Steps -------------------------------------------------------------------------------------------------------------------------------------------------------- * This job usually sits in between an Extract From Micrographs job and a particle processing job like 2D Classification, Non-uniform Refinement, or 3D Classification. [PreviousJob: Generate Micrograph Thumbnails](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-generate-micrograph-thumbnails) [NextJob: Check for Corrupt Particles](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-check-for-corrupt-particles) Last updated 1 month ago * [Description](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-cache-particles-on-ssd#description) * [Input](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-cache-particles-on-ssd#input) * [Output](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-cache-particles-on-ssd#output) * [Common Next Steps](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-cache-particles-on-ssd#common-next-steps) --- # Job: Check for Corrupt Particles | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-check-for-corrupt-particles.md) . [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-check-for-corrupt-particles#description) Description ------------------------------------------------------------------------------------------------------------------------------------------------- Checks if the particle files on disk are corrupt. The job will read the header of each file and compute the expected file size based on the `ny`, `nx` and `nz` values. If the expected file size of a file doesn't match the actual file size on disk, the job will mark the particle as corrupt. In CryoSPARC v4.6.1+, the job will also by default check for NaN values in the particle stack files, and mark the particles as corrupt if any are found. The job issues a warning for each file that is found corrupt. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F3Q2PVTkstfN67vETySae%252FScreen%2520Shot%25202021-11-29%2520at%25204.01.39%2520PM.png%3Falt%3Dmedia%26token%3D11a36787-e4a1-4c3c-8c8e-296e4346353e&width=768&dpr=3&quality=100&sign=e015a0ee&sv=2) A corrupted .mrc file The job will produce an output that includes only the non-corrupt particle files. This output can be used for downstream processing. In CryoSPARC v4.6.1+, the job will run through all input particles, produce warnings, and then throw and error fail at the end if any files were found to be corrupt. The job can then either be marked as complete in order to use the filtered output, or it can be run with the "Fail if corruption is detected" flag off. In CryoSPARC v4.6.1+, the job is also able to check a checksum in the MRC header to ensure that all the data in the file is intact. The checksum is only produced for MRC files written in v4.6.1+. The "verify file checksums" parameter is off by default. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-check-for-corrupt-particles#input) Input ------------------------------------------------------------------------------------------------------------------------------------- * Particles [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-check-for-corrupt-particles#output) Output --------------------------------------------------------------------------------------------------------------------------------------- * Particles [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-check-for-corrupt-particles#common-next-steps) Common Next Steps ------------------------------------------------------------------------------------------------------------------------------------------------------------- You can use the output of the job to carry on processing, since the output only contains files that were found to be intact and not corrupt. After any corrupt files have been identified, you can then find the files on the filesystem and try to rectify the issue that lead to the corruption. [PreviousJob: Cache Particles on SSD](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-cache-particles-on-ssd) [NextJob: Particle Sets Tool](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-particle-sets-tool) Last updated 1 month ago * [Description](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-check-for-corrupt-particles#description) * [Input](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-check-for-corrupt-particles#input) * [Output](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-check-for-corrupt-particles#output) * [Common Next Steps](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-check-for-corrupt-particles#common-next-steps) --- # Tutorial: EER File Support | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-eer-file-support.md) . [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-eer-file-support#electron-event-representation-eer) Electron-event representation (EER) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- CryoSPARC supports reading input movies in electron-event representation (EER) format. EER is a special format developed by Thermo Fisher Scientific currently used for output from Falcon4 cameras. EER records individual electron detection events rather than storing movies as a series of image frames. For more information, please see [Guo et al., Electron-event representation data enable efficient cryoEM file storage with full preservation of spatial and temporal resolution (IUCrJ)](https://doi.org/10.1107/S205225252000929X) . [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-eer-file-support#importing-eer-data) Importing EER data ----------------------------------------------------------------------------------------------------------------------------------------------- When importing EER data for processing in CryoSPARC, there are two additional parameters that must be specified in Import Movies jobs and in the Session Configuration panel in CryoSPARC Live. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNf0KlPlocGvRbrzv-E%252F-MNf0nmzwh3w_pJxFQAk%252FEER_1_Screenshot%2520from%25202020-11-13%252015-12-08.png%3Falt%3Dmedia%26token%3D50f1b524-fd36-449a-a050-8eaac045fdb7&width=768&dpr=3&quality=100&sign=4e707995&sv=2) EER-specific import parameters (Import Movies) CryoSPARC converts the electron events recorded in an EER file into a stack of images, similar to what would be found in an MRC or TIFF file. The "EER Number of Fractions" parameter determines how many "fractions" are in the resulting movie. Too many fractions can result in very large memory demands during motion correction and longer processing times, while too few fractions can result in inadequate motion compensation. The default value of 40 is suitable in many cases. The "EER Upsampling Factor" allows the rasterized images to have a higher resolution than the actual resolution of the camera (similar to super-resolution images from Gatan cameras). For example, an upsampling factor of 2 will create an image with twice as many pixels in each direction as the physical camera. Values of 1 (no upsampling/native resolution), 2 and 4 are supported. Note that using a value of 4 may result in extremely large images and high GPU memory requirements. In some cases, it is possible to recover information beyond the physical Nyquist resolution of the camera using upsampling of e.g., 2. [PreviousTutorial: Phase Plate Data](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/phase-plate-data) [NextTutorial: EPU AFIS Beam Shift Import](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import) Last updated 3 years ago * [Electron-event representation (EER)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-eer-file-support#electron-event-representation-eer) * [Importing EER data](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-eer-file-support#importing-eer-data) --- # Job: Select Volume | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume.md) . [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#at-a-glance) At a Glance ----------------------------------------------------------------------------------------------------------------------------------- Automatically select the highest resolution volume out of a number of input volumes, and output the best volume with its corresponding particle stack. This job can be useful for automated selection between multiple refinements with different parameters, for example CTF refinement parameters, masking parameters, etc. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#description) Description ----------------------------------------------------------------------------------------------------------------------------------- This job type inputs multiple volume-particle stack pairs. The job also takes in a single mask. The resolution of each volume is computed using the mask, and the volume with the highest resolution (measured using area-under-FSC-curve) is selected. The selected volume and its corresponding particle stack is output. ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#when-to-use-select-volume) When to use Select Volume You can use Select Volume to automate scenarios where you would run multiple refinements in parallel, and select the best result for downstream processing. For example: * Run multiple refinement jobs with various CTF refinement parameters enabled. Connect the outputs of all these refinements to a Select Volume job, and then connect the output of Select Volume to a downstream step such as Reference-based Motion Correction. By doing so, the refinement that produces the best resolution will be selected, and the particles that are passed on to the downstream job will have the corresponding higher-order CTF aberrations corrected. * Run multiple refinements with different input particle stacks, and use Select Volume to automatically carry forward the best input stack for downstream processing. For example, run two refinements, one with a full set of particles, and one with a subset chosed by Subset Particles by Statistic. Select Volume will select the refinement with the best resolution, and pass those particles on for downstream steps. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#inputs) Inputs ------------------------------------------------------------------------------------------------------------------------- Before connecting inputs, set the `Number of inputs to compare` parameter based on how many volumes you wish to compare. ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#mask) Mask A single mask that will be used to compute FSC curves for all input volumes. ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#input-volume-x) Input volume X Multiple of these inputs will be created when you set the `Number of inputs to compare` parameter. Each one takes in a volume, requiring the full map and two half-maps. ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#particle-stack-x) Particle stack X The input particles corresponding to input volume X. The selected volume and corresponding particle stack are output unchanged by the job. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#commonly-adjusted-parameters) Commonly Adjusted Parameters --------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#number-of-inputs-to-compare) Number of inputs to compare Sets the number of volume+particle stack inputs to compare. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#outputs) Outputs --------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#volume) Volume This is an unchanged copy of the selected input volume. ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#particles) Particles This is an unchanged copy of the input particle stack corresponding to the selected volume. [PreviousJob: Align 3D maps](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-align-3d-maps) [NextJob: Split Volumes Group](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group) Last updated 1 month ago * [At a Glance](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#at-a-glance) * [Description](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#description) * [When to use Select Volume](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#when-to-use-select-volume) * [Inputs](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#inputs) * [Mask](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#mask) * [Input volume X](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#input-volume-x) * [Particle stack X](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#particle-stack-x) * [Commonly Adjusted Parameters](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#commonly-adjusted-parameters) * [Number of inputs to compare](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#number-of-inputs-to-compare) * [Outputs](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#outputs) * [Volume](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#volume) * [Particles](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume#particles) --- # Tutorial: Phase Plate Data | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/phase-plate-data.md) . [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/phase-plate-data#how-to-specify-phase-plate-data) How to specify phase plate data ---------------------------------------------------------------------------------------------------------------------------------------------------------------- You can specify the data was collected using a phase plate, when you import the corresponding movies or micrographs. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/phase-plate-data#import-phase-plate-movies-or-micrographs) Import phase plate movies or micrographs When importing phase plate movies or micrographs, use the Phase Plate Data toggle in the `Import Movies` or `Import Micrographs` jobs to indicate that you are importing phase plate data. Subsequent jobs that use the imported phase plate data, will adjust their relevant parameters accordingly (for more details, see below: Jobs affected by phase plate data). ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MM2JbJEET5s6ZM-_dik%252F-MM2JqxRNqF82tW_wb_X%252F4neg-stain-phase-plate-4.png%3Falt%3Dmedia%26token%3Da2604985-8e9a-4e53-8b8e-f90f14719a74&width=768&dpr=3&quality=100&sign=bfd1fb74&sv=2) [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/phase-plate-data#jobs-and-parameters-affected-by-phase-plate-data) Jobs and parameters affected by Phase Plate Data -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1. `Import Movies`: "Phase Plate Data" toggle will set `mscope_params/phase_plate` to `1` (True) 2. `Import Micrographs`: "Phase Plate Data" toggle will set `mscope_params/phase_plate` to `1` (True) 3. `Patch CTF Estimation`: Enables phase-shift search & refinement 4. `CTFFIND4 Wrapper`: Enables phase-shift search & refinement [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/phase-plate-data#data-type-reference-chart) Data type reference chart ---------------------------------------------------------------------------------------------------------------------------------------------------- Data Type Data Sign Description Cryo-EM +1 dark-on-light Negative Stain \-1 light-on-dark **Note:** Cryo-EM data is typically recorded as +1 (dark-on-light) but is often inverted during processing. [PreviousTutorial: Negative Stain Data](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/negative-stain-data) [NextTutorial: EER File Support](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-eer-file-support) Last updated 3 years ago * [How to specify phase plate data](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/phase-plate-data#how-to-specify-phase-plate-data) * [Import phase plate movies or micrographs](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/phase-plate-data#import-phase-plate-movies-or-micrographs) * [Jobs and parameters affected by Phase Plate Data](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/phase-plate-data#jobs-and-parameters-affected-by-phase-plate-data) * [Data type reference chart](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/phase-plate-data#data-type-reference-chart) --- # Tutorial: Blob Picker Tuner | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-blob-picker-tuner.md) . The Blob Picker Tuner works by comparing Blob Picker picks across different sizes, shapes, and thresholds to manual picks. It can take some of the guesswork out of using the Blob Picker job by quantitatively comparing sets of parameters. In this tutorial, we will see how to get good manual picks and set up the Blob Picker Tuner job. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-blob-picker-tuner#getting-manual-picks) Getting Manual Picks Getting good manual picks is important for the blob tuner to perform well. The key point is to inform the grid search about the size of the particles by making picks close together: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F8ltCe6IMTfcPm1QfFxLL%252F33_blob_tuner_1.png%3Falt%3Dmedia%26token%3D1cc6473b-304f-47e2-901a-0952c37bf171&width=768&dpr=3&quality=100&sign=9a7e2ed&sv=2) It is also beneficial to choose the highest quality micrographs in the dataset for manual picking. Picking 15-40 particles is the recommended amount, but using more may improve the final result. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-blob-picker-tuner#running-the-job) Running the Job Connect the CTF-corrected micrographs to the micrographs input group. Connect the manually picked particles to the particles input group. The default parameters should be sufficient for most datasets, but the particle agreement distance must be set. This parameter controls the distance at which two picks are considered the same, and is important for this job to run correctly. The particle agreement distance should be roughly the size of the particle in Angstroms. The parameters in the Template Parameters section are the same as in the Blob Picker job, and are not varied in the grid search. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FDJQin8bv8qaSQHtDvZev%252F33_blob_tuner_2.png%3Falt%3Dmedia%26token%3D793f815d-042b-4990-9d61-13eeb65fb262&width=768&dpr=3&quality=100&sign=7b77adcc&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FvRn6Nf1ZbmOumNyvx7Cm%252F33_blob_tuner_3.png%3Falt%3Dmedia%26token%3Dc05eec57-b6f1-4376-a6cc-e09ca19f2e7f&width=768&dpr=3&quality=100&sign=2c232e68&sv=2) When the blob picker finds more than 5 sets of parameters with the same best score, it will suggest labelling more particles. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FbpSZjZxTV79C4D996xSq%252F33_blob_tuner_4.png%3Falt%3Dmedia%26token%3D97fd4434-22e0-4f30-b3db-f2d5055e3684&width=768&dpr=3&quality=100&sign=ec9d87f0&sv=2) The best set of parameters is printed out to the console for use later. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Filyeb4lLmYIExdEhyuUt%252F33_blob_tuner_5.png%3Falt%3Dmedia%26token%3Dade050fa-f023-4ae8-98d7-e8d760fe5a45&width=768&dpr=3&quality=100&sign=957671f0&sv=2) The outputs can be connected to an Inspect Picks job to verify the quality of the picks. [PreviousTutorial: Particle Picking Calibration](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-particle-picking-calibration) [NextTutorial: Helical Processing using EMPIAR-10031 (MAVS)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs) Last updated 1 month ago * [Getting Manual Picks](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-blob-picker-tuner#getting-manual-picks) * [Running the Job](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-blob-picker-tuner#running-the-job) --- # Job: Reassign Particles to Micrographs | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-reassign-particles-to-micrographs.md) . [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-reassign-particles-to-micrographs#description) **Description** ----------------------------------------------------------------------------------------------------------------------------------------------------------- The Reassign Particles to Micrographs job can be used to link particles back to the micrographs they came from by matching their file names. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-reassign-particles-to-micrographs#input) **Input** ----------------------------------------------------------------------------------------------------------------------------------------------- * micrograph dataset * particle dataset * `location` result field [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-reassign-particles-to-micrographs#output) **Output** ------------------------------------------------------------------------------------------------------------------------------------------------- * micrograph dataset (passthrough) * particle dataset (linked to micrograph dataset) [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-reassign-particles-to-micrographs#common-next-steps) **Common Next Steps** ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- * Particle extraction via Extract From Micrographs [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-reassign-particles-to-micrographs#example-use-case) Example Use Case ----------------------------------------------------------------------------------------------------------------------------------------------------------------- Consider a scenario where you receive pre-aligned micrographs from a microscope facility and start processing them in CryoSPARC. After you obtain your final subset of particles, you want to improve the resolution of your structure by aligning the raw movies in CryoSPARC using Patch Motion Correction. The problem is when you imported your movies, CryoSPARC created a new unique dataset for them, so connecting the new micrographs and the particles from the pre-aligned micrographs to the "Extract From Micrographs" job doesn't work, as there is no association between these datasets. In this case, you can use the "Reassign Particles to Micrographs" job. CryoSPARC uses the filename of a movie to create the filename of its micrograph, which is then used to create the filename of its particle stack. You can re-associate a particle dataset back to a micrograph dataset using the common part of the path, which this job helps you do. The usage of this job is similar to the Import Particles and Import Micrographs job. When you run the Reassign Particles to Micrographs job, you'll see an example of the source filename and an example of the query filename. The objective is to cut enough characters from both the query filename and/or the source filename such that the query matches the source. Consider the following case: In order to match the query with the source, we're going to have to cut characters from both strings. Specifically, `002748358406320475195_` from the prefix of the query, `.frames_patch_aligned.mrc` from the suffix of the query, and `.frames_patch_aligned_doseweighted.mrc` from the suffix of the source. You can use a site like[https://charactercounttool.com/](https://charactercounttool.com/) to count the number of characters. The parameters for the job should therefore be: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FS0ux33mE6VmcAuORGNOH%252F33reassign-particles.png%3Falt%3Dmedia%26token%3D4e658c0c-98a4-45a0-94cc-19dcf2284212&width=768&dpr=3&quality=100&sign=bbf48853&sv=2) You can now take the outputs of this job, and connect them to an "Extract From Micrographs" job, or any other job that requires particles and micrographs. [PreviousJob: Particle Sets Tool](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-particle-sets-tool) [NextJob: Remove Duplicate Particles](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-remove-duplicate-particles) Last updated 1 month ago * [Description](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-reassign-particles-to-micrographs#description) * [Input](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-reassign-particles-to-micrographs#input) * [Output](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-reassign-particles-to-micrographs#output) * [Common Next Steps](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-reassign-particles-to-micrographs#common-next-steps) * [Example Use Case](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-reassign-particles-to-micrographs#example-use-case) Copy Loaded particle stack with 8474 items Loaded 20 micrographs Example source micrograph filename: 14sep05c_00024sq_00003hl_00002es.frames_patch_aligned_doseweighted.mrc Example query micrograph filename: 002748358406320475195_14sep05c_c_00003gr_00014sq_00005hl_00002es.frames_patch_aligned.mrc Copy Length of input micrograph path suffix to cut : 38 Length of location/micrograph_path prefix to cut for query : 22 Length of location/micrograph_path path suffix to cut for query : 25 --- # Managing a CryoSPARC Live Session from the CLI (≤v4.7) | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli.md) . The instructions on this page work for CryoSPARC versions up to v4.7; for v5.0+, see [this page](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli-v5.0) . [](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli#overview) Overview ---------------------------------------------------------------------------------------------------------- This guide details how to automate the creation, configuration and overall management of a CryoSPARC Live session in Python via the CryoSPARC API. The instructions are valid for CryoSPARC versions from 3.3 through 4.6. [](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli#setup) Setup ---------------------------------------------------------------------------------------------------- To run the following commands, you can use the built-in python interpreter provided by CryoSPARC by running `cryosparcm icli`. You can also issue one-off calls directly to CryoSPARC by running `cryosparcm cli ""` or `cryosparcm rtpcli “”` for command\_core and command\_rtp functions, respectively. This guide will use the **interactive python shell provided by CryoSPARC** to create a CryoSPARC Live Session. When you run `cryosparcm icli`, you will have access to the command\_core API via `cli`, the command\_rtp API via `rtp`, and the database via `db`. ### [](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli#using-a-script) Using a Script You can also put these functions inside a bash script or a python script. To get access to the CryoSPARC API (`cli` and `rtp`) inside a python script, add the following lines to your script, e.g., called `automate_cryosparc_live.py`: Copy import os from cryosparc_compute import client host = os.environ['CRYOSPARC_MASTER_HOSTNAME'] command_core_port = int(os.environ['CRYOSPARC_COMMAND_CORE_PORT']) command_rtp_port = int(os.environ['CRYOSPARC_COMMAND_RTP_PORT']) cli = client.CommandClient(host=host, port=command_core_port) rtp = client.CommandClient(host=host, port=command_rtp_port) # ... Use CryoSPARC’s provided python interpreter to execute your script (required): [](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli#automate-a-cryosparc-live-session) Automate a CryoSPARC Live Session ------------------------------------------------------------------------------------------------------------------------------------------------------------ ### [](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli#create-a-project-optional) Create A Project \[Optional\] You will need a project to create your Live session. If you already have a project, you can skip this step. 1. **Get your CryoSPARC user ID using your CryoSPARC email address** 2. **Create the project** Note that CryoSPARC will create a folder for the project inside the `project_container_dir` path you specify named after the UID of the project created (e.g., `P12`). The definition of the `project_description` variable and the corresponding `desc` parameter are optional. ### [](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli#create-a-cryosparc-live-session) Create a CryoSPARC Live Session 1. **Get your CryoSPARC user ID using your CryoSPARC email address** 2. **Create the CryoSPARC Live Session** Note that the `title` and `desc` arguments are optional. `project_uid` can be defined explicitly for pre-existing projects, or by using the value returned by `cli.create_empty_project(<..>)`(see above). ### [](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli#configure-your-cryosparc-live-session) Configure your CryoSPARC Live Session 1. Set the session’s compute configuration 2. Set the session’s exposure group parameters Note that the following lines change the exposure group parameters for the first exposure group (`exp_group_id=1`). 3. Set the session’s processing parameters Note that the following parameters are **required** to be set before starting a session. To see the full list of parameters available to be set, see below. ### [](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli#start-the-cryosparc-live-session) Start the CryoSPARC Live Session 1. Start the session If you’ve configured your session properly, CryoSPARC Live will start to find movies and process data. At this point, you can navigate to the user interface and watch as exposures are processed. The next step would be to start Streaming 2D Classification and Streaming Homogeneous Refinement. You can also use the user interface to manually pick particles and use the template picker to identify particle locations. 2. Start Streaming 2D Classification 3. Start Ab-Initio Reconstruction 1. Create a new initial volume 2. Use an existing job’s volume output as the initial volume 4. Start Streaming Homogeneous Refinement Note that an initial volume must be set for the session before starting Streaming Homogeneous Refinement. ### [](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli#stop-the-cryosparc-live-session) Stop the CryoSPARC Live Session 1. Pause the entire CryoSPARC Live Session 2. Clear the CryoSPARC Live Session ### [](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli#export-exposures-and-particles-from-the-cryosparc-live-session) Export Exposures and Particles from the CryoSPARC Live Session 1. Export all exposures from the session 2. Export all particles from the session [](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli#appendix) Appendix ---------------------------------------------------------------------------------------------------------- As of v3.3, these are the parameters you can set for a live session: [PreviousPerformance Metrics](https://guide.cryosparc.com/live/performance-metrics) [NextManaging a CryoSPARC Live Session from the CLI (v5.0+)](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli-v5.0) Last updated 1 month ago * [Overview](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli#overview) * [Setup](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli#setup) * [Using a Script](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli#using-a-script) * [Automate a CryoSPARC Live Session](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli#automate-a-cryosparc-live-session) * [Create A Project \[Optional\]](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli#create-a-project-optional) * [Create a CryoSPARC Live Session](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli#create-a-cryosparc-live-session) * [Configure your CryoSPARC Live Session](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli#configure-your-cryosparc-live-session) * [Start the CryoSPARC Live Session](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli#start-the-cryosparc-live-session) * [Stop the CryoSPARC Live Session](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli#stop-the-cryosparc-live-session) * [Export Exposures and Particles from the CryoSPARC Live Session](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli#export-exposures-and-particles-from-the-cryosparc-live-session) * [Appendix](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli#appendix) Copy # Enter these in the command line eval $(cryosparcm env) export PYTHONPATH="${CRYOSPARC_ROOT_DIR}" python automate_cryosparc_live.py Copy email_address = 'developer@structura.bio' user_id = cli.get_id_by_email(email_address) Copy project_container_dir = '/data/cryoSPARC_projects/' project_title = 'EMPIAR-10025' project_description = 'T20s Processing Project' project_uid = cli.create_empty_project(owner_user_id=user_id, project_container_dir=project_container_dir, title=project_title, desc=project_description) Copy email_address = 'developer@structura.bio' user_id = cli.get_id_by_email(email_address) Copy project_uid = 'P12' # set this variable if you didn't already create a project in the above step session_title = 'EMPIAR-10025 Live Processing Session' session_description = 'Simulated live session' session_uid = rtp.create_new_live_workspace(project_uid=project_uid, created_by_user_id=user_id, title=session_title, desc=session_description) Copy phase_one_lane = 'gpu_lane_name' phase_one_gpus = 2 # optional phase_two_lane = 'gpu_lane_name' phase_two_ssd = True # optional auxiliary_lane = 'gpu_lane_name' auxiliary_ssd = True # optional rtp.update_compute_configuration(project_uid=project_uid, session_uid=session_uid, key='phase_one_lane', value=phase_one_lane) rtp.update_compute_configuration(project_uid=project_uid, session_uid=session_uid, key='phase_one_gpus', value=phase_one_gpus) rtp.update_compute_configuration(project_uid=project_uid, session_uid=session_uid, key='phase_two_lane', value=phase_two_lane) rtp.update_compute_configuration(project_uid=project_uid, session_uid=session_uid, key='phase_two_ssd', value=phase_two_ssd) rtp.update_compute_configuration(project_uid=project_uid, session_uid=session_uid, key='auxiliary_lane', value=auxiliary_lane) rtp.update_compute_configuration(project_uid=project_uid, session_uid=session_uid, key='auxiliary_ssd', value=auxiliary_ssd) Copy file_engine_watch_path_abs = '/data/EMPIAR/10025/data/14sep05c_raw_196/' file_engine_filter = '*.frames.mrc' gainref_path = '/data/EMPIAR/10025/data/14sep05c_raw_196/norm-amibox05-0.mrc' rtp.exposure_group_update_value(project_uid=project_uid, session_uid=session_uid, exp_group_id=1, name='file_engine_watch_path_abs', value=file_engine_watch_path_abs) rtp.exposure_group_update_value(project_uid=project_uid, session_uid=session_uid, exp_group_id=1, name='file_engine_filter', value=file_engine_filter) rtp.exposure_group_update_value(project_uid=project_uid, session_uid=session_uid, exp_group_id=1, name='gainref_path', value=gainref_path) rtp.exposure_group_finalize_and_enable(project_uid=project_uid, session_uid=session_uid, exp_group_id=1) Copy rtp.set_param(project_uid=project_uid, session_uid=session_uid, param_sec='mscope_params', param_name='psize_A', value=0.6575) rtp.set_param(project_uid=project_uid, session_uid=session_uid, param_sec='mscope_params', param_name='accel_kv', value=300) rtp.set_param(project_uid=project_uid, session_uid=session_uid, param_sec='mscope_params', param_name='cs_mm', value=2.7) rtp.set_param(project_uid=project_uid, session_uid=session_uid, param_sec='mscope_params', param_name='total_dose_e_per_A2', value=53) rtp.set_param(project_uid=project_uid, session_uid=session_uid, param_sec='blob_pick', param_name='diameter', value=100) rtp.set_param(project_uid=project_uid, session_uid=session_uid, param_sec='blob_pick', param_name='diameter_max', value=200) rtp.set_param(project_uid=project_uid, session_uid=session_uid, param_sec='extraction', param_name='box_size_pix', value=440) Copy rtp.start_session(project_uid=project_uid, session_uid=session_uid, user_id=user_id) Copy rtp.phase2_class2D_set_param(project_uid=project_uid, session_uid=session_uid, par='class2D_K', value=200) rtp.phase2_class2D_start(project_uid=project_uid, session_uid=session_uid) Copy rtp.phase2_abinit_set_param(project_uid=project_uid, session_uid=session_uid, par='abinit_K', value=3) rtp.phase2_abinit_start(project_uid=project_uid, session_uid=session_uid) Copy rtp.phase2_abinit_set_job(project_uid=project_uid, session_uid=session_uid, abinit_juid='J43') Copy rtp.phase2_refine_set_param(project_uid=project_uid, session_uid=session_uid, par='refine_symmetry', value='D7') rtp.phase2_refine_start(project_uid=project_uid, session_uid=session_uid) Copy rtp.pause_session(project_uid=project_uid, session_uid=session_uid) Copy rtp.clear_session(project_uid=project_uid, session_uid=session_uid) Copy rtp.dump_exposures(project_uid=project_uid, session_uid=session_uid) Copy rtp.dump_particles(project_uid=project_uid, session_uid=session_uid) Copy [{'param_sec': 'mscope_params',\ 'param_name': 'gainref_flip_x',\ 'param_type': 'boolean',\ 'base_value': False,\ 'title': 'Flip gain ref in X?',\ 'desc': 'Flip gain ref left-to-right (in X axis)'},\ {'param_sec': 'mscope_params',\ 'param_name': 'gainref_flip_y',\ 'param_type': 'boolean',\ 'base_value': False,\ 'title': 'Flip gain ref in Y?',\ 'desc': 'Flip gain ref top-to-bottom (in Y axis)'},\ {'param_sec': 'mscope_params',\ 'param_name': 'gainref_rotate_num',\ 'param_type': 'number',\ 'base_value': 0,\ 'title': 'Rotate gain ref?',\ 'desc': 'Rotate gain ref counter-clockwise by 90 degrees this many times'},\ {'param_sec': 'mscope_params',\ 'param_name': 'psize_A',\ 'param_type': 'number',\ 'base_value': None,\ 'title': 'Raw pixel size (A)',\ 'desc': 'Pixel size of the raw movie data in Angstroms'},\ {'param_sec': 'mscope_params',\ 'param_name': 'accel_kv',\ 'param_type': 'number',\ 'base_value': None,\ 'title': 'Accelerating voltage (kV)',\ 'desc': ''},\ {'param_sec': 'mscope_params',\ 'param_name': 'cs_mm',\ 'param_type': 'number',\ 'base_value': None,\ 'title': 'Spherical aberration (mm)',\ 'desc': ''},\ {'param_sec': 'mscope_params',\ 'param_name': 'total_dose_e_per_A2',\ 'param_type': 'number',\ 'base_value': None,\ 'title': 'Total exposure dose (e/A^2)',\ 'desc': ''},\ {'param_sec': 'mscope_params',\ 'param_name': 'phase_plate',\ 'param_type': 'boolean',\ 'base_value': False,\ 'title': 'Phase plate',\ 'desc': 'Were the images collected using a phase plate?'},\ {'param_sec': 'mscope_params',\ 'param_name': 'neg_stain',\ 'param_type': 'boolean',\ 'base_value': False,\ 'title': 'Negative stain',\ 'desc': 'Are the samples negative stain (True) or cryo (False)?'},\ {'param_sec': 'mscope_params',\ 'param_name': 'eer_upsampfactor',\ 'param_type': 'number',\ 'base_value': 2,\ 'title': 'EER upsampling factor',\ 'desc': 'EER upsampling factor (applies to .eer/.ecc format data only.'},\ {'param_sec': 'mscope_params',\ 'param_name': 'eer_numfractions',\ 'param_type': 'number',\ 'base_value': 40,\ 'title': 'EER number of fractions',\ 'desc': 'EER number of fractions (applies to .eer/.ecc format data only.'},\ {'param_sec': 'motion_settings',\ 'param_name': 'res_max_align',\ 'base_value': 5,\ 'title': 'Maximum alignment resolution (A)',\ 'param_type': 'number',\ 'desc': 'Maximum resolution (in A) to consider when aligning frames. Generally, betwen 5A and 3A is best.'},\ {'param_sec': 'motion_settings',\ 'param_name': 'bfactor',\ 'base_value': 500,\ 'title': 'B-factor during alignment',\ 'param_type': 'number',\ 'desc': 'B-factor that blurs frames before aligning. Generally 500 to 100 is best.'},\ {'param_sec': 'motion_settings',\ 'param_name': 'frame_start',\ 'base_value': 0,\ 'title': 'Start frame (included, 0-based)',\ 'param_type': 'number',\ 'desc': 'Which frame number, starting at zero, to begin motion correction from. This value controls how many early frames are dropped from the motion corrected result. This value will also be used in local motion correction.'},\ {'param_sec': 'motion_settings',\ 'param_name': 'frame_end',\ 'base_value': None,\ 'title': 'End frame (excluded, 0-based) ',\ 'param_type': 'number',\ 'desc': 'Which frame number, starting at zero, to not include in motion correction, also excluding all frames after this one. Generally this does not improve results, as later frames are downweighted during dose weighting in local motion correction.'},\ {'param_sec': 'motion_settings',\ 'param_name': 'output_fcrop_factor',\ 'base_value': 1,\ 'title': 'Output F-crop factor',\ 'param_type': 'number',\ 'desc': 'Output Fourier cropping factor. 1.0 means no cropping, 0.5 means crop to 1/2 the resolution, etc. Only 1, 0.75, 0.5, 0.25 are allowed values'},\ {'param_sec': 'motion_settings',\ 'param_name': 'override_total_exp',\ 'base_value': None,\ 'title': 'Override e/A^2',\ 'param_type': 'number',\ 'desc': 'Override the dose (in total e/A^2 over the exposure) that was given at import time but can be overridden here.'},\ {'param_sec': 'motion_settings',\ 'param_name': 'variable_dose',\ 'base_value': False,\ 'title': 'Allow Variable Dose',\ 'param_type': 'boolean',\ 'desc': 'Enable correct processing when frames have variable dose fractionation'},\ {'param_sec': 'motion_settings',\ 'param_name': 'smooth_lambda_cal',\ 'base_value': 0.5,\ 'title': 'Calibrated smoothing',\ 'param_type': 'number',\ 'desc': 'Calibrated smoothing constant applied to trajectories (None to autotune)'},\ {'param_sec': 'motion_settings',\ 'param_name': 'override_K_Z',\ 'base_value': None,\ 'title': 'Override knots Z',\ 'param_type': 'number',\ 'desc': 'Override automatically selected spline order for Z dimension (time)'},\ {'param_sec': 'motion_settings',\ 'param_name': 'override_K_Y',\ 'base_value': None,\ 'title': 'Override knots Y',\ 'param_type': 'number',\ 'desc': 'Override automatically selected spline order for Y dimension (vertical)'},\ {'param_sec': 'motion_settings',\ 'param_name': 'override_K_X',\ 'base_value': None,\ 'title': 'Override knots X',\ 'param_type': 'number',\ 'desc': 'Override automatically selected spline order for X dimension (horizontal)'},\ {'param_sec': 'motion_settings',\ 'param_name': 'optimize_for_gpu_memory',\ 'base_value': False,\ 'title': 'Low-memory mode',\ 'param_type': 'boolean',\ 'desc': 'If running out of GPU memory, this option can be used to prioritize memory use at the expense of speed (BETA). The results are unchanged.'},\ {'param_sec': 'ctf_settings',\ 'param_name': 'amp_contrast',\ 'base_value': 0.1,\ 'title': 'Amplitude Contrast',\ 'param_type': 'number',\ 'desc': 'Amplitude constrast to use. Typically 0.07 or 0.1 for cryo-EM data.'},\ {'param_sec': 'ctf_settings',\ 'param_name': 'res_min_align',\ 'base_value': 25,\ 'title': 'Minimum resolution (A)',\ 'param_type': 'number',\ 'desc': 'Minimum resolution (in A) to consider when estimating CTF.'},\ {'param_sec': 'ctf_settings',\ 'param_name': 'res_max_align',\ 'base_value': 4,\ 'title': 'Maximum resolution (A)',\ 'param_type': 'number',\ 'desc': 'Maximum resolution (in A) to consider when estimating CTF.'},\ {'param_sec': 'ctf_settings',\ 'param_name': 'df_search_min',\ 'base_value': 1000,\ 'title': 'Minimum search defocus (A)',\ 'param_type': 'number',\ 'desc': 'Defocus range for gridsearch.'},\ {'param_sec': 'ctf_settings',\ 'param_name': 'df_search_max',\ 'base_value': 40000,\ 'title': 'Maximum search defocus (A)',\ 'param_type': 'number',\ 'desc': 'Defocus range for gridsearch.'},\ {'param_sec': 'ctf_settings',\ 'param_name': 'do_phase_shift_search_refine',\ 'base_value': False,\ 'title': 'Do phase shift search',\ 'param_type': 'boolean',\ 'desc': 'Whether to carry out search and refinement over phase shift.'},\ {'param_sec': 'ctf_settings',\ 'param_name': 'phase_shift_min',\ 'base_value': 0,\ 'title': 'Min. search phase-shift (rad)',\ 'param_type': 'number',\ 'desc': 'Phase-shift range for gridsearch.'},\ {'param_sec': 'ctf_settings',\ 'param_name': 'phase_shift_max',\ 'base_value': 3.141592653589793,\ 'title': 'Max. search phase-shift (rad)',\ 'param_type': 'number',\ 'desc': 'Phase-shift range for gridsearch.'},\ {'param_sec': 'ctf_settings',\ 'param_name': 'do_phase_shift_refine_only',\ 'base_value': False,\ 'title': 'Do phase refine only',\ 'param_type': 'boolean',\ 'desc': 'Whether to carry out refinement over phase shift only'},\ {'param_sec': 'ctf_settings',\ 'param_name': 'override_K_Y',\ 'base_value': None,\ 'title': 'Override knots Y',\ 'param_type': 'number',\ 'desc': 'Override automatically selected spline order for Y dimension (vertical)'},\ {'param_sec': 'ctf_settings',\ 'param_name': 'override_K_X',\ 'base_value': None,\ 'title': 'Override knots X',\ 'param_type': 'number',\ 'desc': 'Override automatically selected spline order for X dimension (horizontal)'},\ {'param_sec': 'ctf_settings',\ 'param_name': 'classic_mode',\ 'base_value': False,\ 'title': 'Classic mode',\ 'param_type': 'boolean',\ 'desc': 'Uses the old Patch CTF algorithm (cryoSPARC v.2.15 and earlier) intead of the new one.'},\ {'param_sec': 'particle_picking',\ 'param_name': 'current_picker',\ 'base_value': 'blob',\ 'title': 'Current active picker',\ 'param_type': 'enum',\ 'enum_keys': ['blob', 'template', 'deep'],\ 'desc': 'Which picker type to use on future exposures.'},\ {'param_sec': 'blob_pick',\ 'param_name': 'diameter',\ 'base_value': None,\ 'title': 'Minimum particle diameter (A)',\ 'param_type': 'number',\ 'desc': 'Min Particle diameter (A)'},\ {'param_sec': 'blob_pick',\ 'param_name': 'diameter_max',\ 'base_value': None,\ 'title': 'Maximum particle diameter (A)',\ 'param_type': 'number',\ 'desc': 'Max Particle diameter (A)'},\ {'param_sec': 'blob_pick',\ 'param_name': 'use_circle',\ 'base_value': True,\ 'title': 'Use circular blob',\ 'param_type': 'boolean',\ 'desc': ''},\ {'param_sec': 'blob_pick',\ 'param_name': 'use_ellipse',\ 'base_value': False,\ 'title': 'Use elliptical blob',\ 'param_type': 'boolean',\ 'desc': ''},\ {'param_sec': 'blob_pick',\ 'param_name': 'use_ring',\ 'base_value': False,\ 'title': 'Use ring blob',\ 'param_type': 'boolean',\ 'desc': ''},\ {'param_sec': 'blob_pick',\ 'param_name': 'lowpass_res_template',\ 'base_value': 20,\ 'title': 'Lowpass filter to apply to templates (A)',\ 'param_type': 'number',\ 'desc': 'Lowpass filter to apply to templates, (A)s'},\ {'param_sec': 'blob_pick',\ 'param_name': 'lowpass_res',\ 'base_value': 20,\ 'title': 'Lowpass filter to apply (A)',\ 'param_type': 'number',\ 'desc': 'Lowpass filter to apply, (A)s'},\ {'param_sec': 'blob_pick',\ 'param_name': 'angular_spacing_deg',\ 'base_value': 5,\ 'title': 'Angular sampling (degrees)',\ 'param_type': 'number',\ 'desc': 'Angular sampling of templates in degrees. Lower value will mean finer rotations.'},\ {'param_sec': 'blob_pick',\ 'param_name': 'use_ctf',\ 'base_value': False,\ 'title': 'Use CTFs to filter the templates',\ 'param_type': 'boolean',\ 'desc': 'Whether to use micrograph CTF to filter the templates'},\ {'param_sec': 'blob_pick',\ 'param_name': 'min_distance',\ 'base_value': 1.0,\ 'title': 'Min. separation dist (diameters)',\ 'param_type': 'number',\ 'desc': 'Minimum distance between particles in units of particle diameter (min diameter for blob picker). The lower this value, the more and closer particles it picks.'},\ {'param_sec': 'blob_pick',\ 'param_name': 'num_process',\ 'base_value': None,\ 'title': 'Number of mics to process',\ 'param_type': 'number',\ 'desc': 'Number of micrographs to process. None means all.'},\ {'param_sec': 'blob_pick',\ 'param_name': 'num_plot',\ 'base_value': 10,\ 'title': 'Number of mics to plot',\ 'param_type': 'number',\ 'desc': 'Number of micrographs to plot.'},\ {'param_sec': 'blob_pick',\ 'param_name': 'max_num_hits',\ 'base_value': 4000,\ 'title': 'Maximum number of local maxima to consider',\ 'param_type': 'number',\ 'desc': 'Maximum number of local maxima (peaks) considered.'},\ {'param_sec': 'template_pick',\ 'param_name': 'diameter',\ 'base_value': None,\ 'title': 'Particle diameter (A)',\ 'param_type': 'number',\ 'desc': 'Particle diameter (A)'},\ {'param_sec': 'template_pick',\ 'param_name': 'lowpass_res_template',\ 'base_value': 20,\ 'title': 'Lowpass filter to apply to templates (A)',\ 'param_type': 'number',\ 'desc': 'Lowpass filter to apply to templates, (A)s'},\ {'param_sec': 'template_pick',\ 'param_name': 'lowpass_res',\ 'base_value': 20,\ 'title': 'Lowpass filter to apply (A)',\ 'param_type': 'number',\ 'desc': 'Lowpass filter to apply, (A)s'},\ {'param_sec': 'template_pick',\ 'param_name': 'angular_spacing_deg',\ 'base_value': 5,\ 'title': 'Angular sampling (degrees)',\ 'param_type': 'number',\ 'desc': 'Angular sampling of templates in degrees. Lower value will mean finer rotations.'},\ {'param_sec': 'template_pick',\ 'param_name': 'use_ctf',\ 'base_value': True,\ 'title': 'Use CTFs to filter the templates',\ 'param_type': 'boolean',\ 'desc': 'Whether to use micrograph CTF to filter the templates'},\ {'param_sec': 'template_pick',\ 'param_name': 'min_distance',\ 'base_value': 0.5,\ 'title': 'Min. separation dist (diameters)',\ 'param_type': 'number',\ 'desc': 'Minimum distance between particles in units of particle diameter. The lower this value, the more and closer particles it picks.'},\ {'param_sec': 'template_pick',\ 'param_name': 'num_process',\ 'base_value': None,\ 'title': 'Number of mics to process',\ 'param_type': 'number',\ 'desc': 'Number of micrographs to process. None means all.'},\ {'param_sec': 'template_pick',\ 'param_name': 'num_plot',\ 'base_value': 10,\ 'title': 'Number of mics to plot',\ 'param_type': 'number',\ 'desc': 'Number of micrographs to plot.'},\ {'param_sec': 'template_pick',\ 'param_name': 'max_num_hits',\ 'base_value': 4000,\ 'title': 'Maximum number of local maxima to consider',\ 'param_type': 'number',\ 'desc': 'Maximum number of local maxima (peaks) considered.'},\ {'param_sec': 'template_pick',\ 'param_name': 'templates_from_job',\ 'base_value': None,\ 'title': 'Tempates from job (PX.JXX)',\ 'param_type': 'string'},\ {'param_sec': 'template_pick',\ 'param_name': 'templates_selected',\ 'base_value': None,\ 'title': 'Templates selected (comma sep)',\ 'param_type': 'string'},\ {'param_sec': 'extraction',\ 'param_name': 'thresh_score_min',\ 'base_value': None,\ 'title': 'Score threshold min',\ 'param_type': 'number',\ 'desc': 'Minimum picking score threshold'},\ {'param_sec': 'extraction',\ 'param_name': 'thresh_power_min',\ 'base_value': None,\ 'title': 'Power threshold min',\ 'param_type': 'number',\ 'desc': 'Minimum picking power threshold'},\ {'param_sec': 'extraction',\ 'param_name': 'thresh_power_max',\ 'base_value': None,\ 'title': 'Power threshold max',\ 'param_type': 'number',\ 'desc': 'Maximum picking power threshold'},\ {'param_sec': 'extraction',\ 'param_name': 'box_size_pix',\ 'base_value': None,\ 'title': 'Extraction box size (pix)',\ 'param_type': 'number',\ 'desc': 'Size of box to be extracted from micrograph.'},\ {'param_sec': 'extraction',\ 'param_name': 'bin_size_pix',\ 'base_value': None,\ 'title': 'Fourier crop to box size (pix)',\ 'param_type': 'number',\ 'desc': 'Size of particle boxes after they have been extracted. None means use the same as the extraction box size'}] --- # Tutorial: Maximum Box Sizes for Refinement | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/performance-metrics.md) . [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/performance-metrics#maximum-box-sizes-for-various-amounts-of-gpu-memory) Maximum box sizes for various amounts of GPU memory ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The most important factors influencing memory usage for the 3D refinement jobs are the box size of the volume and particles used, as well as the computational batch size. Below is a tabulation of the approximate maximum volume box sizes that are supported by various GPU RAM sizes, valid for the following job types. * Homogeneous Refinement * Helical Refinement (BETA) * Local Refinement (BETA) Note that non-uniform regularization incurs additional memory cost that has not been benchmarked. The box sizes below are tabulated in the case where non-uniform regularization is disabled. GPU VRAM (GB) Approximate Max Volume Box Size (px) 4 682 8 872 11 976 12 1004 16 1110 24 and above 1126 In most 3D refinement and reconstruction jobs, the computational batch size can be changed by setting the "GPU batch size of images" or "Computational minibatch size" parameter. Decreasing this may alleviate GPU out-of-memory issues. You can also use the [Downsample Particles](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/extraction/job-downsample-particles) job to reduce the box size (and maximum resolution) of particles before refinement in order to reduce memory usage during refinement. [PreviousTutorial: Helical Processing using EMPIAR-10031 (MAVS)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs) [NextTutorial: CTF Refinement](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement) Last updated 2 years ago --- # Tutorial: Float16 Support | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-float16-support.md) . CryoSPARC v4.4 introduced support for saving particles and motion-corrected micrographs in float16 format, which retains fewer significant figures than the default setting (float32). This feature is off by default, but when turned on, reduces the size of the data produced by approximately half. To save data in float16 format, look for a toggle parameter called “Save results in 16-bit floating point” in motion correction or extraction jobs, and enable it. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FfJv8XOaYeMjXdLBPg9qO%252Fv4-4-0-float16-builder-1.png%3Falt%3Dmedia%26token%3D59fbfd86-bdd9-48b0-bcde-718e5fd96187&width=768&dpr=3&quality=100&sign=5928e73a&sv=2) Float16 is used as an on-disk format only. Once loaded from disk, computations within CryoSPARC are performed using 32 bits of precision. In principle, the reduction in significant figures has the potential to reduce reconstruction accuracy or detail. In practice, this is not expected to be a problem in most cases. The figures below compare FSC curves and an example of sharpened map quality between two separate homogeneous refinements with the same initial (ab-initio) volume, but different particle stacks (one extracted in float32, and one extracted in float16). FSCs are from **Validation (FSC)** jobs, where the same mask has been used in both cases (the mask from the float32 refinement). The float16 example comes after the float32 example. The protein is Apoferritin. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FXILsvHWkUudubcA24HG9%252Fv4-4-0-float16-fsc-1.png%3Falt%3Dmedia%26token%3D09600444-21fc-4999-afbd-dfd6294fcf61&width=768&dpr=3&quality=100&sign=1504814b&sv=2) Float32 ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FTmlNHEaZ3mkbQBkhDHzb%252Fv4-4-0-float16-fsc-2.png%3Falt%3Dmedia%26token%3Dfbfc76a1-d8a3-4b8d-ac98-d0f6fc8315da&width=768&dpr=3&quality=100&sign=a345d513&sv=2) Float16 And here is a comparison between the two sharpened maps. The blue map was refined from float16 particles, the gray from float32 ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FVRWta6nXc3oASIzxu3Ft%252Fv4-4-0-float16-compare-1.png%3Falt%3Dmedia%26token%3Ddbb70722-fd46-4681-9098-b7d581653697&width=768&dpr=3&quality=100&sign=b0f2ef15&sv=2) Float32 ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F88yKb64FIPzLQjDFXaoO%252Fv4-4-0-float16-compare-2.png%3Falt%3Dmedia%26token%3D3a39fa81-d0c8-42b6-8b66-eb6f8eb37922&width=768&dpr=3&quality=100&sign=2e7b7b4d&sv=2) Float16 ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FexhlVzU9m6ZyyZ5ltWSw%252Fv4-4-0-float16-compare-3.png%3Falt%3Dmedia%26token%3D28991324-c834-4ab6-8180-7317fb69f797&width=768&dpr=3&quality=100&sign=16a7fa01&sv=2) Float32 (gray solid) and Float16 (blue wireframe) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FzpJCcw99Es07dvkHT1Dm%252Fv4-4-0-float16-compare-4.png%3Falt%3Dmedia%26token%3D2197f2b2-dad8-4b36-87da-67cd71639e62&width=768&dpr=3&quality=100&sign=9ea6895b&sv=2) Float16 (blue solid) and Float32 (gray wireframe) For more information on floating-point data representations, refer to the following wikipedia article: [https://en.wikipedia.org/wiki/IEEE\_754](https://en.wikipedia.org/wiki/IEEE_754) or any other source that explains the IEEE 754 standard for floating point arithmetic and representation. [PreviousTutorial: Patch Motion and Patch CTF](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-patch-motion-and-patch-ctf) [NextTutorial: Particle Picking Calibration](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-particle-picking-calibration) Last updated 1 month ago --- # Tutorial: Particle Picking Calibration | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-particle-picking-calibration.md) . [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-particle-picking-calibration#defocus-variation) Defocus variation --------------------------------------------------------------------------------------------------------------------------------------------------------- Particle picking on cryo-EM data can be a challenge due to the presence of various complicating factors such as junk particles, aggregations, ice crystal formation, carbon edges, etc. However, one of the simplest variances amongst images that is not accounted for by most particle picking methods is defocus variation within a dataset. Defocus variation causes the same particles from the same viewing directions to look different, and also changes the signal and noise stastics of each micrograph. Thus, in particle picking tools, particles of equal quality from two micrographs with different defocus can be assigned different scores, making it difficult to set thresholds on picking scores in order to select good particles. CryoSPARC's blob picker and template picker have been susceptible to this problem. In CryoSPARC v2.13+, there is a new feature that directly calibrates pick scores against defocus, making it much easier to set thresholds when using the `inspect picks` job type. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-particle-picking-calibration#new-default-parameters-in-inspect-picks-job-v2.13) New default parameters in Inspect Picks job (v2.13+) ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To use picking calibration, perform blob or template picking as normal. Then, connect the `particles` output to an `inspect picks` job, along with `micrographs`. This job now has new parameters called `Calibrate Pick Score to CTF` and `Calibrate Power Score to CTF` that are turned on by default. When you run the `Inspect Picks` job with these default parameters, you will see plots that show micrograph median pick scores versus defocus in the streamlog: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNeBQxP7DOIFHyjNqP8%252F-MNeE0_evnNFKa4t18qn%252FCalib_particle-picking-calibration-1.png%3Falt%3Dmedia%26token%3D09e3c87b-d2b0-4e00-8b92-e01d66d186f7&width=768&dpr=3&quality=100&sign=af260621&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNeBQxP7DOIFHyjNqP8%252F-MNeE3epyQ6QrDDUlOXn%252FCalib_particle-picking-calibration-2.png%3Falt%3Dmedia%26token%3Dd142bda8-e1ef-45a0-893f-5d9d6b2ea25e&width=768&dpr=3&quality=100&sign=ea4c8fa3&sv=2) There will generally be a correlation for both the pick score and the power scores, which measure independently the shape and density of a particle candidate, respectively. You will also see fit calibration lines on these plots. After calibration, the scores of each particle will be recorded relative to the calibration line, and these values will be shown in the interactive part of the job that allows setting thresholds on the parameters. With calibration on, you should notice that good particles are clustered together more tightly in the pick score vs. power score plot. You should also notice that setting a threshold to select good particles in a particular micrograph also yields good picks on other micrographs that have very different defocus: ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-particle-picking-calibration#a-micrograph-at-1.2-mm-defocus) A micrograph at 1.2 **μm defocus** ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNeBQxP7DOIFHyjNqP8%252F-MNeEAIEATU0XYmlPLqS%252FCalib_particle-picking-calibration-3.png%3Falt%3Dmedia%26token%3Dc4f4ed76-5aa2-4786-ba0f-c2673a11e48b&width=768&dpr=3&quality=100&sign=35758ccc&sv=2) ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-particle-picking-calibration#the-same-thresholds-for-a-micrograph-at-2.2-mm-defocus) **The same thresholds, for a micrograph at 2.2 μm defocus** ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNeBQxP7DOIFHyjNqP8%252F-MNeECoaVOdhyg5LeYdP%252FCalib_particle-picking-calibration-4.png%3Falt%3Dmedia%26token%3D6e8cc03d-f353-42f2-a9eb-8eadf635dbf2&width=768&dpr=3&quality=100&sign=80e626ee&sv=2) [PreviousTutorial: Float16 Support](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-float16-support) [NextTutorial: Blob Picker Tuner](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-blob-picker-tuner) Last updated 3 years ago * [Defocus variation](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-particle-picking-calibration#defocus-variation) * [New default parameters in Inspect Picks job (v2.13+)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-particle-picking-calibration#new-default-parameters-in-inspect-picks-job-v2.13) * [A micrograph at 1.2 μm defocus](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-particle-picking-calibration#a-micrograph-at-1.2-mm-defocus) * [The same thresholds, for a micrograph at 2.2 μm defocus](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-particle-picking-calibration#the-same-thresholds-for-a-micrograph-at-2.2-mm-defocus) --- # Tutorial: BILD files | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-bild-files.md) . BILD files are now produced for all viewing direction distribution figures in CryoSPARC v4.4+. [BILD files](https://www.cgl.ucsf.edu/chimera/docs/UsersGuide/bild.html) are simple format for producing 3D shapes in UCSF Chimera(X). These files can be useful to visualize 3D viewing direction distributions using coloured cylinders of varying height. Prior to CryoSPARC v4.4., a standard way to construct BILD files from CryoSPARC refinement jobs required the use of the `csparc2star.py` and `star2bild.py` scripts within the [external open-source package pyem](https://github.com/asarnow/pyem/blob/master/star2bild.py) . To make this process easier, CryoSPARC v4.4 now includes BILD files with every viewing direction distribution figure in the event log. To use these files, click the `[bild]` link on top of any viewing direction figure to download the BILD file: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FasLV3V4CDBxf4qqKb1Df%252Fv4_4_bild_file_viewingdist_fig.png%3Falt%3Dmedia%26token%3D45035045-76bc-4357-a297-8bff354872ed&width=768&dpr=3&quality=100&sign=d8d80480&sv=2) Once downloaded, you should be able to open this file with [UCSF Chimera (X)](https://www.cgl.ucsf.edu/chimerax/) and overlay it on top of a 3D map: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FsHq07frQaMEXLcm9yF3o%252Fv4_4_bild_file_chimera.png%3Falt%3Dmedia%26token%3D3c3aed39-c990-4009-a8c8-3e623f407019&width=768&dpr=3&quality=100&sign=2609a49f&sv=2) UCSF Chimera X window with a refined map and an overlayed viewing distribution ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-bild-files#bild-file-structure) BILD File Structure Following `star2bild.py` , we generate BILD files by creating spherical bins using the [HealPix algorithm](https://healpix.jpl.nasa.gov/) . Each spherical bin is visualized as a cylinder whose base is on a sphere with radius r\=12Npr = \\frac{1}{2} N pr\=21​Np , where NNN is the box size and ppp is the pixel size in Angstroms. Cylinder height is scaled linearly with relative particle population size, ranging from 0 (displayed as flat blue circle) to 0.3r0.3 r0.3r (displayed as a red cylinder). [PreviousTutorial: Orientation Diagnostics](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-orientation-diagnostics) [NextTutorial: Mask Creation](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera) Last updated 2 years ago --- # Job: Split Volumes Group | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group.md) . [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#at-a-glance) At a Glance ----------------------------------------------------------------------------------------------------------------------------------------- Split a volumes group into several individual volume outputs, optionally splitting particles as well. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#description) Description ----------------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#split-volumes-group-job) Split Volumes Group Job The Split Volumes Group job type allows for converting a volumes group into a number of individual volume outputs. This is useful if specific volumes from within a volumes group need to be manually selected or connected to downstream jobs. A set of particles corresponding to the volumes group can also be connected as input to the job, and those particles will be split according to their classification amongst the volumes in the group. ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#volumes-groups) Volumes groups Volumes groups are new in CryoSPARC v4.5+. In CryoSPARC, jobs such as Ab-Initio Reconstruction, Heterogeneous Refinement, 3D Classification, Regroup 3D Classes, Heterogeneous Reconstruct Only, Orientation Diagnostics and Align 3D Maps, all produce one or more `volume` outputs. Where multiple volumes are produced, each is output separately, e.g. Volume class 0, Volume class 1, etc. In v4.5+, the above job types also produce a **Volumes group** output, denoted by the type `volume_multi`, which contains multiple volumes in a single output. Some jobs in v4.5+ also take volumes groups as input. Unlike an individual volume output, a volumes group can contain an arbitrary number of volumes unknown at the job’s build time, can include multiple versions of each volume such as `map` and `map_sharp`, and can also contain metadata information about each of those volumes such as resolution info. For jobs that produces many volumes in a volumes group, all the volumes can be connected to a downstream job with a single connection of the volumes group, rather than many individual drag-and-drop actions to connect each individual volume. Volumes group connections can also be used when setting up [Workflows](https://guide.cryosparc.com/application-guide/workflows) . Note that a volumes group (`volume_multi` type) is not the same as an individual 3D volume output (`volume` type). These two types cannot be interchanged, and jobs requiring `volume_multi` as input can not take in `volume` inputs. Jobs run in CryoSPARC versions prior to v4.5 do not produce volumes group outputs and therefore cannot be connected to jobs that require volumes group inputs in v4.5+. `volume_multi` outputs have a new output field, `alignments3D_multi`. We do not expect users will need to interact with this field in typical CryoSPARC usage, but some external software (such as pyem) may need to be updated to handle the new data format. ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#when-to-use-split-volumes-groups) When to use Split Volumes Groups Split Volumes Groups can be used with any job that produces a volumes group in order to separate the volumes from the group into individual volume outputs, as well as associated particle outputs if a particle stack is provided. For example: * After Reference Based Auto Select 3D, use Split Volumes Groups to split apart the Volumes selected output into individual volumes, so that a particular volume can be used for refinement. * After 3D Variability Display, use Split Volumes Groups to split apart one of the Volume series outputs in order to use a few volumes from different positions along the 3D variability components as input volumes for Heterogeneous Refinement or 3D Classification. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#inputs) Inputs ------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#all-particles) All particles This is an optional input. If provided, particles require 3D alignments corresponding to the volumes group to be split, and so should come from the same job that produced the volumes group connected to the All volumes input. ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#all-volumes) All volumes The volumes group to be split. This group can contain an arbitrary number of volumes. If the group contains multiple versions of each volume (e.g., `map` , `map_sharp`, etc.), then all versions of each volume will be output in each of the individual volume outputs. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#commonly-adjusted-parameters) Commonly Adjusted Parameters --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#static-number-of-output-volumes) Static number of output volumes This is an optional parameter. Fix the number of single volume outputs this job will produce, regardless of how many volumes are in the input group. When this parameter is set to N, the first N volumes from the input volumes group will be produced as individual output volumes. This parameter is necessary for using the Split Volumes Group job in Workflows, so that volume outputs from this job are available for making connections downstream before this job is actually run. If there are more volumes present in the input group than this parameter, the extras will not be output; if there are fewer volumes present, the job will contain some unpopulated outputs. When not set, the job will dynamically produce volume outputs at run time. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#outputs) Outputs --------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#particles-class-x) Particles class X _This output is only produced if particles were connected to the All particles input._ Subset of input particles that correspond to class X. ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#volume-class-x) Volume class X An individual volume output containing all versions of the volume for class X. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#downloading-a-volumes-group-series) Downloading a volumes group: `series` ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Note that some jobs that output volumes groups also provide a `series` result as part of the volumes group. The `series` (or `series_sharp`, etc.) results serve as a convenient way to download a zip file containing all of the volumes in the group, for visualization: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FggThM87QwZu0B7j1o2WN%252Fv4-5-0_download_series.png%3Falt%3Dmedia%26token%3D5a5d3e7f-d54f-4acc-93c5-608ef2540bd1&width=768&dpr=3&quality=100&sign=a864aeaf&sv=2) [PreviousJob: Select Volume](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume) [NextJob: Orientation Diagnostics](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-orientation-diagnostics) Last updated 1 month ago * [At a Glance](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#at-a-glance) * [Description](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#description) * [Split Volumes Group Job](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#split-volumes-group-job) * [Volumes groups](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#volumes-groups) * [When to use Split Volumes Groups](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#when-to-use-split-volumes-groups) * [Inputs](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#inputs) * [All particles](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#all-particles) * [All volumes](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#all-volumes) * [Commonly Adjusted Parameters](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#commonly-adjusted-parameters) * [Static number of output volumes](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#static-number-of-output-volumes) * [Outputs](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#outputs) * [Particles class X](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#particles-class-x) * [Volume class X](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#volume-class-x) * [Downloading a volumes group: series](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group#downloading-a-volumes-group-series) --- # Case Study: Processing EMPIAR-10291 (300 Micrographs) to 3.4Å in 1 hour 25 minutes | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-processing-empiar-10291-300-micrographs-to-3.4a-in-1-hour-25-minutes.md) . [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-processing-empiar-10291-300-micrographs-to-3.4a-in-1-hour-25-minutes#structure-of-an-undocked-hemichannel-of-the-n-terminal-deleted-inx-6-in-a-nanodisc) Structure of an Undocked Hemichannel of the N-terminal-deleted INX-6 in a Nanodisc ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Three structures from a [recent paper in _Science Advances_](https://advances.sciencemag.org/content/6/7/eaax3157) (Burendei _et. al_, _Science Advances_ 2020) depicting the Cryo-EM structures of undocked innexin-6 hemichannels in phospholipids were recently released on EMPIAR, Electron Microscopy Public Image Archive: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNjBy5PG6rWy1X0buQF%252F-MNjDMELLyOD0XQ0lO6R%252FHEMI_1_EMPIAR-10289-10290-10291.png%3Falt%3Dmedia%26token%3De476fb61-6b94-475f-9185-c4d801f18855&width=768&dpr=3&quality=100&sign=1910a3ec&sv=2) EMPIAR-10289 in light blue, EMPIAR-10290 in green, EMPIAR-10291 in light yellow This case study focuses on [EMPIAR-10291](https://www.ebi.ac.uk/pdbe/emdb/empiar/entry/10291/) , which contains 300 motion-corrected micrographs. In CryoSPARC, you can resolve a 3.4Å structure (published resolution of 3.6Å) with no manual picks and little configuration in less than 90 minutes: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNjdTmO4HRsVlp_Gy_Z%252F-MNjdbd8xz5b9dsBs9mt%252Fempiar-10291-gif-1.gif%3Falt%3Dmedia%26token%3D4c198d0f-08fa-49f6-a74b-319d34bc7dd3&width=768&dpr=3&quality=100&sign=2026073d&sv=2) ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-processing-empiar-10291-300-micrographs-to-3.4a-in-1-hour-25-minutes#workflow-in-cryosparc) Workflow in CryoSPARC ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNjBy5PG6rWy1X0buQF%252F-MNjDapiKppOG2GPZWyl%252FHEMI-2_EMPIAR-10291-Case-Study.png%3Falt%3Dmedia%26token%3D9d41491c-92c9-4b81-a7ea-1058aabf2e6a&width=768&dpr=3&quality=100&sign=18bdad3b&sv=2) [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-processing-empiar-10291-300-micrographs-to-3.4a-in-1-hour-25-minutes#iterative-workflows-in-cryosparc) Iterative Workflows in CryoSPARC --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The primary benefit of CryoSPARC's speed is the ability to quickly iterate through the cryo-EM data processing pipeline and experiment along the way. It is recommended to perform a first-pass workflow from raw data through to a refined structure (as outlined above) to get a sense of the quality of your data before (or concurrently with) proceeding to optimize various stages of the processing pipeline. The quality of a reconstruction is dependent on optimizing various stages of the pipeline: ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-processing-empiar-10291-300-micrographs-to-3.4a-in-1-hour-25-minutes#id-1-pre-processing-exposure-curation-patch-ctf-estimation) **1) Pre-processing (Exposure Curation, Patch CTF Estimation)** * Exposure curation can assist with filtering exposures with poor CTF fits or bad ice; this helps increase the quality of particle picks * Generally, CTF estimation auto-tunes parameters based on the input data and does not require tweaking. It also handles tilt data directly with no changes. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-processing-empiar-10291-300-micrographs-to-3.4a-in-1-hour-25-minutes#id-2-particle-curation-picking-2d-classification) **2) Particle curation (picking, 2D classification)** * Experimenting with the minimum and maximum particle diameter parameters in the Blob Picker job in combination with different particle extraction box sizes in 2D Classification to better the quality of 2D Classes. Generally, the particle should be half or less the width of the box. * We experimented with using refined volumes (after getting a first reconstruction) and the "Create Templates" job to generate 2D projections and feed those into a "Template Picking" job to improve picks. This was found not to yield a higher resolution result because the blob picker already did very well. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-processing-empiar-10291-300-micrographs-to-3.4a-in-1-hour-25-minutes#id-3-reconstruction-and-refinement-variations) **3) Reconstruction and refinement variations** * In this scenario, a single class ab-initio resulted in the best input for the refinement * Multi-class ab-initio can help to filter particles in addition to 2D Classification * Multi-class heterogenous refinement is also useful for pruning outlier particles in later stages of processing. This can be started from multiple ab-initio volumes. * 'New Homogeneous Refinement' is recommended as it features many performance enhancements and the ability to perform on-the-fly CTF refinement versus the Legacy Refinement. We were able to complete the first refinement in under 10 minutes on a single GPU. * In this case, we found that CTF refinement did not help in the final reconstruction, as the particle is a membrane protein and so the disorder in the micelle makes it difficult for CTF refinement to correctly estimate the defocus or higher order aberrations present during imaging. * Non-uniform refinement was used due to the micelle surrounding the target. This provided a small improvement in structure resolution, but did take 1 hour instead of the 10 minutes for a standard homogeneous refinement. Below is an example of this iterative workflow in action: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNjBy5PG6rWy1X0buQF%252F-MNjDlFjJMNcqIlEnxUP%252FHEMI-3_empiar-10291-iterative.png%3Falt%3Dmedia%26token%3De7a6431c-a187-43bc-b2b6-7005d600842c&width=768&dpr=3&quality=100&sign=52f6e556&sv=2) Multiple rounds of the New Refinement and Non-Uniform Refinement were conducted on different ab-initio classes. Last updated 1 year ago * [Structure of an Undocked Hemichannel of the N-terminal-deleted INX-6 in a Nanodisc](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-processing-empiar-10291-300-micrographs-to-3.4a-in-1-hour-25-minutes#structure-of-an-undocked-hemichannel-of-the-n-terminal-deleted-inx-6-in-a-nanodisc) * [Workflow in CryoSPARC](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-processing-empiar-10291-300-micrographs-to-3.4a-in-1-hour-25-minutes#workflow-in-cryosparc) * [Iterative Workflows in CryoSPARC](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-processing-empiar-10291-300-micrographs-to-3.4a-in-1-hour-25-minutes#iterative-workflows-in-cryosparc) * [1) Pre-processing (Exposure Curation, Patch CTF Estimation)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-processing-empiar-10291-300-micrographs-to-3.4a-in-1-hour-25-minutes#id-1-pre-processing-exposure-curation-patch-ctf-estimation) * [2) Particle curation (picking, 2D classification)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-processing-empiar-10291-300-micrographs-to-3.4a-in-1-hour-25-minutes#id-2-particle-curation-picking-2d-classification) * [3) Reconstruction and refinement variations](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-processing-empiar-10291-300-micrographs-to-3.4a-in-1-hour-25-minutes#id-3-reconstruction-and-refinement-variations) --- # Case Study: Exploratory data processing by Oliver Clarke | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-exploratory-data-processing-by-oliver-clarke.md) . _The guides presented here are kindly reproduced from Oliver Clarke, PhD, Assistant Professor of Physiology and Cellular Biophysics at Columbia University. They are friendly, approachable introductions to cryoEM data processing in CryoSPARC with a focus on the exploratory, "try-it-and-see" nature of single-particle analysis._ _Both guides cover similar topics, but the 2024 version includes some steps which require CryoSPARC v4.4 or later. Below we directly reproduce the general outline section of each guide. The full guide is available in the linked PDF._ * * * [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-exploratory-data-processing-by-oliver-clarke#id-2024-version) 2024 Version -------------------------------------------------------------------------------------------------------------------------------------------------------------------- 13MB [exploratory\_data\_processing\_stockholm\_2024.pdf](https://1916621962-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7DGv3GkRvGGpbVPCgg%2Fuploads%2FDj7poJysPDJj5I7VQKnN%2Fexploratory_data_processing_stockholm_2024.pdf?alt=media&token=39d180e7-5f2a-4673-aca5-f26a120d2383) PDF Download[Open](https://1916621962-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7DGv3GkRvGGpbVPCgg%2Fuploads%2FDj7poJysPDJj5I7VQKnN%2Fexploratory_data_processing_stockholm_2024.pdf?alt=media&token=39d180e7-5f2a-4673-aca5-f26a120d2383) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FSb5EoaiqxtSAUvVSdOGC%252Fimage.png%3Falt%3Dmedia%26token%3D598e28a7-da30-4a91-b683-4fd828f56843&width=768&dpr=3&quality=100&sign=5d7ffebf&sv=2) General workflow for the tutorial (sections 2-8) This workshop is intended to provide an introduction to "exploratory" data processing in CryoSPARC - that is, data processing with the goal of quickly identifying, reconstructing and refining the molecular species present in a heterogeneous sample. CryoSPARC is used here, but the same general principles & workflow apply to single particle processing in any software package (the particles and micrographs should be directly importable into RELION - just convert the `particle.cs` file to a STAR file using `csparc2star.py` in `pyem` (see [here](https://github.com/asarnow/pyem) ) and you should be good to go). _Note: some parts (e.g. symmetry relaxation) require CS 4.4 or later._ **General principles to keep in mind** 1. Process _small, clean, subsets_ of your dataset before tackling the whole. There are many choices to make during data processing - What picking strategy to use? What cleaning/classification strategy to use? What molecular species are present, and which to focus on? In many cases, the only way to identify the best performing strategy is by trial and error. This is _**much**_ faster working with a smaller subset of data, and can provide 3D volumes and strategies which can then be used to seed analysis of the entire dataset. 2. Iterate! Often, optimal processing of a heterogeneous dataset will benefit from multiple passes. The first quick pass identifies any potential issues (non-optimal orientation distribution, variable behavior of particles in different ice thickness regimes) and facilitates identification of the very best micrographs (those with the most particles remaining after initial picking and classification), which can then be used to train a neural network picker such as Topaz to repick the entire dataset. 3. Experiment/explore! There is no single valid strategy for processing a heterogeneous dataset, and this workshop is only a brief guide to some possible approaches. Mix and match, test what works best, and then apply these strategies to your own data! I have included two subsets of data for the first part of the workshop (micrographs and extracted & Fourier cropped particles) derived from a publicly available heterogeneous dataset - [EMPIAR-11043](https://www.ebi.ac.uk/empiar/EMPIAR-11043/) , the erythrocyte ankyrin-1 complex purified from digitonin extracts of human red blood cell membranes (PMID: [35835865](https://pubmed.ncbi.nlm.nih.gov/35835865/) ). For the second part of the workshop, addressing mixed symmetry and pseudosymmetry, I have included subsets of data from [EMPIAR-10425](https://www.ebi.ac.uk/empiar/EMPIAR-10425/) (the MlaBDEF complex, PMID: [34188171](https://pubmed.ncbi.nlm.nih.gov/34188171/) ), as well as [EMPIAR-10059](https://www.ebi.ac.uk/empiar/EMPIAR-10059/) (TRPV1-DkTx complex, PMID: [27281200](https://pubmed.ncbi.nlm.nih.gov/27281200/) ). These datasets are intended to provide a lightweight and portable starting point for data processing initiated from either CTF estimation and picking (micrographs) or _ab-initio_ volume generation and classification (particles), which can be easily accommodated even on systems with limited storage and processing power. Both sets of data are relatively small, but large enough to allow for identification and characterization of multiple species over the course of the workshop. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-exploratory-data-processing-by-oliver-clarke#id-2023-version) 2023 Version -------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2MB [exploratory\_data\_processing\_workshop.pdf](https://1916621962-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7DGv3GkRvGGpbVPCgg%2Fuploads%2FvtAyL9jvpWJyc3uDDLKd%2Fexploratory_data_processing_workshop.pdf?alt=media&token=9fe1d30f-ec82-4c62-b933-7435369c58b7) PDF Download[Open](https://1916621962-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-M7DGv3GkRvGGpbVPCgg%2Fuploads%2FvtAyL9jvpWJyc3uDDLKd%2Fexploratory_data_processing_workshop.pdf?alt=media&token=9fe1d30f-ec82-4c62-b933-7435369c58b7) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F6dUGEGyhInxsZDnAF7xU%252Fimage.png%3Falt%3Dmedia%26token%3D439127fb-7a70-41e5-8dfc-d4c19a0fec05&width=768&dpr=3&quality=100&sign=8559f1f0&sv=2) General workflow for the tutorial. This workshop is intended to provide an introduction to "exploratory" data processing in CryoSPARC - that is, data processing with the goal of quickly identifying, reconstructing and refining the molecular species present in a heterogeneous sample. CryoSPARC is used here, but the same general principles & workflow apply to single particle processing in any software package (the particles and micrographs should be directly importable into RELION - just convert the `particle.cs` file to a STAR file using `csparc2star.py` in `pyem` (see [here](https://github.com/asarnow/pyem) ) and you should be good to go). **General principles to keep in mind:** 1. Process _small, clean, subsets_ of your dataset before tackling the whole. There are many choices to make during data processing - What picking strategy to use? What cleaning/ classification strategy to use? What molecular species are present, and which to focus on? In many cases, the only way to identify the best performing strategy is by trial and error. This is _**much**_ faster working with a smaller subset of data, and can provide 3D volumes and strategies which can then be used to seed analysis of the entire dataset. 2. Iterate! Often, optimal processing of a heterogeneous dataset will benefit from multiple passes. The first quick pass identifies any potential issues (non-optimal orientation distribution, variable behavior of particles in different ice thickness regimes) and facilitates identification of the very best micrographs (those with the most particles remaining after initial picking and classification), which can then be used to train a neural network picker such as Topaz to repick the entire dataset. 3. Experiment/explore! There is no single valid strategy for processing a heterogeneous dataset, and this workshop is only a brief guide to some possible approaches. Mix and match, test what works best, and then apply these strategies to your own data! I have included two subsets of data (micrographs and extracted & Fourier cropped particles) derived from a publicly available heterogeneous dataset - [EMPIAR-11043](https://www.ebi.ac.uk/empiar/EMPIAR-11043/) , the erythrocyte ankyrin-1 complex purified from digitonin extracts of human red blood cell membranes. These datasets are intended to provide a lightweight and portable starting point for data processing initiated from either CTF estimation and picking (micrographs) or _ab-initio_ volume generation and classification (particles), which can be easily accommodated even on systems with limited storage and processing power. Both sets of data are relatively small, but large enough to allow for identification and characterization of multiple species over the course of the workshop. [PreviousCase Study: Picking-induced Orientation Bias in HA Trimer (EMPIAR-10096 and -10097)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-picking-induced-orientation-bias-in-ha-trimer-empiar-10096-and-10097) [NextTutorial: Tips for Membrane Protein Structures](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures) Last updated 1 month ago * [2024 Version](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-exploratory-data-processing-by-oliver-clarke#id-2024-version) * [2023 Version](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-exploratory-data-processing-by-oliver-clarke#id-2023-version) --- # Tutorial: Patch Motion and Patch CTF | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-patch-motion-and-patch-ctf.md) . [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-patch-motion-and-patch-ctf#patch-based-motion-correction) Patch-Based Motion Correction ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- `Patch Motion` is a very fast, auto-tuning patch-based local motion correction method built on top of CryoSPARC's original `Local Motion Correction` method (a variant of _alignparts_). `Patch Motion` performs the functions of both `Full-frame Motion Correction` and `Local Motion Correction` in one job. Particle locations are **not needed** beforehand. The job outputs non-dose weighted **and** dose-weighted micrographs, ready for CTF Estimation using the new `Patch CTF` job. **Performance:** Less than 10s/movie (single GPU) for both full-frame and local anisotropic motion correction. Uses less than 10GB of GPU RAM, allowing for large movies to be processed seamlessly on commonly available GPUs. Multi-GPU mode also available. Natively supports parallelized TIFF and MRC.BZ2 decompression. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-patch-motion-and-patch-ctf#patch-based-ctf-estimation) Patch-Based CTF Estimation ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- `Patch CTF` fits a locally-variable CTF landscape to each micrograph and is based on new robust methods for reference-free background and CTF envelope estimation, and _LBFGS_ optimization to maximize simulated 2D CTF fit. Particle locations are **not needed**. Together these enable `Patch CTF` to work very well on tilt data, bent/deformed ice, exposures containing very small particles, phase plate data and a range of defocus values. **Performance:** Outputs per-particle local CTF estimates in ~3s/micrograph (single GPU). Multi-GPU mode also available. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-patch-motion-and-patch-ctf#workflow-example) Workflow Example ----------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-patch-motion-and-patch-ctf#id-1.-import) **1\. Import** You will need to import raw movies using the `Import Movies` job, ensuring you specify the `Gain reference path` if available, `Raw pixel size (A)`, `Accelerating voltage (kV)`, `Spherical aberration (mm)` and `Total exposure dose (e/A^2)`. If movies have been previously imported, you can start this workflow directly. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-patch-motion-and-patch-ctf#id-2.-patch-motion) **2\. Patch Motion** Build a new `Patch Motion Correction` job and drag in the imported movies as Inputs. You can choose to limit processing to a certain number of movies by entering an integer value in `Only process this many movies`. This parameter is useful if you want to understand motion from a subset of movies in a large dataset. Once `Queued` and running, rigid and patch motion trajectory plots can be viewed in the job streamlog for each exposure (example below). The job will output dose weighted and non-dose weighted micrographs as a single output group. (Check out this [tutorial on Inputs and Outputs in CryoSPARC](https://guide.cryosparc.com/guides-for-v3/job-builder-tutorial) to learn more about working with individual outputs.) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNe9qMGqxhKztbCGyNm%252F-MNeAqa-kn7-_9FA1DGH%252FPatch_patch_motion.png%3Falt%3Dmedia%26token%3Deba2e03f-370f-4961-b997-e4b19a793f67&width=768&dpr=3&quality=100&sign=6c6b1799&sv=2) ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-patch-motion-and-patch-ctf#id-3.-patch-ctf) **3\. Patch CTF** Drag and drop the motion-corrected micrographs into a new `Patch CTF` job. Once `Queued` and running, you can view progress and diagnostic plots in the streamlog: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNe9qMGqxhKztbCGyNm%252F-MNeAtiI24ZmOq-axl-c%252FPatch_3d_patch.png%3Falt%3Dmedia%26token%3D3424a12d-bbb4-4f33-a0b7-a9fc01d18c0f&width=768&dpr=3&quality=100&sign=74d1035c&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNe9qMGqxhKztbCGyNm%252F-MNeAwx10CSDwhB2mxv7%252FPatch_2d_patch.png%3Falt%3Dmedia%26token%3D9b7487b0-6c07-4206-b7ee-ab4adefc3292&width=768&dpr=3&quality=100&sign=eb7107de&sv=2) The 3D surface plot above shows the local defocus estimated across the micrograph. Units of the X Y and Z axes are all Angstroms. The 1D CTF fit plot shows the fit between the simulated and observed Thon rings in the micrograph (correcting for defocus variations and astigmatism). The light blue line indicates the cross-correlation fit level, and the CTF fit resolution is the resolution at which this value drops below a threshold. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-patch-motion-and-patch-ctf#id-4.-picking-if-no-prior-particle-locations-available-or-you-wish-to-pick-particles-again) **4\. Picking (If no prior particle locations available or you wish to pick particles again)** If working with fresh movies for which no particle locations are available, or if you wish to perform picking again, you can proceed to particle picking using the `Manual Picker`, New `Blob-Based Picker` or `Template Picker`. (**NB.** CryoSPARC's new Blob Picker is a fully automatic picker that uses circular and/or elliptical blobs in less than 1s/micrograph on a single GPU.) ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-patch-motion-and-patch-ctf#id-5.-extraction) **5\. Extraction** Following picking, particles can be extracted using `Extract from Micrographs`. This job will also extract local CTF values from the Patch CTF model that has been estimated. Note that `Local Motion Correction` **no longer** needs to be performed. If you already have a group of extracted particles in an existing project, and have only re-computed the Patch CTF for corresponding micrographs, you can proceed to re-extract _only_ the CTF values per-particle using a `Patch CTF Extraction` job, ensuring you drag the CTF-estimated exposures and particle locations as inputs. This will leave previously extracted or local-motion-corrected particle data intact. [PreviousTutorial: EPU AFIS Beam Shift Import](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import) [NextTutorial: Float16 Support](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-float16-support) Last updated 2 years ago * [Patch-Based Motion Correction](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-patch-motion-and-patch-ctf#patch-based-motion-correction) * [Patch-Based CTF Estimation](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-patch-motion-and-patch-ctf#patch-based-ctf-estimation) * [Workflow Example](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-patch-motion-and-patch-ctf#workflow-example) * [1\. Import](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-patch-motion-and-patch-ctf#id-1.-import) * [2\. Patch Motion](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-patch-motion-and-patch-ctf#id-2.-patch-motion) * [3\. Patch CTF](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-patch-motion-and-patch-ctf#id-3.-patch-ctf) * [4\. Picking (If no prior particle locations available or you wish to pick particles again)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-patch-motion-and-patch-ctf#id-4.-picking-if-no-prior-particle-locations-available-or-you-wish-to-pick-particles-again) * [5\. Extraction](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-patch-motion-and-patch-ctf#id-5.-extraction) --- # Job: Volume Alignment Tools | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools.md) . [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools#description) Description -------------------------------------------------------------------------------------------------------------------------------------------- The Volume Alignment Tools utility is a collection of low-level tools to adjust the position and orientation of volumes and particles in 3D space. The main functions of the utility are: 1. To apply an arbitrary rigid transformation (shift and/or rotation) to a 3D volume 2. To align a volume to CryoSPARC's symmetry axis conventions, via a rigid transformation The utility can also ingest a stack of particles. If particles are supplied to the job, their alignments will be modified in concert with the volume to keep them correctly aligned to the output volume. If a mask is supplied, it will be transformed by the same rigid transformation applied to the volume. Note that if both shifting and rotation are performed, then the shifting will occur before the rotation. The major use cases of this job are in: * shifting and/or rotating the volume and particles to focus on a subunit (for subsequent local refinement, or for re-centering the particles prior to extraction from micrographs) * aligning the volume and particles to the correct symmetry axes [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools#input) Input -------------------------------------------------------------------------------------------------------------------------------- * Volume (required) * Mask (optional) * Particles (optional) [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools#common-parameters) Common Parameters -------------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools#re-centering-parameters) Re-centering Parameters ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FP5BfsroVMhtIkpk4Eeuo%252Fv4-5-0-vol-align-tools-recenter-params.png%3Falt%3Dmedia%26token%3D065b4c19-b9d9-4c5b-ac7c-176dbb5a6916&width=768&dpr=3&quality=100&sign=4f797c5d&sv=2) * Re-center to mask center of mass: Activate this to re-center the volume, particles, and mask to the center of mass of the supplied mask. If activated, this will override the value supplied to the "3D coordinates of new center" parameter. * 3D coordinates of new center (A or px): provide as comma-separated triplet, relative to the corner of the volume. Should be a string of the form `x,y,z` with a suffix of either `A` or `px` to specify the units (Angstroms or pixels, respectively); no suffix will be interpreted as pixels. If provided, volume will be shifted such that its center is at this location, and particle shifts will be modified to preserve their alignment. Leave blank to do no shifting. Shifting will be done before symmetry alignment, if enabled. Enter as a comma separated triplet, e.g. `120, 100, 135 px` * 3D rotation euler angles (deg or rad): provide as comma-separated triplet. The inputs will be transformed via these Euler angles, following the ZYZ convention. Will be applied after shift. Should be a string of the form `Z,Y,Z` with a suffix of either `deg` or `rad` to specify the units (degrees or radians, respectively); no suffix will be interpreted as radians. For example, to rotate the inputs by 90 degrees around the Z-axis, the rotation string `0, 0, 1.57 rad` may be used ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools#symmetry-alignment-parameters) Symmetry Alignment Parameters ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FRjGaZomyz4LldLGtjTNE%252Fv4-1-0-vol-align-tools-sym-align-params-0.png%3Falt%3Dmedia%26token%3Db4007b5e-008c-40de-9cc4-8504989d3dca&width=768&dpr=3&quality=100&sign=37fd3e41&sv=2) * Symmetry string: C, D, I, O, or T, indicating the desired symmetry group to use for alignment. * Helical twist (degrees), rise (Å), and symmetry order. * Ignore shifts in symmetry alignment: Activate to prevent searching over shifts during symmetry alignment; this will speed up the job if you know the volume is correctly centred. * Verbose: Activate to print intermediate rotation matrices and shifts. * Advanced Parameters * Search over all of SO(3): If active, will search over rotations covering all of SO(3), rather than just the asymmetric unit. For higher order symmetries (octahedral or icosahedral), deactivating this can speed up job significantly. * Initial Fourier-space radius (voxels in Fourier space): Controls the initial resolution used for symmetry alignment. * Maximum alignment resolution (Å): Controls the maximum resolution information used for symmetry alignment. * Initial shift extent (voxels): Maximum spatial extent of shifts to search over. May be increased to search over a wider range of shifts. * Initial shift grid sampling: Number of initial shift grid points to evaluate (along one axis). * Initial rotation grid sampling: Number of initial rotation grid points to evaluate (along one axis). ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools#particle-processing-parameters) Particle Processing Parameters * Reassign UIDs: Actviate this to generate fresh UIDs for the output particles, hence causing CryoSPARC to consider the downstream particles as unique particles from the input. This is necessary if output particles are combined with input particles in the same job, for refining with non-crystallographic symmetry operators or arbitrary symmetry transforms. * After UIDs are re-generated, low-level result groups from upstream processing cannot be used to override downstream result groups. The output particles are considered to be distinct from the inputs. * Note that all connected result groups from the input particles will be re-outputted as new groups, overriding the passthrough system. [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools#output) Output ---------------------------------------------------------------------------------------------------------------------------------- * Aligned volume * Aligned mask * Aligned particles [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools#common-next-steps) Common Next Steps -------------------------------------------------------------------------------------------------------------------------------------------------------- * Extract from micrographs (may be used to re-extract particles centred on the new origin) * 3D Refinement * Local Refinement * 3D Classification [PreviousJob: Volume Tools](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools) [NextJob: Align 3D maps](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-align-3d-maps) Last updated 1 month ago * [Description](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools#description) * [Input](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools#input) * [Common Parameters](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools#common-parameters) * [Re-centering Parameters](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools#re-centering-parameters) * [Symmetry Alignment Parameters](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools#symmetry-alignment-parameters) * [Particle Processing Parameters](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools#particle-processing-parameters) * [Output](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools#output) * [Common Next Steps](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools#common-next-steps) --- # Job: Volume Tools | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools.md) . [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools#description) **Description** -------------------------------------------------------------------------------------------------------------------------------------- Use this job to one or more of the following operations on an input Volume or Mask: * Upsample/Downsample * Crop * Add soft padding * Flip handedness * Lowpass filter * Invert density (subtract input from ones) * Note that both volumes and masks can be inverted. As of CryoSPARC v4.4, if inversion is activated, it will take place after any thresholding, dilation, or padding (if used). In addition, masks can be generated from thresholded input volumes or masks. Mask generation has the following operations: * Dilation * Applying a soft padding * Clip a mask along the z-axis (for helical refinements) [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools#input) **Input** -------------------------------------------------------------------------------------------------------------------------- * A map volume _or_ mask [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools#common-parameters) **Common Parameters** -------------------------------------------------------------------------------------------------------------------------------------------------- **Note**: Volume operations are done in the following order: 1. Resampling, lowpass filtering, and cropping 2. Thresholding, dilation, and padding (if enabled) 3. Inversion (if enabled) * `Type of input volume`: Choose whether to use the `volume` or `mask` input slot. **The volume operations are only applied to the input at the selected slot** * `Type of output volume`: The type of the output once the Volume Tools operations are complete. Can be the same or different from the type of the input volume * `Lowpass filter (A)`: Lowpass filter the input volume to this resolution in Angstroms, prior to additional processing. * `Lowpass Filter Type`: Select either a rectangular filter ("rect") or a [Butterworth filter](https://en.wikipedia.org/wiki/Butterworth_filter) . Rectangular filters do not include any information beyond the selected frequency, but may introduce ringing artifacts in the filtered map. Butterworth filters include a small amount of information beyond the selected frequency but have reduced ringing. This tradeoff is controlled by the Butterworth filter's order. * In versions of CryoSPARC before v5.0, this parameter defaulted to "rect" * In versions of CryoSPARC starting with v5.0, this parameter defaults to "butterworth" * `Lowpass Filter Order`: The order of the Butterworth filter. Lower order filters have less ringing but have a slower falloff, meaning some information just below the selected cutoff frequency is attenuated and more information beyond the cutoff is included. A higher order filter has a stricter cutoff, but introduces more ringing. A rectangular filter could be thought of as a Butterworth filter with an order of infinity. * This parameter has no effect when `Lowpass Filter Type` is "rect". * In versions of CryoSPARC before v5.0, the default value was 10. * Starting in CryoSPARC v5.0, the default value is 8. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F0DDj0yTKoWtg1063k7lo%252Fbutterworth-explainer.png%3Falt%3Dmedia%26token%3D8c0fced7-aadc-4a70-8a76-8f67a34aa1eb&width=768&dpr=3&quality=100&sign=82ec81a6&sv=2) ### [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools#for-mask-generation) For Mask Generation * `Threshold`: This is the threshold value that is used to binarize the input density map, to convert it into a mask. See callout above about the order of operations applied to the volume. * Note: To generate a mask from a volume that has been lowpass filtered, run an initial Volume Tools job to lowpass filter the volume, then view the density in the volume viewer (or download and view in UCSF Chimera) to choose a threshold value, and run a second job with the threshold set. * `Dilation radius (pix)`: The radius of the spherical mask used to dilate the mask, in units of pixels of the final resampled volume. Leave as 0 to skip dilation. * `Soft padding width (pix)`: The width of the cosine-padded region at the edge of the mask, in units of pixels of the final resampled volume. Setting to 0 will skip soft padding. **Note: the soft padding should never be skipped if the mask is to be used for any downstream jobs in CryoSPARC, since softly-padded masks are required by all jobs that use masks!** [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools#output) **Output** ---------------------------------------------------------------------------------------------------------------------------- * The edited input map volume or mask [](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools#common-next-steps) **Common Next Steps** -------------------------------------------------------------------------------------------------------------------------------------------------- * [Refinement](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement) * [Particle Subtraction](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta) followed by [Local Refinement](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement) for a partial volume * Align 3D Maps for aligning with other imported or generated volumes [PreviousJob: Symmetry Expansion](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-symmetry-expansion) [NextJob: Volume Alignment Tools](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools) Last updated 6 months ago * [Description](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools#description) * [Input](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools#input) * [Common Parameters](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools#common-parameters) * [For Mask Generation](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools#for-mask-generation) * [Output](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools#output) * [Common Next Steps](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools#common-next-steps) --- # Installing 3DFlex Dependencies (v4.1–v4.3) | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement/installing-3dflex-dependencies.md) . All 3D Flex requirements are installed with CryoSPARC v4.4+. Skip this section unless you are running v4.1–v4.3. In CryoSPARC v4.1–v4.3, 3DFlex jobs will not work without first installing the dependencies required to run the jobs. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement/installing-3dflex-dependencies#prerequisites) Prerequisites -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Please ensure your system meets the following requirements: * CryoSPARC v4.1 or newer * Internet connection for downloading additional packages * Nvidia driver version is 460.32.03 or newer on all GPU machines. Run `nvidia-smi` to verify * No CUDA directories are in your `PATH` or `LD_LIBRARY_PATH` environment variables before running this command. To display the variables set in your environment, run `export`. Also ensure that the command `which nvcc` does **not** return a path to `nvcc`. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement/installing-3dflex-dependencies#installation-of-dependencies) Installation of Dependencies -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Log onto each machine where `cryosparc_worker` is installed, and run the command `cryosparcw install-3dflex` inside the `cryosparc_worker` folder. For example: Copy cd cryosparc_worker ./bin/cryosparcw install-3dflex Aside from the Nvidia driver, dependencies for 3DFlex jobs are downloaded during the `install-3dflex` commands. There is no need to supply any external dependencies. The `install-3dflex` command does the following: * Download and install CUDA Toolkit * Download and install PyTorch with CUDA Toolkit * Reinstall PyCUDA with CUDA Toolkit * Verify PyTorch can use CUDA (Requires an NVIDIA GPU) If you run this command on a machine without GPUs, you may see the message `PyTorch not installed correctly, or NVIDIA GPU not detected.` You may safely ignore this if there are no other error messages and the remaining verification tests pass. To further verify that your CryoSPARC instance is ready to run 3DFlex jobs, use the [Installation Tests](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/guide-installation-testing-with-cryosparcm-test#testing-pytorch) to check that PyTorch is working on every worker node connected to your CryoSPARC instance. For example: To uninstall the 3DFlex dependencies and return the worker to its original state, run `cryosparcw forcedeps`. For example: [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement/installing-3dflex-dependencies#update-of-dependencies) Update of Dependencies -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- After updating CryoSPARC to a new full or point release, 3DFlex dependencies can be updated by this sequence of two `cryosparcw` commands: [PreviousTutorial: 3D Flexible Refinement](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement) [NextTutorial: 3D Flex Mesh Preparation](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation) Last updated 1 month ago * [Prerequisites](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement/installing-3dflex-dependencies#prerequisites) * [Installation of Dependencies](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement/installing-3dflex-dependencies#installation-of-dependencies) * [Update of Dependencies](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement/installing-3dflex-dependencies#update-of-dependencies) Copy cryosparcm test workers P1 --test gpu --test-pytorch Copy cd cryosparc_worker ./bin/cryosparcw forcedeps Copy cd cryosparc_worker ./bin/cryosparcw forcedeps ./bin/cryosparcw install-3dflex --- # Tutorial: Dynamic Masking in Refinements (v5.0+) | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-dynamic-masking-in-refinements-v5.0.md) . This guide page covers automatic masking in Homogeneous, Non-Uniform, and Heterogeneous refinement jobs created in CryoSPARC versions beginning with v5.0. Older versions of CryoSPARC use an automatic masking algorithm not described here. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-dynamic-masking-in-refinements-v5.0#masking-during-refinement) Masking During Refinement -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 3D Refinements require two masks: * a _refinement mask_, used during each refinement iteration to remove extraneous density from the current 3D volume which would otherwise degrade particle alignment, and * a _resolution mask_, used to separate target molecule density from solvent during FSC calculation so that the resulting resolution estimate reflects the quality of the target density rather than the background CryoSPARC automatically generates resolution masks during refinement. It also automatically generates a refinement mask if one was not provided by the user. It is important that both masks be appropriately tight, i.e. not too tight, masking too close to target molecule density. It is also important that the masks be appropriately soft, i.e. transitioning smoothly from completely erasing density (mask has value 0.0) to completely retaining density (mask has value 1.0). Masks that are too tight or too hard can lead to refinements where spurious features develop in the density during refinement and/or FSC resolution is overestimated. Users should always inspect both the refinement mask and the resolution mask from a refinement job to confirm that they are not too tight (cutting into the density) or too hard (creating a sharp edge). The [high resolution phase randomization diagnostic](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#high-resolution-phase-randomization) can also be used to detect when a mask is too tight or hard. In CryoSPARC versions before v5.0, automatic masks were generated in refinement jobs based on user-adjustable looseness and softness parameters. These parameters were held fixed during refinement iterations. In some cases, especially for cryo-EM datasets reaching only low or intermediate resolutions (e.g. 4Å or coarser), the default parameters could produce masks that were too tight or too hard. **CryoSPARC v5.0 introduces a new mask generation method** (described below) that is robust across a wide range of resolutions and produces appropriate refinement results and resolution estimates without users having to tune masking parameters. The most significant improvement introduced by the new method is to scale the mask's tightness and softness based on the 3D map's current resolution estimate, rather than using a fixed value over iterations. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-dynamic-masking-in-refinements-v5.0#dynamic-masking-in-cryosparc-v5.0) Dynamic masking in CryoSPARC v5.0+ ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In this section we walk step-by-step through the process by which CryoSPARC produces masks. This process applies to both refinement and resolution masks. Here we create a refinement mask. Consider this 3D map at the end of a particular iteration of a refinement: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FzCpCaIxAbBMdWea0gn0q%252Fbase.png%3Falt%3Dmedia%26token%3Dfed2c85b-7279-4cde-8eea-35cb0c47d0a0&width=768&dpr=3&quality=100&sign=1ab05264&sv=2) First, we binarize the map. The threshold used is the maximum value in the map times the `Dynamic mask threshold` parameter. This has a default value of 0.2 for the refinement mask, but can be changed by the user. The resolution mask uses a fixed value of 0.5. This produces a map with either a zero or one in every voxel. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FAz3r8fhLXN0YPqPBFOmC%252Fthresholded.png%3Falt%3Dmedia%26token%3D82bc912c-f4ef-402f-85ce-6c88fa830f2b&width=768&dpr=3&quality=100&sign=859d17b1&sv=2) Next, the map is dilated by a number of voxels equal to the current resolution times the `Dynamic mask near multiplier`. This has a default value of 2.0 for the refinement mask, but can be changed by the user. The resolution mask uses a fixed value of 2.0. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FKP9I5Ob78ARNi0Y4jC9h%252Fdilated.png%3Falt%3Dmedia%26token%3D92549044-5199-403d-902a-a542719e172a&width=768&dpr=3&quality=100&sign=cf237d06&sv=2) The dilation step creates a mask that is appropriately loose, but it is still has a completely hard edge — voxels are either 1.0 or 0.0. We therefore add a soft edge, which falls gradually from a value of 1.0 at the distance determined by the `Dynamic mask near multiplier`, to a value of 0.0 at the distance determined by the `Dynamic mask far multiplier`. The width of this soft edge is thus equal to the current resolution multiplied by the _difference_ between the far and near multipliers. The value of the `Dynamic mask far multiplier` is 5.0 by default in refinements, and is fixed to 5.0 for resolution masks. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F9d1LPwF1TtvX1LCDpPb0%252Fsoft.png%3Falt%3Dmedia%26token%3D867761f3-db9d-415d-86ba-33f5ebac65ed&width=768&dpr=3&quality=100&sign=ead30d2f&sv=2) Now the mask is dilated and soft, as required for refinement masks. When we plot these masks, it’s best to see the underlying volume so that you can ensure that the relevant portions of the volume are contained within the mask. We could simply display the masked volume: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Ffs0qqVQDt7Te4mJOk2FF%252Fmasked.png%3Falt%3Dmedia%26token%3D76909481-e5c7-420a-b55c-8c1bb73b5ac3&width=768&dpr=3&quality=100&sign=adaed20f&sv=2) But this would hide any density outside the mask which you may wish to see. For example, faint density near the target might indicate a binding partner. If that density was too faint to be included in the initial binarized mask, it would not appear in the masked volume. To avoid this scenario, we instead plot the _unmasked_ map, but with overlays indicating where the mask has a value of 1.0 (solid lines) and where the soft edge ends (that is, where the mask has a value of 0.0; dashed line). ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FCB7EKUtCY6eBA7jSkU5a%252Ffinal_base.png%3Falt%3Dmedia%26token%3Dfca3c3d7-5321-414a-b308-881ef988e8d4&width=768&dpr=3&quality=100&sign=59b9cea&sv=2) The region outside the dashed line is slightly shaded to focus the eye on the region inside the mask, but it is still visible to aid downstream analyses of regions outside the automatic mask. Note that both the dilation and padding width depend on resolution. When a map is poor, the mask is therefore very wide and soft. This reduces the possibility of the mask introducing spurious information into the map. As GSFSC resolution improves during refinement, the mask tightens automatically. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FUAnwgs3y5fCM84m1BkxX%252Flp_10.png%3Falt%3Dmedia%26token%3D2d8e45b6-8cd7-40a7-bfd8-78563be2ad5d&width=768&dpr=3&quality=100&sign=3e31efa8&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fy1KhUUWvSLbSKtsQRCXD%252Flp_6.png%3Falt%3Dmedia%26token%3Da48d8c7c-7d5c-4946-a214-e4f663d321a2&width=768&dpr=3&quality=100&sign=f7ea79b9&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F2QghvKvtZgkAZF0MMf00%252Flp_4.png%3Falt%3Dmedia%26token%3Dcfdd2fc6-7c24-455a-bee8-d0781d1e05dc&width=768&dpr=3&quality=100&sign=61f2e5cb&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FfSXT9HQGta1mcuDXLX6Q%252Flp_2.5.png%3Falt%3Dmedia%26token%3Dac92f432-ce9e-4d46-8df8-b864c907e3d4&width=768&dpr=3&quality=100&sign=6811346&sv=2) Throughout the examples above, we have been plotting a single mask. CryoSPARC refinements plot both the refinement and resolution masks in the same image, and indicate which is which using the same name as the mask output. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FDIkA7Hz0MyQ9z9gFO9KG%252Fmask-names.png%3Falt%3Dmedia%26token%3Dbea62f74-7e48-4e79-a21a-ffe8d1ecdfa5&width=768&dpr=3&quality=100&sign=714593f1&sv=2) [PreviousTutorial: Mask Creation](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera) [NextCase Study: Yeast U4/U6.U5 tri-snRNP](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp) Last updated 4 months ago * [Masking During Refinement](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-dynamic-masking-in-refinements-v5.0#masking-during-refinement) * [Dynamic masking in CryoSPARC v5.0+](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-dynamic-masking-in-refinements-v5.0#dynamic-masking-in-cryosparc-v5.0) --- # Tutorial: Ewald Sphere Correction | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ewald-sphere-correction.md) . Standard cryo-EM 3D reconstruction algorithms are based on modelling particle images as _tomographic projections_ of the underlying 3D structure. This assumption is fundamental to the derivation of reconstruction algorithms, has enabled structures to be solved with resolutions well below 2Å, and is an excellent approximation to the underlying physical process governing the formation of the image. However, due to the specifics of electron wave diffraction, the projection approximation breaks down at very high resolutions. The details behind this follow the geometric construction known as the _Ewald sphere._ The nature of the problem was described well by DeRosier in the context of electron microscopy \[1\], and since then, various algorithms have been proposed for correcting for Ewald sphere curvature in cryo-EM. Relatively recently, structures have been solved by cryo-EM to such high resolutions that the curvature of the Ewald sphere limits their resolutions \[2\]. As such, work surrounding the correction of Ewald sphere curvature has recently proved useful in the recovery of structures from experimental data \[3, 4\]. In v3.3+, CryoSPARC supports the correction of Ewald sphere curvature during refinement and CTF refinement. While the standard reconstruction of a 3D density is based upon maximizing the likelihood of the data given the image poses, the algorithm used for Ewald sphere correction is an improvement on the "simple insertion" method developed by \[3\]. This itself is an approximation to the maximum likelihood method, while accounting for the geometry of the Ewald sphere. CryoSPARC supports Ewald sphere corrected reconstruction in the `Homogeneous Refinement` and `Homogeneous Reconstruction Only` job types. CryoSPARC also supports Ewald sphere corrected (latent variable) inference in both `Global CTF Refinement` and `Local CTF Refinement` jobs, as well as in `Homogeneous Refinement`. In these jobs, Ewald sphere correction must be activated via the corresponding parameter, which is off by default. Typically, the most significant resolution gains come from Ewald sphere corrected reconstruction; we have not observed significant benefits from using Ewald sphere corrected latent inference, although this may change as cryo-EM continues to break resolution limits. On this page we detail the main considerations with Ewald sphere correction, including typical cases where it may be fruitful to use. We also present an [example workflow](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ewald-sphere-correction#tutorial-aav2-dataset-empiar-10202) that would typically be done following the refinement of a high-resolution structure. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ewald-sphere-correction#practical-notes) Practical Notes The curvature of the Ewald sphere may limit the accuracy of the highest frequency Fourier coefficients in a 3D reconstruction. Because of this, Ewald sphere correction should be done only in the final stages of processing a dataset, after an initial high-resolution structure is obtained via refinement. Considering the curvature of the Ewald sphere may help improve resolutions in datasets where any of the following hold: * The dataset refines to a high-resolution, homogeneous structure (e.g. < 3 Å) * The structure is physically large in size (e.g. a large virus capsid) * The microscope voltage is low (e.g. ≤ 200 kV) The dependence on resolution is due to the fact that the effect of Ewald sphere curvature grows approximately quadratically with frequency, meaning only the highest resolution Fourier coefficients are significantly affected. In 2000, DeRosier presented a rule of thumb that gives an approximate frequency (RRR) at which Ewald curvature will result in significant phase errors \[1\]. This is given in terms of the diameter of the protein (ttt) and electron wavelength (λ\\lambdaλ): R\=2∗0.7/(tλ)R = \\sqrt{ 2 \* 0.7 / (t\\lambda) }R\=2∗0.7/(tλ)​ Since the presence of Ewald sphere curvature results in the breakdown of the projection approximation, images are no longer invariant under reflections. Specifically, reflecting a raw 2D image is equivalent to _rotating_ it in 3D space by 180º around the image plane, and this corresponds to _inverting_ the sign of the Ewald sphere's curvature. Thus, the presence of any additional reflections in the raw data, which may arise during data collection or image processing, introduces an indeterminacy of the _sign_ of the Ewald sphere curvature. In all jobs that consider the curvature of the Ewald sphere, an additional parameter controls the sign of the curvature, which can either be `positive` or `negative`. The implications of this are that in practice, reconstructions of particles using both positive and negative curvatures must be done. In cases where Ewald sphere curvature is significant for your dataset, you should expect to see one curvature sign increase the resolution (FSC) of the structure, and the other to decrease the resolution of the structure. In cases where Ewald sphere curvature is insignificant, both reconstructions with differing curvatures should report the same resolution. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ewald-sphere-correction#tutorial-aav2-dataset-empiar-10202) Tutorial: AAV2 Dataset (EMPIAR-10202) ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- To illustrate a typical workflow, we will use Ewald sphere correction on the [EMPIAR-10202 dataset](https://www.ebi.ac.uk/pdbe/emdb/empiar/entry/10202/) , which is an adeno-associated virus serotype 2 variant (AAV2) \[2\]. This virus capsid is an ideal candidate for examining the effect of Ewald sphere curvature, due to its large size, high resolution, and icosahedral symmetry. Indeed, in the original publication, it was solved to a resolution of 1.86 Å by correcting the curvature of the Ewald sphere using the simple insertion algorithm. Here, we show that correcting for the Ewald sphere curvature, together with correcting high-order aberrations (HOA), can result in further resolution increases. From EMPIAR-10202, we will download image set 6 "Final particle stack of AAV2-L336C using frames 5 to 19", which consists of 30,515 particles. We can use the `Import Particles` job together with the `.star` file that accompanies the image set, to import the particles with their CTF values and alignments. Once the particles have finished importing, we will connect them to a `Homogeneous Reconstruction Only` job, which will reconstruct a 3D density from the aligned particle stack. This job will allow us to isolate the effects of Ewald sphere correction during reconstruction, while keeping the particles' poses constant. First, we will reconstruct the density with all parameters as default, except using icosahedral symmetry (and without Ewald sphere correction). This initial reconstruction reached an unmasked resolution of 2.39 Å, and a masked resolution of 1.94 Å. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FSyBT4gXETfNhM4W9DNUs%252F33_esc_.png%3Falt%3Dmedia%26token%3D0200cff4-f59b-4f23-ae25-06b920e289b4&width=768&dpr=3&quality=100&sign=73476c17&sv=2) Tile image for our initial reconstruction Taking a capsid diameter of ~250 Å and an electron wavelength of 0.0197 Å (from a microscope voltage of 300 kV), DeRosier's rule of thumb gives 1/R≈1.9A˚1/R \\approx 1.9 Å1/R≈1.9A˚. This indicates that beyond ~1.9 Å, curvature of the Ewald sphere will result in significant phase errors. Since our reconstruction of this dataset is approaching ~1.9 Å, it will likely be beneficial to continue on with Ewald sphere correction. Next, we will run two more reconstructions, one with positive curvature, and one with negative curvature. We can do this by launching two more reconstruction jobs, activating the "Do EWS Correction" parameter, and changing the curvature parameter on the jobs to be either positive or negative. Note that the curvature sign depends on various factors during data collection and image processing, and it is frequently unknown what the true curvature sign is. Thus, it is always necessary to perform this step to deduce the true curvature. If Ewald sphere curvature is in fact contributing to the resolution limit, we would expect one reconstruction to produce a worse resolution than the standard reconstruction, and the other to produce a better resolution. We will also connect the generated mask from the initial reconstruction to all subsequent reconstructions, so that all reported FSCs are calculated using the same mask. After the reconstructions are complete, we found that the reconstruction with positive curvature reached a masked resolution of 2.07 Å, and negative curvature reached a resolution of 1.89 Å. Since the 0.143 FSC threshold value and the FSC curves both appear more favourable when reconstructing with negative curvature, this indicates that the curvature sign in this dataset is negative. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FEvCFJykvx27gIE7Rd7e2%252F33_esc_2.png%3Falt%3Dmedia%26token%3Db5927bdf-bae0-4c9a-8874-8b27f963b7f9&width=768&dpr=3&quality=100&sign=e81fb16b&sv=2) Tile images for reconstructions with negative (left) and positive (right) Ewald curvature Finally, in order to push the resolution as much as possible, we can repeat the above reconstructions after fitting beam tilt and other high order CTF aberrations. CryoSPARC's implementation of Ewald sphere correction will also simultaneously correct for high order CTF aberrations and anisotropic magnification, if they have been previously estimated for a given particle stack. To do this, we will launch a `Global CTF Refinement` job and connect the particles to it, as well as the reference map generated by the negative-curvature reconstruction. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FByjGrjhgYRMfLuB1nejs%252F33_esc_3.png%3Falt%3Dmedia%26token%3D78b95394-a395-47d4-8406-e7f49ed32e0d&width=768&dpr=3&quality=100&sign=317cc818&sv=2) Odd (left) and even (right) phase delay plots for the global CTF refinement The left hand plot shows that there is quite significant beam tilt, which suggests we may be able to further improve resolution by accounting for it. To complete our experiment, we will do two final reconstructions, one with the CTF-refined particles and without Ewald curvature, and one with negative Ewald curvature. Doing both reconstructions will allow us to see the individual effects of accounting for aberrations and Ewald curvature. Below is a tree view for our full set of experiments on this dataset. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FQlvKnYE7jnqkplTYx7oq%252F33_esc_4.png%3Falt%3Dmedia%26token%3D87930cf5-1fe7-4787-b89d-8047be70efc6&width=768&dpr=3&quality=100&sign=cdceba89&sv=2) Tree view for our experiments with EMPIAR-10202 Using the five reconstruction jobs and the generated FSC files, we have summarized in the FSC comparison plot shown below. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FueORN5Er1ojE8ZHgPDyr%252F33_esc_5.png%3Falt%3Dmedia%26token%3D0fa819f2-714c-4809-858c-e64bda72839f&width=768&dpr=3&quality=100&sign=8f74e54f&sv=2) Effect of the reconstruction method on reported FSC; note that "HOA" refers to high-order aberrations We can see that accounting for high-order aberrations alone (red curve), or Ewald sphere curvature alone (orange curve) produces similar resolutions between 1.87 - 1.89 Å, whereas accounting for both effects bumps the resolution to 1.74 Å. Below are images taken from two of the maps, after sharpening to a B-factor of -50 and zeropadding in fourier space to a box size of 1600. One can note the slight increase in definition of the aromatic ring. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FKV6g06PoSW4aq4Lo7sm7%252F33_esc_6.png%3Falt%3Dmedia%26token%3D8b0e47e8-6779-4c61-8809-362792213415&width=768&dpr=3&quality=100&sign=d6f337ca&sv=2) Left: Reconstruction with only HOA corrected. Right: Reconstruction with HOA and Ewald curvature One may also be interested in re-doing a `Homogeneous Refinement` that corrects Ewald sphere curvature, now that the sign of the curvature is known. This can be done by building the job and activating the Ewald sphere correction parameter, along with inputting the true curvature sign. On this dataset, we did not find significant resolution improvements after a second refinement. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ewald-sphere-correction#references) References \[1\] D. DeRosier, "Correction of high-resolution data for curvature of the Ewald sphere", Ultramicroscopy, vol. 81, no. 2, pp. 83-98, 2000. Available: 10.1016/s0304-3991(99)00120-5 \[2\] Y. Tan et al., "Sub-2 Å Ewald curvature corrected structure of an AAV2 capsid variant", Nature Communications, vol. 9, no. 1, 2018. Available: 10.1038/s41467-018-06076-6 \[3\] M. Wolf, D. DeRosier and N. Grigorieff, "Ewald sphere correction for single-particle electron microscopy", Ultramicroscopy, vol. 106, no. 4-5, pp. 376-382, 2006. Available: 10.1016/j.ultramic.2005.11.001 \[4\] C. Russo and R. Henderson, "Ewald sphere correction using a single side-band image processing algorithm", Ultramicroscopy, vol. 187, pp. 26-33, 2018. Available: 10.1016/j.ultramic.2017.11.001 [PreviousTutorial: CTF Refinement](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement) [NextTutorial: Symmetry Relaxation](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation) Last updated 1 month ago * [Practical Notes](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ewald-sphere-correction#practical-notes) * [Tutorial: AAV2 Dataset (EMPIAR-10202)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ewald-sphere-correction#tutorial-aav2-dataset-empiar-10202) * [References](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ewald-sphere-correction#references) --- # Tutorial: Negative Stain Data | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/negative-stain-data.md) . [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/negative-stain-data#marking-data-as-negative-stain-data) Marking data as negative stain data --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Negative stain data must be treated differently from cryo-EM data in several important ways, summarized below. CryoSPARC will make these adjustments automatically when the data has been marked as negative stain data, either during import or later on using Exposure Tools. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/negative-stain-data#jobs-and-parameters-affected-by-negative-stain-data) Jobs and parameters affected by negative stain data Job Type Difference Import Movies "Negative Stain" toggle will set `mscope_params/neg_stain` field to `1` (True). Import Micrographs "Negative Stain" toggle will set `mscope_params/neg_stain` field to `1` (True). Import Particles "Data Sign" drop-down list will set `blob/sign` field to `+1` (cryo-EM data) if `dark-on-light` is selected or `-1` (negative stain data) if `light-on-dark` is selected. The default is `light-on-dark`, as cryo-EM data is usually inverted during processing. Exposure Tools Allows user to set `mscope_params/neg_stain` and `mscope_params/phase_plate` manually. Manual Picker Extracted particles are indicated with the correct sign (+1 for dark-on-light particles and -1 for light-on-dark particles). Template Picker Templates used to pick particles are flipped (dark-on-light to light-on-dark). Blob Picker Blob templates used to pick particles are flipped (dark-on-light to light-on-dark). Local Motion Correction Extracted particles are indicated with the correct sign (+1 for dark-on-light particles and -1 for light-on-dark particles). Extract From Micrographs Extracted particles are indicated with the correct sign (+1 for dark-on-light particles and -1 for light-on-dark particles). Topaz Train Preprocessed micrographs are inverted (light-on-dark to dark-on-light) before being used for model training Topaz Extract Input micrographs are inverted (light-on-dark to dark-on-light) before being processed by Topaz Extract **NB.** Certain jobs plot particles while taking into account their particle sign (+1 for dark-on-light (cryo-EM data) and -1 for light-on-dark (negative stain data)). You might notice that your negative stain data looks like cryo-EM data in the plot, but please note that the actual particles on disk are still negative stain data. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/negative-stain-data#indicating-negative-stain-data-during-import) Indicating negative stain data during import When importing negative stain movies or micrographs, use the Negative Stain Data toggle in the `Import Movies` or `Import Micrographs` jobs to indicate that you are processing negative stain data (light-on-dark). Subsequent jobs that use the imported negative stain data, will adjust their relevant parameters accordingly (for more details, see below: Jobs affected by negative stain data). ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252Fsync%252Fb067e0f43b3d6fd8a095a1cfe075d91dc8b36a9d.png%3Fgeneration%3D1589377630813486%26alt%3Dmedia&width=768&dpr=3&quality=100&sign=779bafad&sv=2) If importing particles, specify the type of data being imported in the `Import Particles` job. Subsequent jobs that use the imported negative stain data, will adjust their relevant parameters accordingly (for more details, see below: Jobs affected by negative stain data). ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252Fsync%252Fae98f880c0cba9583f6272c6123925935d8b303f.png%3Fgeneration%3D1589377632056012%26alt%3Dmedia&width=768&dpr=3&quality=100&sign=760ce400&sv=2) ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/negative-stain-data#marking-an-existing-set-of-exposures-as-negative-stain-data) Marking an existing set of exposures as negative stain data Use the Exposure Tools job to manually set the Negative Stain Data values for an exposure dataset: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MM2IsyKBxXbKdpc68-H%252F-MM2JPLmbvcWa9jdijfx%252F3neg-stain-phase-plate-3.png%3Falt%3Dmedia%26token%3D1f1c8c52-c698-453d-991a-a3d9f38da4a3&width=768&dpr=3&quality=100&sign=b08774d5&sv=2) [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/negative-stain-data#turn-off-ctf-correction) Turn off CTF correction --------------------------------------------------------------------------------------------------------------------------------------------------- For negative stain data, CTF correction may decrease the quality of 2D classification and reconstruction results. To bypass CTF estimation, the data can be imported in the Import Movies or Import Micrographs job with the Output Constant CTF toggle on: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fi1s9f56vaHRMBepIaon2%252Fv4-0-0-output-constant-ctf-toggle.png%3Falt%3Dmedia%26token%3D82cbd7ac-f0ec-47c7-baa5-beddc1a80b87&width=768&dpr=3&quality=100&sign=8722f1d3&sv=2) Alternatively, the 2D Classification job can be run without CTF correction, by setting the Do CTF correction toggle: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FjJaQxpuTSzll6fMGreXk%252Fv4-0-0-do-ctf-correction-2dclass-toggle.png%3Falt%3Dmedia%26token%3D8b5b1ccb-f44f-47d9-bf41-89a099bc4d8b&width=768&dpr=3&quality=100&sign=49557258&sv=2) [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/negative-stain-data#data-type-reference-chart) Data type reference chart ------------------------------------------------------------------------------------------------------------------------------------------------------- Data Type Data Sign Description Cryo-EM +1 dark-on-light Negative Stain \-1 light-on-dark **Note:** Cryo-EM data is typically recorded as +1 (dark-on-light) but is often inverted during processing. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/negative-stain-data#picking-negative-stain-data) Picking negative stain data ----------------------------------------------------------------------------------------------------------------------------------------------------------- In some cases, CryoSPARC's blob picker may not pick the highest-contrast particles in a negative stain dataset. If this occurs, picking may improve if the particle diameter is set to a lower value. The high contrast at the edges of a negative stain particle creates a significant amount of contrast. This makes the power score slider of the Inspect Particle Picks job particularly effective at selecting these particles. [PreviousTutorial: Common CryoSPARC Plots](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots) [NextTutorial: Phase Plate Data](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/phase-plate-data) Last updated 2 years ago * [Marking data as negative stain data](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/negative-stain-data#marking-data-as-negative-stain-data) * [Jobs and parameters affected by negative stain data](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/negative-stain-data#jobs-and-parameters-affected-by-negative-stain-data) * [Indicating negative stain data during import](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/negative-stain-data#indicating-negative-stain-data-during-import) * [Marking an existing set of exposures as negative stain data](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/negative-stain-data#marking-an-existing-set-of-exposures-as-negative-stain-data) * [Turn off CTF correction](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/negative-stain-data#turn-off-ctf-correction) * [Data type reference chart](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/negative-stain-data#data-type-reference-chart) * [Picking negative stain data](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/negative-stain-data#picking-negative-stain-data) --- # Tutorial: Orientation Diagnostics | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-orientation-diagnostics.md) . Orientation Diagnostics, a new job in CryoSPARC v4.4+, can help diagnose the presence of preferred orientation. In this tutorial, we’ll use the untilted and tilted **Influenza Hemagglutinin Trimer (HA Timer)** data (Tan et al. (2017); deposited in EMPIAR entries 10096 and 10097 respectively) to help elucidate the types of diagnostics one should expect to see with and without preferred orientation. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-orientation-diagnostics#ha-trimer-case-study) HA Trimer Case Study ---------------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-orientation-diagnostics#untilted-dataset-empiar-10096) Untilted dataset ([EMPIAR 10096](https://www.ebi.ac.uk/empiar/EMPIAR-10096/) ) To begin, we’ll process the deposited 447 movies via a typical CryoSPARC processing workflow of Patch Motion, Patch CTF, Blob picking / curation, 2D classification, 2D class selection, and Ab-Initio reconstruction to arrive at a set of ~82 000 curated particles and an initial map. Refining this initial volume with homogeneous refinement with C3 symmetry yields a structure with a reported GSFSC resolution of ~3.1 Å (see figure below). By inspecting the volume visually, however, we see that the map lacks the features one would expect at this resolution. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FuXY03H3nEyAG6MM9L80y%252Fv4-4-0-orient-diag-tut-untilted-hatrimer.png%3Falt%3Dmedia%26token%3D249a5c5d-9d83-4e4f-a648-3e626859cfb9&width=768&dpr=3&quality=100&sign=a4a3fba3&sv=2) HA Trimer (untilted data) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FfvU14SU6YKQUM3O2YK0N%252Fv4-4-0-orient-diag-tut-untilted-hatrimer-fsc.png%3Falt%3Dmedia%26token%3D999e629d-63f7-4fe4-a931-8ddf751d4a24&width=768&dpr=3&quality=100&sign=ccfab794&sv=2) What’s more, the vertical streaks in the map are clear indications that the poor quality may be due to the presence of preferred orientation within the particles. To assess further, we’ll take the refined volume, mask, and particles and connect it to an Orientation Diagnostics job, and set the symmetry parameter to C3. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-orientation-diagnostics#orientation-diagnostics) Orientation Diagnostics ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FdUVUD7r17aiMH8EdgrHa%252Fv4-4-0-orient-diag-tut-untilted-hatrimer-od.png%3Falt%3Dmedia%26token%3D8b21cd1e-537e-40eb-a134-28f34e1b9797&width=768&dpr=3&quality=100&sign=588ac810&sv=2) Once complete, orientation diagnostics will generate a number of visualizations. One natural starting point to investigate preferred orientation is to look at the conical FSC summary plot. This plot generalizes the GSFSC curves shown above to incorporate the notion of directional resolution. Note that this plot shows very similar information to the figure generated via the legacy 3DFSC job (Tan et al., 2017). ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FlEtRDKWYmfzYMIpsGHtv%252Fv4-4-0-orient-diag-tut-untilted-hatrimer-cfsc-summary.png%3Falt%3Dmedia%26token%3Dd9d31b96-1764-4ead-b486-4f7f1f813999&width=768&dpr=3&quality=100&sign=ca034d3e&sv=2) A conical FSC (cFSC) is a Fourier Shell Correlation of two half maps with a conical mask of a specified half angle and axis in Fourier space. To assess directional signal content, the Orientation Diagnostics job computes a set of cFSC curves with conical axes sampled along a uniform spherical distribution. The figure below illustrates this process for four cFSC cones. In blue, the cFSC summary plot visualizes the mean, minimum, maximum, and standard deviation value of the correlations at each spatial frequency. In green, we also overlay a histogram of 0.143 crossings, which correspond to the spread of resolution values over direction. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FYlwHiQ6vNHzQcjcknb6F%252Fv4-4-0-orient-diag-tut-cfsc-explainer.png%3Falt%3Dmedia%26token%3D0101601d-0008-4ca9-a750-a8b1fb391fdd&width=768&dpr=3&quality=100&sign=6b3c9858&sv=2) By computing multiple conical FSC curves, we can assess how correlation varies as a function of direction. Although they can both be represented via azimuth and elevation angles, the conical axis of a cFSC should be carefully distinguished from viewing direction. Concretely, low cFSC values along a particular conical axis do not imply that more views are necessary from that direction. This is due to the fact that a particle contributes Fourier information to a Fourier slice whose components are orthogonal to the viewing direction — this fact is elucidated further in the mathematical definition of the SCF within the Orientation Diagnostics job page, and in the SCF publications (Baldwin and Lyumkis, 2020, 2021). When cFSC curves do not vary significantly as function of conical axis, the structure has a directional resolution that is constant across the viewing sphere. Here, this is clearly not the case. In the worst case, we see a cFSC resolution worse than 11 Å! To quantify orientation bias, Orientation Diagnostics provides two metrics: the conical FSC Area Ratio, or cFAR, and the Sampling Compensation Factor, or SCF\*. Both metrics range from 0 to 1, where 0 indicates a strong orientation bias, and 1 indicates no bias. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F98oeK3UkVs5tcdyZ9EIM%252Fv4-4-0-orient-diag-tut-untilted-hatrimer-cFAR.png%3Falt%3Dmedia%26token%3D3dd8e970-289a-4e9a-a3b8-aa3a92e23c65&width=768&dpr=3&quality=100&sign=c03c698a&sv=2) cFAR is the ratio of the minimum to maximum area under the cFSC curves summarized above. To account for the fact that higher frequencies correspond to a larger shell of Fourier components, the area is weighted at each spatial frequency by the surface area of the corresponding shell in Fourier space. In other words, we summarize each cFSC with a weighted area-under-curve number (’wAuC’), that quantifies the total ‘mass’ of the cFSC cone in units of correlation. wAuC as a function of conical axis on the viewing sphere is shown in the plot above. The ratio of the minimum to the maximum value in this plot defines the cFAR. For a mathematical definition of cFAR and wAuC, please see the [Orientation Diagnostics job page](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-orientation-diagnostics). In this dataset, cFAR is 0.02, which indicates severe orientation bias. In general, we find that a cFAR value of 0.5 serves as a reasonable threshold for the presence, or lack thereof, of preferred orientation. To complement cFAR, we also report the Sampling Compensation Factor (Baldwin & Lyumkis, 2020, 2021). The SCF assesses the degree to which certain Fourier voxels are under sampled by the set of particle alignments. It is important to note that SCF does not consider the signal content within each particle; junk particles and true particles contribute equally to the final metric. An SCF value of 0.81 corresponds to the case where we have one ‘band’ of viewing directions. As a result, the original authors of SCF (Baldwin and Lyumkis, 2021) argue that values above 0.81 generally indicate good sampling (though not necessarily isotropic signal content). ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FKA5XbyGpiHwxlJ3N6Vd0%252Fv4-4-0-orient-diag-tut-untilted-hatrimer-SCF.png%3Falt%3Dmedia%26token%3Dcd5cc3bf-bcf6-4e3a-9895-1d2698b2afb5&width=768&dpr=3&quality=100&sign=3d00915f&sv=2) ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-orientation-diagnostics#tilted-dataset-empiar-10097) Tilted dataset ([EMPIAR 10097](https://www.ebi.ac.uk/empiar/EMPIAR-10097/) ) To see the effect of stage tilting on this data, we turn to the data deposited in EMPIAR entry 10097. As before, we process the raw movies using a typical CryoSPARC workflow to arrive at an initial volume and approximately 58,000 curated particles. We then apply homogeneous refinement with C3 symmetry and arrive at the map depicted below. Note that the global GSFSC resolution is actually worse than the untilted data, but visually the map quality is significantly improved. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fe6Un4ztAOZg5lhloOjQt%252Fv4-4-0-orient-diag-tut-tilted-hatrimer.png%3Falt%3Dmedia%26token%3D613ba448-4534-40bc-82b5-0c5622288fcc&width=768&dpr=3&quality=100&sign=7ee52413&sv=2) HA Trimer (tilted data) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FgorcFm05qMhGrNzCg3k1%252Fv4-4-0-orient-diag-tut-tilted-hatrimer-fsc.png%3Falt%3Dmedia%26token%3Dda8a9154-bd53-4b7b-a71e-29303555395a&width=768&dpr=3&quality=100&sign=c18486a5&sv=2) Applying the Orientation Diagnostics job (with C3 symmetry set) to the outputs of this refinement, we see much higher cFAR and SCF scores, much smaller cFSC curve variation and directional resolutions that only differ by approximately 0.5 Å. We see further that in many cases, the cFAR score is more sensitive to directional anisotropy than SCF\* as it accounts for both insufficient sampling of the Fourier domain, and for anisotropic distributions of signal (e.g., junk optimized into certain regions). ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FlwendpE3me4QaIEGOAKN%252Fv4-4-0-orient-diag-tut-tilted-hatrimer-cfsc-summary.png%3Falt%3Dmedia%26token%3Dcb970210-2a0e-46cc-910f-1d2bd475a95f&width=768&dpr=3&quality=100&sign=728f4004&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FV0IIMbPc6d1RoqUSHsVU%252Fv4-4-0-orient-diag-tut-tilted-hatrimer-cfar.png%3Falt%3Dmedia%26token%3Df68dda37-e6ef-43ac-ba40-4ec4b385fd40&width=768&dpr=3&quality=100&sign=4420bc28&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FemtDd6aIQoyUhBX1vK1b%252Fv4-4-0-orient-diag-tut-tilted-hatrimer-scf.png%3Falt%3Dmedia%26token%3D3951e3a6-0627-4e20-b673-4919ffa03585&width=768&dpr=3&quality=100&sign=ffe2fe31&sv=2) [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-orientation-diagnostics#references) References -------------------------------------------------------------------------------------------------------------------------------------- Tan et al. (2017), Addressing preferred specimen orientation in single-particle cryo-EM through tilting. _Nat Methods_ 14(8), 793-796. Baldwin, P. R., & Lyumkis, D. (2020). Non-uniformity of projection distributions attenuates resolution in Cryo-EM. _Progress in biophysics and molecular biology_ 150, 160-183. Baldwin, P. R., & Lyumkis, D. (2021). Tools for visualizing and analyzing Fourier space sampling in Cryo-EM. _Progress in biophysics and molecular biology_ 160, 53-65. [PreviousTutorial: Symmetry Relaxation](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation) [NextTutorial: BILD files](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-bild-files) Last updated 1 month ago * [HA Trimer Case Study](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-orientation-diagnostics#ha-trimer-case-study) * [Untilted dataset (EMPIAR 10096)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-orientation-diagnostics#untilted-dataset-empiar-10096) * [Tilted dataset (EMPIAR 10097)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-orientation-diagnostics#tilted-dataset-empiar-10097) * [References](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-orientation-diagnostics#references) --- # Tutorial: 3D Variability Analysis (Part Two) | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-two.md) . [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-two#id-3d-variability-analysis-tutorial-part-two) 3D Variability Analysis Tutorial: Part Two ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In [Part One](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one) of this tutorial, we covered the basics of 3D Variability Analysis (3DVA) and showed how it can be used to solve for variability components (i.e., eigenvectors of the 3D covariance of images). This second part of the tutorial covers some of the more advanced ways that 3D variability results can be used to interpret heterogeneity in a dataset. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-two#improvements-to-the-3dva-algorithm) Improvements to the 3DVA algorithm ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In CryoSPARC v2.12+, many improvements have been made to the 3D Variability algorithm. It can now be used to solve more variability components simultaneously (12+), resolving more detailed motion, and works for smaller particles. For example the following videos show the first two variability components solved at a high resolution (4Å) for the T20S proteasome. These components show two types of orthogonal variability in the molecule, corresponding to extension of the barrel and twisting of the top and bottom subunits. Only two components are shown, though in this case, 6 components were solved in total. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNuyJn0D5vkDi4BdlVS%252F-MNuzdhABId5qKJMhgHD%252F3dva_p2_1_gif.gif%3Falt%3Dmedia%26token%3D7595e6e8-3af8-4f35-b4c3-a636c1a250d0&width=768&dpr=3&quality=100&sign=33c78749&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNuyJn0D5vkDi4BdlVS%252F-MNuzih0if-WTNLHHTf_%252F3dva_p2_2_gif.gif%3Falt%3Dmedia%26token%3Da61d2ac6-633b-41d0-aec0-edd71fa4a1de&width=768&dpr=3&quality=100&sign=ec5d6a0d&sv=2) 3D Variability Analysis has also been used in recent cryo-EM projects and has already appeared in a number of publications: * Structural basis for the docking of mTORC1 on the lysosomal surface (Rogala et al. Science 2019) (Note: This video is from the publication supplementary materials, [found here](https://science.sciencemag.org/content/suppl/2019/10/09/science.aay0166.DC1) ) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNuyJn0D5vkDi4BdlVS%252F-MNuzorn3ph3kZykYjw-%252F3dva_p2_3_gif.gif%3Falt%3Dmedia%26token%3D8eeeff87-01ba-46e0-aa8e-b39c7686febd&width=768&dpr=3&quality=100&sign=2e59bfc8&sv=2) * Cryo–electron microscopy structures of human oligosaccharyltransferase complexes OST-A and OST-B (Ramirez, Kowal, et al. Science 2019) * Cryo-EM reveals an asymmetry in a novel single-ring viral chaperonin (Stanishneva-Konovalova et al. JSB 2019) * Cryo-electron Microscopy Structure of the _Acinetobacter baumannii_ 70S Ribosome and Implications for New Antibiotic Development (Morgan et al. mBio 2020) Almost all proteins of biological interest have some amount of conformational heterogeneity, especially continuous heterogeneity. For almost all users, it will be a good idea to use 3D Variability Analysis to directly discover the heterogeneity that cannot be easily solved using traditional 3D classification approaches. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-two#clustering) Clustering ----------------------------------------------------------------------------------------------------------------------------------------------- 3D variability excels at resolving continuous heterogeneity and motion within a molecule, but it is equally powerful for separating discrete conformational changes. One of the key theoretical concepts that helps us understand 3D classification of heterogeneity in cryo-EM, is that classification (i.e. clustering) problems are naturally difficult, being non-convex and NP-complete in the worst case. This is the reason that, for example, running more than one 3D classification job (e.g., _ab-initio_ and heterogeneous refinement using initial models) will yield different clustering results. Depending on initialization and stochastic steps taken by the clustering algorithms, some clusters can be found twice (i.e., duplicate classes) and some clusters can be missed entirely. There is no simple way to know whether additional undiscovered clusters remain in the dataset. Clustering algorithms must explore the space of conformations to find clusters that explain the particle data well. This exploration process becomes more difficult with: * more noise in the data * more data points that need to be compared to one another * high dimensionality data points Raw cryo-EM particle data unfortunately exhibits all three of the above characteristics. Particle images are very noisy, there are hundreds of thousands or millions of them and each one is represented using thousands of dimensions (pixels). 3D Variability Analysis steps around this issue, making clustering much simpler. It relies on a simple theoretical result: a linear manifold formed from eigenvectors of the data covariance (i.e., 3D Variability components) will, under some mild conditions, span the subspace in which clusters lie, **without** needing to know the cluster identities or the number of clusters ahead of time. For cryo-EM heterogeneity, this means that when there are discrete clusters present in a dataset, the first several 3D Variability components will directly show us the difference between clusters, separating them as clearly as is possible given the noise in images. In this way, the problem of finding clusters becomes much simpler - they will show up as visual "clusters" when we visualize the particles in their reaction coordinates. Then, we can simply cluster particles by their low-dimensional reaction coordinates, rather than having to look at all the pixels in every image and explore 3D conformations simultaneously. The following example, using 3DVA on a dataset of 50S ribosome particles (data from David, Tan, et. al _Cell_ 2016, EMPIAR10076) shows how 3D Variability components can separate clusters corresponding to discrete conformational changes. 131,899 particles were processed, first through ab-initio reconstruction (single class) then homogeneous refinement to obtain a consensus refinement of the ribosome. This structure showed regions of low and high resolution where there was variability. 3DVA was then run, solving 4 components. The components themselves are shown below: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNuyJn0D5vkDi4BdlVS%252F-MNuzyAiv_wu6YLg0QOh%252F3dva_p2_4.png%3Falt%3Dmedia%26token%3Df20177d3-36db-4e7f-b4e5-a6df2e7a8fe3&width=768&dpr=3&quality=100&sign=f342551e&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNuyJn0D5vkDi4BdlVS%252F-MNv--hqsFkgyf3L2S3I%252F3dva_p2_5.png%3Falt%3Dmedia%26token%3Dcfc0a775-5ac4-4786-932b-657c96395303&width=768&dpr=3&quality=100&sign=2d477b&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNuyJn0D5vkDi4BdlVS%252F-MNv-2n6gQpr9ypCPjRT%252F3dva_p2_6.png%3Falt%3Dmedia%26token%3Dfd089341-4cb5-4253-a292-849db0b408ce&width=768&dpr=3&quality=100&sign=9e353be2&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNuyJn0D5vkDi4BdlVS%252F-MNv-6Gw5wdaTW0xLkbL%252F3dva_p2_7.png%3Falt%3Dmedia%26token%3Df4a6b584-da72-4a9c-9fe7-1357ec85bd70&width=768&dpr=3&quality=100&sign=658abf6f&sv=2) Using the new in-line 3D interactive visualization capability in CryoSPARC v2.13+, we are able to inspect three reaction coordinate dimensions of the particles, and see clear separation of clusters. This was done by creating a 3D Variability Display job in `cluster` mode and selecting 6 as the number of clusters. Clustering in the reaction coordinate space is done using a Gaussian Mixture Model, and each particle is then assigned to a cluster. These are each displayed with a different colour. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNuyJn0D5vkDi4BdlVS%252F-MNv-N7NIExG3vl1jGDF%252F3dva_p2_8_gif.gif%3Falt%3Dmedia%26token%3D05c3c9cc-1d2a-478d-ae37-7e180a393e5c&width=768&dpr=3&quality=100&sign=ef8ead96&sv=2) This is a great example where clusters are clearly separated in the reaction coordinate space. The new visualization features in CryoSPARC v2.13+ make it easy to see the topology of the particles in reaction coordinate space. Each cluster represents a different conformation of the ribosome in this case. The `3D Variability Display` job in CryoSPARC, when set to cluster mode, also uses the clusters that are found here to reconstruct the different conformations individually: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNuyJn0D5vkDi4BdlVS%252F-MNv-QMg0v8bHuqPFcrl%252F3dva_p2_9.png%3Falt%3Dmedia%26token%3D4e3cd7d1-df63-4c89-a0e1-42c868ff0b0b&width=768&dpr=3&quality=100&sign=9bd1c5de&sv=2) In this case, several large conformational changes of the ribosome have been automatically separated. We can then apply 3D Variability again to particles within each cluster, in order to find discrete sub-classes and continuous flexibility that may be present within each overall cluster. We can also take the particles and reconstruction from each cluster and apply standard homogeneous refinement to improve particle alignments and resolution. The `3D Variability Display` job outputs reconstructions and particle sets from each cluster to make these workflows possible. Currently, the user must specify the number of clusters, though in principle it will be possible to automatically detect the number of clusters in a future version. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-two#visualizing-clusters) Visualizing Clusters ------------------------------------------------------------------------------------------------------------------------------------------------------------------- In order to see the interactive visual 3D scatter plot as shown above, use the following steps. 1. Run a 3D Variability Analysis job 2. Connect the 3DVA job outputs to a 3D Variability Display job 3. Set the 3D Variability Display job to `cluster` mode and set a number of clusters using the `Number of Frames/Clusters` parameter. 4. Run the 3D Variability Display job 5. In the streamlog of the 3D Variability Display job, you will see plots showing static 3D images of the scatter plot. Hovering your mouse over these plots will prompt you to click to start interactive mode. Interactive mode allows for zooming, panning, rotating, and turning on/off each cluster's points. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNuyJn0D5vkDi4BdlVS%252F-MNv-V8ZW74OZFOtZDBl%252F3dva_p2_10.png%3Falt%3Dmedia%26token%3D7cfbd19c-3b2a-4758-97b8-96dccb4aed83&width=768&dpr=3&quality=100&sign=b2a2069d&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNuyJn0D5vkDi4BdlVS%252F-MNv-YcuMJHs8wh_3mO0%252F3dva_p2_11.png%3Falt%3Dmedia%26token%3Dee2aec88-8548-4504-ac7f-7b64d86f58e1&width=768&dpr=3&quality=100&sign=a27e65f7&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNuyJn0D5vkDi4BdlVS%252F-MNv-awaN_F5p1CNS0ZU%252F3dva_p2_12.gif%3Falt%3Dmedia%26token%3D96a6d4c4-b178-4e1d-8b10-a48e2c7bc457&width=768&dpr=3&quality=100&sign=75da38e3&sv=2) [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-two#intermediate-reconstructions) Intermediate Reconstructions ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Continuous heterogeneity is often well modelled by 3D Variability components, as the first examples in this tutorial demonstrate. In the default `simple` mode, CryoSPARC's `3D Variability Display` job will create a volume series using the consensus refinement and the 3D Variability component itself to show how the 3D structure changes. This results in a series that contains a smoothly varying, **linear** change in 3D density values from one end of the series to the other. This linear approximation is often very good for showing small detailed motions and conformational changes within the protein structure, but it can break down for larger motions. In those cases, it can be helpful to construct a volume series by reconstructing separate volumes directly from subsets of the particle images, sorted and chosen by their reaction coordinate value. This technique, of sub-sampling a dataset in the latent space (i.e., reaction coordinates) for visualization, has been used by other methods that involve representing particles in a reaction coordinate space. These reconstructions are called `intermediate` reconstructions in CryoSPARC, and can be created using the `3D Variability Display` job in `intermediate` mode. In this mode, particles are sorted along each variability dimension, and then split into (overlapping) subsets, weighted by their position along the variability dimension. This creates a "rolling window" of particles selected for creating each intermediate reconstruction in a volume series. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNuyJn0D5vkDi4BdlVS%252F-MNv-iUyw4_As_ygEUpr%252F3dva_p2_13.png%3Falt%3Dmedia%26token%3D0299a034-12f8-4618-8ae3-a7b0970600d5&width=768&dpr=3&quality=100&sign=18bab345&sv=2) Each "triangle" window here shows the weighting for one selection of particles along the sequence that defines the series. Each selection is reconstructed independently using the weighted particles, and these together form a series that can help visualize non-linear changes in a dataset. The 3D Variability Display Job has a parameter that allows setting the "width" of the rolling window used. This can increase or decrease the number of particles per sub-selection of particles. Increasing the width leads to better signal-to-noise levels in reconstructions, while decreasing it leads to less blurring within each intermediate reconstruction due to the particle's flexing motion. A width of zero is also allowed, which will use "tophat" windows instead. For further downstream analysis, this job (as of CryoSPARC v3.3) can output particle subsets that correspond to each intermediate reconstruction along one variability component. To enable this functionality, set the parameter `Intermediates: output particle subsets` to true and select a 3D variability component via the parameter `Intermediates: output particle component`. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-two#symmetry) Symmetry ------------------------------------------------------------------------------------------------------------------------------------------- There is ongoing discussion about how symmetry should be handled when solving 3D Variability components, but in general, the Symmetry Expansion job should be used before using 3D Variability, for symmetric particles. To explain further: when there is “symmetry” in a particle, this only applies to the consensus structure - any 3D variability mode can break the symmetry. However, the variability mode (assuming the underlying symmetry is a true symmetry) should naturally occur in all symmetric versions of itself. In this sense there are two kinds of variability modes: modes that are changes/motion within just a single subunit, and modes where there are changes/motion across the entire particle in a coordinated fashion. The first kind are modes where each particle image contains information about _multiple_ positions along the mode (since each subunit is in potentially a different position along that mode). The second kind are modes where each particle image contains information about only one position along the mode (since the entire particle is in only one position) but due to the symmetry, the image could be used as information for all symmetric copies of that same mode. So for the first kind of mode, symmetry expansion is best. In this case it’s also fine to create a mask around a single subunit, but this is not necessary (since using the subunit mask will make it impossible to find motions across the entire particle). For the second kind of mode, using symmetry expansion will mean that there are many copies of the same mode that can be found (i.e. imaging a symmetric molecule bending along its entire length, in one direction. There are equivalent copies of that mode where the molecule bends in the symmetric versions of that one direction). Symmetry expansion makes sure that every particle counts for each one of these copies, rather than just the single one with which the particle is arbitrarily aligned in the input poses. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-two#high-pass-resolution) High-pass Resolution ------------------------------------------------------------------------------------------------------------------------------------------------------------------- Since cryoSPARC v2.12, 3D Variability jobs have a parameter that allows setting a High-pass Resolution in Angstroms. This option adds a “high-pass prior” to the 3D Var components limiting the amount of power they can have at low frequencies. This essentially ignores variability that is larger than a certain scale. Many smaller/membrane proteins have a large amount of “structured noise” present in the images at low resolutions. This could be from pancaked particles floating around at the air water interfaces, empty micelles above/below the particles, etc. These phenomena causing most of the variability modes to be full of large blobs appearing and disappearing, rather than actually probing the motion or flexing of the molecule. In these cases, turning on the high-pass resolution can improve results. A typical value for the high-pass resolution is 20Å. [PreviousTutorial: 3D Variability Analysis (Part One)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one) [NextTutorial: 3D Flexible Refinement](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement) Last updated 3 years ago * [3D Variability Analysis Tutorial: Part Two](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-two#id-3d-variability-analysis-tutorial-part-two) * [Improvements to the 3DVA algorithm](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-two#improvements-to-the-3dva-algorithm) * [Clustering](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-two#clustering) * [Visualizing Clusters](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-two#visualizing-clusters) * [Intermediate Reconstructions](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-two#intermediate-reconstructions) * [Symmetry](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-two#symmetry) * [High-pass Resolution](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-two#high-pass-resolution) --- # Performance Metrics | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/live/performance-metrics.md) . [](https://guide.cryosparc.com/live/performance-metrics#cryosparc-live-performance-metrics) CryoSPARC Live Performance Metrics ----------------------------------------------------------------------------------------------------------------------------------- CryoSPARC Live is built and tuned for high performance pre-processing and streaming reconstruction of single particle data, and can use multiple concurrent GPUs for to maximize throughput. **CryoSPARC Live preprocessing includes four steps: (1) motion correction, (2) CTF estimation, (3) particle picking and (4) extraction.** CryoSPARC Live can sustain a throughput of 450 or more exposures per hour, per GPU, for K3 data. On a 4-GPU machine, that can scale to 1800+ exposures per hour! For K2 or Falcon data, performance can be even higher, upwards of 650 exposures per hour per GPU. Depending on your hardware configuration (particularly raw data storage disk access speed), each preprocessing worker can sustain a throughput of at least one movie every 30 seconds, which is equal to ~2,500 movies per day per GPU. In our internal tests, we have seen performance on well-tuned systems (like the testing hardware below) reaching up to 8,000 movies per GPU per day. See the [Hardware Configurations used for Benchmarking section](https://guide.cryosparc.com/live/performance-metrics#hardware-configurations-used-for-benchmarking) to see details on what hardware was used to run the benchmarks. All 3D renderings were captured in ChimeraX from maps created by cryoSPARC Live. [](https://guide.cryosparc.com/live/performance-metrics#hardware-configurations-used-for-benchmarking) Hardware Configurations Used for Benchmarking --------------------------------------------------------------------------------------------------------------------------------------------------------- All pre-processing timings were measured with **Configuration 1**, unless otherwise noted. Component Configuration 1 Configuration 2 Configuration 3 CPU AMD Ryzen Threadripper 2950x AMD Ryzen Threadripper 3960x AMD Ryzen Threadripper 3960x Memory Bandwidth 128 GB/s 144GB/s 144GB/s RAM 128GB DDR4 2666MHz 256GB DDR4 2933MHz 256GB DDR4 2933MHz GPU 0 Quadro GV100 Quadro RTX 8000 GeForce RTX 3090 GPU 1 Quadro GV100 Quadro RTX 8000 GeForce RTX 3090 GPU 2 Quadro RTX 5000 GTX 1080Ti \- GPU 3 GTX 1080Ti Tesla K40c \- Fast CPU memory bandwidth is a major contributing factor to high performance in cryoSPARC Live. Please make note of this metric when selecting your system's CPU and RAM. [](https://guide.cryosparc.com/live/performance-metrics#k2-mrc-ha-trimer) K2 MRC (HA Trimer) ------------------------------------------------------------------------------------------------- Benchmark results for 668 MRC-format uncompressed movies from a GATAN K2 4k × 4k detector. The first 40 of 100 frames were used. Exposures from this dataset were captured with the stage tilted 40º. Particles were selected with the Template Picker strategy. Streaming 2D Classification, Ab-initio Reconstruction and Streaming Refinement yielded 3.0Å resolution map from ~230,000 particles. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNyVAIDiXU3mvmPVIh0%252F-MNyVWvrHZxYh4vGxUtd%252FPM_1_live-perf-hatrimer.png%3Falt%3Dmedia%26token%3D24270348-bb30-4893-817f-29754de95cdb&width=768&dpr=3&quality=100&sign=e34e1d15&sv=2) [![Logo](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2Fwww.ebi.ac.uk%2Fem_static%2Fempiar%2Ffavicon%2Fapple-touch-icon.png&width=20&dpr=3&quality=100&sign=50e2f750&sv=2)EMPIAR-10097 40 Degree Tilted Single-Particle CryoEM of Highly Preferred Orientated Influenza Hemagglutinin Trimerwww.ebi.ac.uk](https://www.ebi.ac.uk/pdbe/emdb/empiar/entry/10097/) Dataset on EMPIAR ### [](https://guide.cryosparc.com/live/performance-metrics#data-properties) Data Properties Property Value Detector Gatan K2 Number of Movies 668 File Format MRC Frame Size 3838 x 3710 Frames per Movie 100 (40 used) Pixel Size 1.13Å Particle Extraction Box Size 144 × 144 ### [](https://guide.cryosparc.com/live/performance-metrics#benchmarks) Benchmarks Metric Value Movies Pre-processed Per Hour Per GPU 430 Movies Pre-processed Per Day Per GPU 10290 Average Pre-processing Time Per Movie 8.4s [](https://guide.cryosparc.com/live/performance-metrics#k2-tiff-nav1.7) K2 TIFF (Nav1.7) --------------------------------------------------------------------------------------------- Benchmark results for ~24,000 TIFF-LZW compressed movies from a GATAN K2 4k × 4k detector. Particles were selected with the Blob Picker strategy. Streaming 2D Classification, Ab-initio Reconstruction and Streaming Refinement yielded 3.3Å resolution map from ~300,000 particles. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNyVAIDiXU3mvmPVIh0%252F-MNyVZm9MBUjGXptF4Dy%252FPM_2_live-perf-nav17.png%3Falt%3Dmedia%26token%3D05b06aea-1246-416c-9297-ac811a67098d&width=768&dpr=3&quality=100&sign=1b89e9fe&sv=2) [![Logo](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2Fwww.ebi.ac.uk%2Fem_static%2Fempiar%2Ffavicon%2Fapple-touch-icon.png&width=20&dpr=3&quality=100&sign=50e2f750&sv=2)EMPIAR-10261 CryoEM micrographs of ProTx2-bound Nav1.7 VSD2-NavAb chimeric channelwww.ebi.ac.uk](https://www.ebi.ac.uk/pdbe/emdb/empiar/entry/10261/) Dataset on EMPIAR ### [](https://guide.cryosparc.com/live/performance-metrics#data-properties-1) Data Properties Property Value Detector Gatan K2 Summit Number of Movies 25084 File Format TIF-LZW Frame Size 3838 × 3710 Frames per Movie 40 Pixel Size 0.85Å Particle Extraction Box Size 512 × 512 Particle Extraction Bin Size 256 x 256 Applied Symmetry C2 ### [](https://guide.cryosparc.com/live/performance-metrics#benchmarks-1) Benchmarks Metric Value Movies Pre-processed Per Hour Per GPU 650 Movies Pre-processed Per Day Per GPU 15600 Average Pre-processing Time Per Movie 5.5s [](https://guide.cryosparc.com/live/performance-metrics#k2-super-res-tiff-t20s) K2 super-res TIFF (T20S) ------------------------------------------------------------------------------------------------------------- Benchmark results for ~200 TIFF-LZW compressed movies from a GATAN K2 detector with super-resolution capture. Particles were selected with the Template Picker strategy. Streaming 2D Classification, Ab-initio Reconstruction and Streaming Refinement yielded a 2.5Å resolution map from ~130,000 particles. The target T20S Proteasome has D7 symmetry. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNyVAIDiXU3mvmPVIh0%252F-MNyVbBGUUzv05C9BNWi%252FPM_3_live-perf-t20s.png%3Falt%3Dmedia%26token%3Ded5e8f96-3351-40bc-b6be-b4f6a9387ee0&width=768&dpr=3&quality=100&sign=bb3b51da&sv=2) [![Logo](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2Fwww.ebi.ac.uk%2Fem_static%2Fempiar%2Ffavicon%2Fapple-touch-icon.png&width=20&dpr=3&quality=100&sign=50e2f750&sv=2)EMPIAR-10025 T20S Proteasome at 2.8 Å Resolutionwww.ebi.ac.uk](https://www.ebi.ac.uk/pdbe/emdb/empiar/entry/10025/) Dataset on EMPIAR ### [](https://guide.cryosparc.com/live/performance-metrics#data-properties-2) Data Properties Property Value Detector Gatan K2 (super-res) Number of Movies 196 File Format TIFF-LZW Frame Size 7420 × 7676 Frames per Movie 38 Pixel Size 0.6575Å Particle Extraction Box Size 448 × 448 Applied Symmetry D7 ### [](https://guide.cryosparc.com/live/performance-metrics#benchmarks-2) Benchmarks Metric Value Movies Pre-processed Per Hour Per GPU 254 Movies Pre-processed Per Day Per GPU 6096 Average Pre-processing Time Per Movie 14.2s [](https://guide.cryosparc.com/live/performance-metrics#k3-tiff-tt-oad2) K3 TIFF (TT-OAD2) ----------------------------------------------------------------------------------------------- Benchmark results for ~200 TIFF-LZW compressed movies from a GATAN K3 detector. The first 40 of 64 frames were used. Particles were selected with the Blob Picker strategy. Post-processing (2D Classification, Refinement, etc.) was not run on this dataset. Exposures in this dataset were captured with beam-induced tilt. [![Logo](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2Fwww.ebi.ac.uk%2Fem_static%2Fempiar%2Ffavicon%2Fapple-touch-icon.png&width=20&dpr=3&quality=100&sign=50e2f750&sv=2)EMPIAR-10346 Cryo-EM of GLP-1 receptor bound to TT-OAD2 non-peptidic agonistwww.ebi.ac.uk](https://www.ebi.ac.uk/pdbe/emdb/empiar/entry/10346/) Dataset on EMPIAR ### [](https://guide.cryosparc.com/live/performance-metrics#data-properties-3) Data Properties Property Value Detector Gatan K3 Number of Movies 3159 File Format TIFF-LZW Frame Size 5760 × 4092 Frames per Movie 62 (40 used) Pixel Size 0.826Å Particle Extraction Box Size 144 × 144 ### [](https://guide.cryosparc.com/live/performance-metrics#benchmarks-3) Benchmarks Metric Value Movies Pre-processed Per Hour Per GPU 420 Movies Pre-processed Per Day 10050 Average Pre-processing Time Per Movie 8.6s [](https://guide.cryosparc.com/live/performance-metrics#k3-super-res-tiff-tt-oad2) K3 super-res TIFF (TT-OAD2) ------------------------------------------------------------------------------------------------------------------- Benchmark results using super-resolution variants from super-resolution variant of [previous dataset](https://guide.cryosparc.com/live/performance-metrics#k3-tiff-tt-oad-2) . Only the first 40 frames of each exposure were used. ### [](https://guide.cryosparc.com/live/performance-metrics#data-properties-4) Data Properties Property Value Detector Gatan K3 (super-res) Number of Movies 4259 File Format TIFF-LZW Frame Size 11520 × 8184 Frames per Movie 67 (40 used) Pixel Size 0.413Å Particle Extraction Box Size 288 × 288 ### [](https://guide.cryosparc.com/live/performance-metrics#benchmarks-4) Benchmarks Metric Value Movies Pre-processed Per Hour Per GPU 192 Movies Pre-processed Per Day Per GPU 4608 Average Pre-processing Time Per Movie 18.7s [](https://guide.cryosparc.com/live/performance-metrics#falcon-iii-tiff-pac1) Falcon III TIFF (PAC1) --------------------------------------------------------------------------------------------------------- Benchmark results for ~3000 TIFF-LZW compressed movies from a Falcon III detector. Particles were selected with the Blob Picker strategy. Post-processing (2D Classification, Refinement, etc.) was not run on this dataset. [![Logo](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2Fwww.ebi.ac.uk%2Fem_static%2Fempiar%2Ffavicon%2Fapple-touch-icon.png&width=20&dpr=3&quality=100&sign=50e2f750&sv=2)EMPIAR-10351 Cryo-EM structure of the human PAC1 receptor coupled to an engineered heterotrimeric G proteinwww.ebi.ac.uk](https://www.ebi.ac.uk/pdbe/emdb/empiar/entry/10351/) ### [](https://guide.cryosparc.com/live/performance-metrics#data-properties-5) Data Properties Property Value Detector TFS Falcon III Number of Movies 2895 File Format TIFF-LZW Frame Size 4096 × 4096 Frames per Movie 64 Pixel Size 0.835Å Particle Extraction Box Size 420 × 420 ### [](https://guide.cryosparc.com/live/performance-metrics#benchmarks-5) Benchmarks Metric Value Movies Pre-processed Per Hour Per GPU 493 Movies Pre-processed Per Day Per GPU 11832 Average Pre-processing Time Per Movie 7.3s [](https://guide.cryosparc.com/live/performance-metrics#falcon-iv-eer-apoferritin) Falcon IV EER (Apoferritin) ------------------------------------------------------------------------------------------------------------------- Benchmark results for ~3000 Electron Event Representation (EER) movies from a Falcon IV detector. The particle is highly symmetric. The target apoferritin is highly symmetric. Enough information is present in the dataset to approach atomic resolution. Particles were selected with the ring template picker strategy. Streaming 2D Classification, Ab-initio Reconstruction and Streaming Refinement yielded a 1.9Å resolution map from ~700,000 particles without any additional processing. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNyVAIDiXU3mvmPVIh0%252F-MNyVf1R_km5Xpru4zSC%252FPM_4_live-perf-apoferritin.png%3Falt%3Dmedia%26token%3Dca35fdfc-fb6e-48b3-908b-cc1d84bd704b&width=768&dpr=3&quality=100&sign=6ae95611&sv=2) [![Logo](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2Fwww.ebi.ac.uk%2Fem_static%2Fempiar%2Ffavicon%2Fapple-touch-icon.png&width=20&dpr=3&quality=100&sign=50e2f750&sv=2)EMPIAR-10424 Atomic resolution structure of apoferritinwww.ebi.ac.uk](https://www.ebi.ac.uk/pdbe/emdb/empiar/entry/10424/) Dataset on EMPIAR ### [](https://guide.cryosparc.com/live/performance-metrics#data-properties-6) Data Properties Property Value Detector TFS Falcon IV Number of Movies 3370 File Format EER Frame Size 8192 × 8192 Frames per Movie 434 (40 used) Pixel Size 0.457Å Particle Extraction Box Size 512 × 512 ### [](https://guide.cryosparc.com/live/performance-metrics#benchmarks-6) Benchmarks Metric Value Movies Pre-processed Per Hour Per GPU 303 Movies Pre-processed Per Day Per GPU 7272 Average Pre-processing Time Per Movie 11.9s [](https://guide.cryosparc.com/live/performance-metrics#k2-tiff-with-preprocessing--2d-3d-streaming-cb1-gi) K2 TIFF with Preprocessing + 2D/3D Streaming (CB1 Gi) ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- Benchmark results for ~3000 TIFF-LZW compressed movies from a GATAN K2 detector. The target complex is a small, flexible membrane protein. Particles were selected with the Template Picker strategy. Streaming 2D Classification, Ab-initio Reconstruction and Streaming Refinement yielded a 3.9Å resolution map from ~700,000 particles. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNyVAIDiXU3mvmPVIh0%252F-MNyVhuQRXkF0IkFeyi3%252FPM_5_live-perf-cb1.png%3Falt%3Dmedia%26token%3Db5c458eb-b424-412c-ae11-96bf1ced481c&width=768&dpr=3&quality=100&sign=b3dba5ca&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNyVAIDiXU3mvmPVIh0%252F-MNyVk4K9Pqv5YopMyYQ%252FPM_6_live-perf-cb1.png%3Falt%3Dmedia%26token%3D0942ae46-7468-413c-97fe-4e221b1cfae6&width=768&dpr=3&quality=100&sign=fea96fd9&sv=2) [![Logo](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2Fwww.ebi.ac.uk%2Fem_static%2Fempiar%2Ffavicon%2Fapple-touch-icon.png&width=20&dpr=3&quality=100&sign=50e2f750&sv=2)EMPIAR-10288 Cryo electron microscopy of Cannabinoid Receptor 1-G Protein Complexwww.ebi.ac.uk](https://www.ebi.ac.uk/pdbe/emdb/empiar/entry/10288/) Dataset on EMPIAR ### [](https://guide.cryosparc.com/live/performance-metrics#data-properties-7) Data Properties Property Value Detector Gatan K2 Number of Movies 2756 File Format TIFF-LZW Frame Size 3838 × 3710 Frames per Movie 40 Pixel Size 0.86Å Particle Extraction Box Size 360 × 360 Particle Extraction Bin Size 256 × 256 ### [](https://guide.cryosparc.com/live/performance-metrics#benchmarks-7) Benchmarks Pre-processing and streaming results for this dataset measured with [Hardware Configuration 3](https://guide.cryosparc.com/live/performance-metrics#hardware-configurations-used-for-benchmarking) Metric Value Movies Pre-processed Per Hour Per GPU 870 Movies Pre-processed Per Day Per GPU 20880 Average Pre-processing Time Per Movie 4.13s #### [](https://guide.cryosparc.com/live/performance-metrics#post-processing-256-256-box-size-50-2d-classes) Post-Processing: 256 × 256 box size, 50 2D Classes Metric Value Particles Extracted for 2D Classification 79,278 Time to 2D Classify Extracted Particles 4 minutes, 16 seconds Particles Used for Reconstruction 100,000 Time to Reconstruct Initial Volume (Ab-initio) 5 minutes, 11 seconds Particles Selected for Refinement 278,312 Time to Refine Final Volume 22 minutes, 55 seconds [PreviousLive Jobs and Session-Level Functions](https://guide.cryosparc.com/live/live-jobs-and-session-level-functions) [NextManaging a CryoSPARC Live Session from the CLI (≤v4.7)](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli) Last updated 5 years ago * [CryoSPARC Live Performance Metrics](https://guide.cryosparc.com/live/performance-metrics#cryosparc-live-performance-metrics) * [Hardware Configurations Used for Benchmarking](https://guide.cryosparc.com/live/performance-metrics#hardware-configurations-used-for-benchmarking) * [K2 MRC (HA Trimer)](https://guide.cryosparc.com/live/performance-metrics#k2-mrc-ha-trimer) * [Data Properties](https://guide.cryosparc.com/live/performance-metrics#data-properties) * [Benchmarks](https://guide.cryosparc.com/live/performance-metrics#benchmarks) * [K2 TIFF (Nav1.7)](https://guide.cryosparc.com/live/performance-metrics#k2-tiff-nav1.7) * [Data Properties](https://guide.cryosparc.com/live/performance-metrics#data-properties-1) * [Benchmarks](https://guide.cryosparc.com/live/performance-metrics#benchmarks-1) * [K2 super-res TIFF (T20S)](https://guide.cryosparc.com/live/performance-metrics#k2-super-res-tiff-t20s) * [Data Properties](https://guide.cryosparc.com/live/performance-metrics#data-properties-2) * [Benchmarks](https://guide.cryosparc.com/live/performance-metrics#benchmarks-2) * [K3 TIFF (TT-OAD2)](https://guide.cryosparc.com/live/performance-metrics#k3-tiff-tt-oad2) * [Data Properties](https://guide.cryosparc.com/live/performance-metrics#data-properties-3) * [Benchmarks](https://guide.cryosparc.com/live/performance-metrics#benchmarks-3) * [K3 super-res TIFF (TT-OAD2)](https://guide.cryosparc.com/live/performance-metrics#k3-super-res-tiff-tt-oad2) * [Data Properties](https://guide.cryosparc.com/live/performance-metrics#data-properties-4) * [Benchmarks](https://guide.cryosparc.com/live/performance-metrics#benchmarks-4) * [Falcon III TIFF (PAC1)](https://guide.cryosparc.com/live/performance-metrics#falcon-iii-tiff-pac1) * [Data Properties](https://guide.cryosparc.com/live/performance-metrics#data-properties-5) * [Benchmarks](https://guide.cryosparc.com/live/performance-metrics#benchmarks-5) * [Falcon IV EER (Apoferritin)](https://guide.cryosparc.com/live/performance-metrics#falcon-iv-eer-apoferritin) * [Data Properties](https://guide.cryosparc.com/live/performance-metrics#data-properties-6) * [Benchmarks](https://guide.cryosparc.com/live/performance-metrics#benchmarks-6) * [K2 TIFF with Preprocessing + 2D/3D Streaming (CB1 Gi)](https://guide.cryosparc.com/live/performance-metrics#k2-tiff-with-preprocessing--2d-3d-streaming-cb1-gi) * [Data Properties](https://guide.cryosparc.com/live/performance-metrics#data-properties-7) * [Benchmarks](https://guide.cryosparc.com/live/performance-metrics#benchmarks-7) --- # Tutorial: Symmetry Relaxation | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation.md) . [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation#introduction) Introduction -------------------------------------------------------------------------------------------------------------------------------------- Symmetry relaxation is an option available in Homogeneous and Non-Uniform refinement jobs that can help resolve pseudosymmetry and symmetry-mismatched complexes. This tutorial describes the specific case that symmetry relaxation attempts to address, and walks through how it can be applied to a dataset. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation#alignment-and-symmetry-mismatches) Alignment and Symmetry Mismatches All refinement jobs in CryoSPARC require estimating the orientation from which each particle is viewed, relative to the density reference. This “alignment” step enables iterative refinement, and allows for successive reconstruction of an updated density map. This step is fairly costly though, as it requires solving a 5-dimensional search problem (3 pose dimensions and 2 shift dimensions) _independently for every particle_, via matching projections of the rotated and shifted density to particle images. In all refinements with global pose search, CryoSPARC uses a technique known as _Branch and Bound_ (BnB) to accelerate the alignment step. This was described in our 2017 Nature Methods publication, [cryoSPARC: algorithms for rapid unsupervised cryo-EM structure determination](https://www.nature.com/articles/nmeth.4169) (Punjani et al., 2017). Since poses and shifts range over a continuum of possible values, the search space of poses and shifts must be discretized, so that a projection-matching based likelihood calculation can be performed at all poses in the discrete space. Discretization of poses usually does not cause issues for asymmetric structures, for which each particle will have only one correct pose. Discretization isn’t problematic with perfectly symmetric structures either: each particle has a set of multiple correct poses, the number of which is equal to the symmetry order of the imposed symmetry group. Each of these correct poses is related to each other via symmetry transform. In this case, the alignment algorithm can find any of the optimal poses, and the entire set of symmetry-related poses will be used to reconstruct the molecule. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FmkZBQWpiVzdtfDQZiEoO%252Fv4-4-0-symmetry_vs_pseudosymmetry-1.png%3Falt%3Dmedia%26token%3D0afa8994-837e-45fc-b78c-155f4154f915&width=768&dpr=3&quality=100&sign=b71f8922&sv=2) An illustration of the pitfalls of enforcing symmetry on pseudosymmetric molecules. Where this strategy encounters difficulties is with nearly symmetric molecules. There are many terms used to refer different forms of approximate symmetries, including pseudosymmetry, symmetry-mismatches, or symmetry-breaking features. An excellent characterization of different forms of these symmetries is presented in [Huiskonen, 2018](https://portlandpress.com/bioscirep/article/38/2/BSR20170203/57149/Image-processing-for-cryogenic-transmission) . Pseudosymmetric particles may have several plausible poses at low resolution, and the correct pose may only be distinguishable if each of the symmetry-related poses are compared directly. The discretization of the pose search space used by CryoSPARC provides no guarantee that the symmetry-related poses are compared directly, unless symmetry relaxation is activated (as described in the subsequent section). Note that there is no single processing strategy that will best resolve all types of symmetry-mismatched complexes, and optimal reconstruction of symmetry-mismatched complexes is an active area of research that is usually heavily informed by the specific sample being solved. For a thorough catalogue of different processing strategies used for resolving symmetry-mismatched molecules, refer to, [Abrishami et al., 2021](https://www.sciencedirect.com/science/article/pii/S0079610720300390?via%3Dihub) published in Progress in Biophysics and Molecular Biology. With symmetry-mismatched structures, keen attention to map details and quality is required. Simple resolution indicators like FSC are usually not the best indicator of optimal final results, and manual map inspection is strongly recommended. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation#symmetry-relaxation) Symmetry Relaxation In effort to make CryoSPARC more robust to symmetry-mismatched complexes, we have updated the Homogeneous and Non-Uniform Refinement job types to include a tool known as s_ymmetry relaxation_. Symmetry relaxation is a modification to the orientation search procedure, which forces CryoSPARC to be more thorough during orientation search, to avoid placing particles into incorrect poses that are related to the true pose by a symmetry transform. It is recommended to enable symmetry relaxation when dealing with pseudosymmetric or symmetry-breaking molecules. When symmetry relaxation is enabled, regular BnB alignment proceeds as normal for each particle, and the BnB-optimal pose is stored. After the BnB-optimal pose is found, symmetry relaxation imposes an extra step in which the alignment objective function is evaluated at _all poses that are symmetry-related to the current optimal pose_. The symmetry-related pose angles are computed analytically, and the objective function is evaluated at the current FSC resolution of the density. If any of the symmetry-related poses are found to have a better objective value, these new poses will be used to reconstruct the next iteration’s density map. In CryoSPARC v4.4’s Homogeneous and Non-Uniform Refinement jobs, symmetry relaxation is made available via the `Symmetry relaxation method` parameter. There are three options available: * `none`: This disables symmetry relaxation. The input symmetry group will be enforced as usual, and each particle will be used during reconstruction N times, where N is the order of the symmetry group * `maximization`: This option enables symmetry relaxation. Once the BnB-optimal pose is found, the alignment objective will be evaluated over each of the N symmetry-related poses. The single best pose will carry forward to the reconstruction (”backprojection”) step. * `marginalization`: This option enables symmetry relaxation via marginalization. Once the BnB-optimal pose is found, the alignment objective will be evaluated over a small search radius covering each of the N symmetry-related modes. The single optimal mode will be carried forward to the reconstruction step, and each pose within the search radius will be weighted during backprojection by its normalized posterior probability. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F5NSvqi42kbM60b4XSb2K%252Fv4-4-0-symrelax-max-marg.png%3Falt%3Dmedia%26token%3D16ab3489-90e3-4b37-9653-cb8b60f8755b&width=768&dpr=3&quality=100&sign=96f3a3ea&sv=2) A comparison of the three methods of pose search: Standard alignment (”None”), Maximization, and Marginalization Whether maximization or marginalization should be used depends on the size of the protein, size of the mask, and overall signal-to-noise ratio (SNR) of the dataset. For larger proteins, masks, and higher SNRs, maximization may be sufficient. For smaller proteins, masks, and lower SNRs, marginalization may produce better results. This advice is congruent with our general recommendation that marginalization is preferred when working with smaller proteins and lower SNRs. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation#additional-refinement-iterations) Additional refinement iterations ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FuXKp9qGskTWTUIIGOILJ%252Fiterations-and-sym-relaxation_iter-subset.png%3Falt%3Dmedia%26token%3D2f7de539-9cd1-4609-8c23-3c9d52401311&width=768&dpr=3&quality=100&sign=eeccce2a&sv=2) Several iterations of a Non-Uniform Refinement of EMPIAR 10256 are shown. Homogeneous and Non-Uniform Refinements end when the GSFSC resolution stops improving. In the typical case, this is desirable — further refinements will have little to no effect, so additional iterations are wasted. However, when performing symmetry relaxation, the particles may still move between symmetry-related poses (and therefore improve the quality of the map) even when the GSFSC resolution has stopped improving. We thus typically recommend increasing the `Number of extra final passes` when performing symmetry relaxation, perhaps starting with a value between five and ten. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation#symmetry-relaxation-walk-through) Symmetry Relaxation Walk Through ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ In this walk-through, we will use a dataset with pseudo-icosahedral symmetry to illustrate how symmetry relaxation can help resolve symmetry breaking features. The density map we’ll be using is available on the Electron Microscopy Data Bank under entry [#8254](https://www.ebi.ac.uk/emdb/EMD-8254) , “**Phage Qbeta asymmetric reconstruction**”. This structure was solved in [Gorzelnik et al, 2016](https://www.pnas.org/doi/full/10.1073/pnas.1609482113) and subsequently referred to in [Huiskonen, 2018](https://portlandpress.com/bioscirep/article/38/2/BSR20170203/57149/Image-processing-for-cryogenic-transmission) as an example of a dataset containing a symmetry mismatch. Since the raw movies or particles were not released, we will be using the solved density map to generate simulated particles within CryoSPARC v4.4 using the [Simulate Data job](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-gpu) , and then using these simulated particles to reconstruct the pseudo-icosahedral density. While the use of synthetic data is not representative of most workflows used in practice, we’re presenting this walk-through as it allows us to directly compare the faithfulness of each refinement method (with or without symmetry relaxation) both in terms of map quality and in terms of pose discrepancy from ground truth. With real datasets, we do not have access to ground truth orientations and thus cannot provide as robust of an analysis of the estimated latent variables. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation#synthetic-data-generation) Synthetic Data Generation To begin, we’ll download the [EMD-8254](https://www.ebi.ac.uk/emdb/EMD-8254) volume and run an Import 3D Volumes job in CryoSPARC to import the volume into CryoSPARC. After the volume is imported, we need to ensure that the volume is aligned to CryoSPARC’s icosahedral symmetry axes conventions, which can be done via the symmetry alignment feature in the Volume Alignment Tools job. Even though we are working with an asymmetric volume, we intend on using symmetry relaxation under the icosahedral symmetry group, thus we require the volume to be (as best as possible) aligned to the icosahedral symmetry axes. This step can be done via connecting the imported volume to the Volume Alignment Tools job, and setting the following parameters: * Do symmetry alignment: True * Symmetry string: I The output volume is now aligned to CryoSPARC’s icosahedral symmetry axes convention. Next, we’ll generate the synthetic data using the [Simulate Data job](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations) . Connect the aligned volume to Simulate Data, and simulate 20,000 particles at the default signal-to-noise ratio. We will also use [Volume Tools](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools) to generate a mask (via thresholding, dilating, and padding) the aligned volume to be used downstream during refinement. Our workflow thus far is included below. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FwhILHCmJjkJzQ2makPZQ%252Fv4-4-0-symrelax-tree-initial.png%3Falt%3Dmedia%26token%3Dbdff6a9f-31b6-443d-86fd-01c2585948ba&width=768&dpr=3&quality=100&sign=ecbf466c&sv=2) Workflow (tree view) for particle and mask generation. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation#initial-3d-reconstruction) Initial 3D Reconstruction Next, we will proceed with 3D reconstruction as usual: Ab-initio reconstruction followed by refinement. We run Ab-initio reconstruction with the following parameters: * Maximum resolution (Angstroms): 8 * Initial resolution (Angstroms): 20 * Symmetry: I\* \*Icosahedral symmetry is applied at this point to ensure Ab-initio finds an output volume that is aligned to the symmetry axes, and to avoid the problem of “flattened” models with high-symmetry data. In the next step, we’ll create an asymmetric reference from the Ab-initio output. Using Homogeneous Reconstruction Only, we’ll connect the particles from Ab-initio and set the following parameters: * Symmetry: I * Break Symmetry: True This will take the input particle stack and reconstruct a volume where each particle’s pose has been randomly permuted by one of the icosahedral symmetry group’s operations. This step effectively “breaks” the perfect symmetry that may be present in the initial set of particle poses, and can be used on the outputs of a symmetry-enforced ab-initio reconstruction or refinement job. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation#comparing-refinement-methods) Comparing Refinement Methods Now we’ll compare the different methods of refinement to see how symmetry relaxation can help resolve symmetry mismatches. The three methods we’ll compare are: * Standard asymmetric refinement (SAR) * Symmetry relaxation (SR) via maximization * Symmetry relaxation (SR) via marginalization Build three Homogeneous Refinement jobs. In all jobs, set the following parameters: * **Initial lowpass resolution (A): 20** * **Force re-do GS split: False** * **Initialize noise model from images: True** For the standard asymmetric refinement, leave all other parameters as default. For the symmetry relaxation via maximization and marginalization, set the following additional parameters: * Symmetry: I * Symmetry relaxation method: `maximization` or `marginalization` Once the three jobs have completed, we can see that both symmetry relaxation (SR) methods recovered the correct asymmetric volume, whereas the SAR volume still appears fully symmetric. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FGvHlgXGIBaYF9OFXpuep%252Fv4-4-0-symrelax-comparison-sidebyside.png%3Falt%3Dmedia%26token%3Db4f944f9-9d75-45ed-99d7-eaec2c036423&width=768&dpr=3&quality=100&sign=26520d52&sv=2) Standard Asymmetric Refinement (SAR, in gray) compared to symmetry relaxation (SR) via maximization (in yellow) and via marginalization (in blue). Note how the volume reconstructed using symmetry relaxation has resolved the internal RNA helices stretching across the inside of the viral capsid. Since we’re working with synthetic data, we can compare the estimated particle poses to their ground truth values, for each of the different methods. Below is a set of histograms in polar coordinates, displaying the number of particles with a given error between the ground truth pose and the pose found by Branch and Bound. We also investigated whether adding extra iterations to SAR helped; the plot below shows the results with 0, 1, and 2 extra iterations added. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FIiYQxg3giWpB2qpfUkht%252Fv4-4-0-symrelax-poseerror-compare.png%3Falt%3Dmedia%26token%3D113dbd86-7f2d-49bb-8ea4-32686b078962&width=768&dpr=3&quality=100&sign=8c455e67&sv=2) Comparison of the magnitude of the pose error between the ground truth pose and the pose found by Branch and Bound. Each polar-plot histogram displays the number of particles with a given pose error, which starts at 0º at the 12 o’clock position, and increases clockwise. Pose errors are displayed for standard asymmetric refinement (SAR) with 0, 1, and 2 extra iterations (in purple, orange, and yellow, respectively). Both symmetry relaxation methods are displayed in the lower right hand, showing that either symmetry relaxation method better recovers the ground truth for the majority of particles, compared to asymmetric refinement. Histograms concentrated near 0º (12 o’clock position) indicate the majority of particles had their poses correctly recovered. Manually forcing extra refinement iterations helped somewhat to reduce the number of misaligned particles (compare the orange and yellow histograms to the purple one), but both symmetry relaxed methods performed even better. The tree view for this workflow is displayed below. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fg17328fAaRSIMbhTbjan%252Fv4-4-0-symrelax-tree-final.png%3Falt%3Dmedia%26token%3Dc97eb131-e99e-4134-8972-5d8a81cde1ff&width=768&dpr=3&quality=100&sign=db7a307a&sv=2) [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation#references) References ---------------------------------------------------------------------------------------------------------------------------------- * A. Punjani, J. L. Rubinstein, D. J. Fleet, and M. A. Brubaker, “[CryoSPARC: Algorithms for rapid unsupervised cryo-em structure determination](https://www.nature.com/articles/nmeth.4169) ,” _Nature Methods_, vol. 14, no. 3, pp. 290–296, 2017. doi:10.1038/nmeth.4169 * J. T. Huiskonen, “[Image processing for cryogenic transmission electron microscopy of symmetry-mismatched complexes](https://portlandpress.com/bioscirep/article/38/2/BSR20170203/57149/Image-processing-for-cryogenic-transmission) ,” _Bioscience Reports_, vol. 38, no. 2, 2018. doi:10.1042/bsr20170203 * V. Abrishami _et al._, “[Localized reconstruction in scipion expedites the analysis of symmetry mismatches in cryo-EM data](https://www.sciencedirect.com/science/article/pii/S0079610720300390?via%3Dihub) ,” _Progress in Biophysics and Molecular Biology_, vol. 160, pp. 43–52, 2021. doi:10.1016/j.pbiomolbio.2020.05.004 * K. V. Gorzelnik _et al._, “[Asymmetric cryo-EM structure of the canonical allolevivirus qβ reveals a single maturation protein and the genomic ssrna in situ](https://www.pnas.org/doi/full/10.1073/pnas.1609482113) ,” _Proceedings of the National Academy of Sciences_, vol. 113, no. 41, pp. 11519–11524, 2016. doi:10.1073/pnas.1609482113 [PreviousTutorial: Ewald Sphere Correction](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ewald-sphere-correction) [NextTutorial: Orientation Diagnostics](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-orientation-diagnostics) Last updated 1 month ago * [Introduction](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation#introduction) * [Alignment and Symmetry Mismatches](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation#alignment-and-symmetry-mismatches) * [Symmetry Relaxation](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation#symmetry-relaxation) * [Additional refinement iterations](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation#additional-refinement-iterations) * [Symmetry Relaxation Walk Through](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation#symmetry-relaxation-walk-through) * [Synthetic Data Generation](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation#synthetic-data-generation) * [Initial 3D Reconstruction](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation#initial-3d-reconstruction) * [Comparing Refinement Methods](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation#comparing-refinement-methods) * [References](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation#references) --- # Tutorial: 3D Flexible Refinement | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement.md) . 3DFlex (BETA) is available in CryoSPARC v4.1+. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#overview) Overview --------------------------------------------------------------------------------------------------------------------------------- 3D Flexible Refinement (3DFlex) is a motion-based deep generative model for continuous heterogeneity. It can model non-rigid motion and flexibility of a protein molecule across its conformational landscape, and can use the motion model to combine signal from particle images in different conformations to improve refinement resolution in flexible regions. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FS9AddtLOT9R69FVrKsHt%252Fv4-1-0-3dflex-model.png%3Falt%3Dmedia%26token%3D63f6571e-489e-4670-b6a0-806cbd148265&width=768&dpr=3&quality=100&sign=2a757b1d&sv=2) The 3DFlex model represents the flexible 3D structure of a protein as deformations of a single **canonical** 3D density map _V_. Under the model, a single particle image is associated with a low-dimensional **latent coordinate** _z_ that encodes the conformation for the particle in the image. A neural **flow generator network** _f\_θ_ converts the latent coordinate into the flow field _u_ and a convection operator then deforms the canonical density to generate a **convected** map _W_. This map can then be projected along the particle viewing direction determined by the pose _φ_, CTF-corrupted, and compared against the experimental image. Complete details of the architecture and training of 3DFlex can be found in the [bioRxiv preprint here](https://doi.org/10.1101/2021.04.22.440893) . 3DFlex (BETA) is included in **CryoSPARC v4.1**. This tutorial shows how to run the new job types in CryoSPARC used for creating, training, and using a 3DFlex model. It also covers some of the practical aspects of using the algorithm such as parameter tuning and customizing inputs. Much of the content is covered in a tutorial video below. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#example-results) Example Results ----------------------------------------------------------------------------------------------------------------------------------------------- This video shows results of 3DFlex on a dataset of 102,500 particle images of a tri-snRNP spliceosome particle (EMPIAR-10073). 3DFlex is run with a K=5-dimensional latent space, and different regions of the space correspond to different parts of the particle's conformational landscape. This video shows the output of the 3DFlex generative model as latent coordinates are varied along three axes (coordinates 1, 3, and 5). These dimensions encode non-rigid motion of the head region of the protein, where different parts and subunits move and bend relative to each other. 3DFlex applied to 58,433 particle images of a translocating ribosome (EMPIAR-10792). Traversing the latent space shows that 3DFlex has learned coordinated motion of multiple parts (e.g., large and small subunits, elongation factor, etc.) including the overall ratcheting motion of the ribosome. For this result, a segmentation was used to specify a tetrahedral mesh topology allowing adjacent subunits to deform separately (see Mesh Generation below). 3DFlex applied to 113,511 particle images of the SARS-CoV-2 spike protein (EMPIAR-10516). 3DFlex is run with a K=3 dimensional latent space and has learned a combination of motions of the RBD and NTD domains. The up-RBD in particular undergoes a lot of motion which limits its resolution in rigid refinement. In contrast, flexible refinement improves the resolution of the up-RBD. This result also used a segmentation to enable the adjacent RBD and NTD domains to deform separately (see Mesh Generation below). This video shows results of 3DFlex on a dataset of 200,000 particle images of a TRPV1 ion channel (EMPIAR-10059). 3DFlex is run with a K=2-dimensional latent space. The video shows the output of the 3DFlex generative model as latent coordinates are varied along each of the two dimensions. The first dimension reveals inward and outward coordinated bending of opposite flexible subunits in the soluble domain. The second dimension reveals twisting of the subunits around the pore axis. This video shows a comparison between the reconstructed density map from a conventional refinement and flexible refinement using 3DFlex for the TRPV1 ion channel. Map quality and local resolutions are substantially improved in the peripheral helices. Notably, local focused refinement using a mask around the flexible part cannot improve the reconstruction compared to a conventional refinement, because the flexible parts are non-rigid and too small for individual pose alignment. 3DFlex applied to 84,266 particle images of an αV β8 integrin (EMPIAR-10345). 3DFlex using two latent dimensions, learns large bending motions of the flexible arm of the integrin particle, as well as flexibility in the Fabs that are bound. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#installing-3dflex-cryosparc-v4.1-v4.3-only) Installing 3DFlex (CryoSPARC v4.1–v4.3 only) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- All 3D Flex requirements are installed with CryoSPARC v4.4+. Skip this section unless you are running v4.1–v4.3 3DFlex job types are available in **CryoSPARC v4.1+** but in v4.1–v4.3, the new dependencies required for 3DFlex are not installed. To ensure a CryoSPARC worker can run 3DFlex, please see the following instructions: [Installing 3DFlex Dependencies (v4.1–v4.3)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement/installing-3dflex-dependencies) [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#job-types) Job Types ----------------------------------------------------------------------------------------------------------------------------------- The 3DFlex workflow in CryoSPARC involves five new job types. These jobs are described in more detail in the tutorial video below. * **3D Flex Data Prep**: Prepares particles for use in 3DFlex training and reconstruction. * 3D Flex Reconstruction cannot use box sizes larger than 440 pixels, so ensure that the downsample and crop settings of 3D Flex Data Prep produce final images no larger than 440 pixels. * In CryoSPARC versions prior to v4.4, this job outputs pre-computed CTF values for use by downstream jobs. * In CryoSPARC v4.4+, the job no longer outputs full-resolution CTF values and the downstream jobs now compute CTF values (including higher order aberrations) on the fly. This change reduces disk space and CPU RAM requirements substantially and allows for higher resolution reconstructions. * **3D Flex Mesh Prep:** Takes in a consensus (rigid) refinement density map, plus optionally a segmentation and generates a tetrahedral mesh for 3DFlex. See Mesh Generation below. * **3D Flex Training**: Uses a mesh and prepared particles (at a downsampled resolution) to train a 3DFlex model. Parameters control the number of latent dimensions, size of the model, and training hyperparameters. This job outputs checkpoints during training. * **3D Flex Generator:** Takes in a checkpoint from training and generates volume series from it, to show what the model is learning about the motion of the particle. This job can be run while training is ongoing to see progress along the way. This job can also optionally take in a high-resolution density map (e.g., from 3D Flex Reconstruction) and will upscale the deformation model and apply deformations to the high resolution map. * **3D Flex Reconstruction**: Takes in a checkpoint from training as well as prepared high-resolution particles and performs high-resolution refinement using L-BFGS under the 3DFlex model. This is the stage at which improvements to density in high-res regions are computed. Outputs two half-maps that can be used for FSC validation, sharpening, and other downstream tasks. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#tutorial-video) Tutorial Video --------------------------------------------------------------------------------------------------------------------------------------------- Please watch the following tutorial video that covers usage of 3DFlex. It explains details of the job types, parameter tuning, and other considerations. Most of these details are not currently in written form in the documentation so we encourage users to watch the entire video. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#mesh-generation) Mesh Generation ----------------------------------------------------------------------------------------------------------------------------------------------- The tetrahedral mesh is an important concept in 3D Flexible Refinement. We cover this topic in significantly more detail in the [dedicated guide page on the topic](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation) . As discussed in the preprint, regularization of deformations is critical for a method like 3DFlex. Without strong regularization, the deep generative model can easily overfit to noise in the data and learn unrealistic deformations. 3DFlex uses a tetrahedral mesh (similar to Finite Element Methods) to represent deformation, and applies a rigidity prior that encourages the model to avoid non-rigidity unless it is well supported by the data. In 3DFlex, we define a tetrahedral mesh (or tetramesh) using: * a set of vertices * a set of tetra cells, each connecting four vertices * a “tetra index map”, which is an NxNxN map of indices indicating for each voxel, which tetra cell that voxel belongs to. The tetramesh is defined during the setup of a model. During training, the flow generator outputs a deformation field as a set of deformations of each vertex of the tetramesh, and the convection operator uses the tetra index map to determine how to convect the canonical density based on the movement of the mesh vertices. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#mesh-topology) Mesh Topology By default, the 3D Flex Mesh Prep job will automatically generate a regular tetrahedral mesh of specified coarseness, and this typically yields good results, but the 3DFlex method works with any mesh geometry. The mesh topology can be adjusted to introduce additional inductive bias into the model. This is particularly useful for resolving motion of adjacent domains that move differently from each other. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fx4WfdvWWMMUGzIbH70Ur%252Fv4-1-0-3dflex-mesh.png%3Falt%3Dmedia%26token%3D5bfdc95b-29c6-4b74-8b8d-13104f5ab7cf&width=768&dpr=3&quality=100&sign=33007505&sv=2) For example, for the SARS-CoV-2 spike protein we obtained good results with a mesh constructed using a sub-mesh for each RBD and NTD domain, fused to a sub-mesh for the central trimer of S2 domains (Figure 11). To construct such a mesh, we provided coarse boundaries between adjacent RBD and NTD domains to the 3D Flex Mesh Prep job, along with the desired topology of the mesh (i.e. which parts are connected to which other parts). The job then automatically generates sub-meshes and fuses them together to form a complete mesh. Please watch the following tutorial video for details about how to use the 3D Flex Mesh Prep job to adjust mesh topology. The 3D Flex Mesh Prep job supports input of `.seg` files generated by [UCSF Chimera’s Segger](https://github.com/gregdp/segger) tool. This is the easiest way to denote coarse boundaries between segments. The job also supports input of your own custom `.mrc` files that you can create that label each voxel with a segment number. The use of custom mesh topology provides helpful inductive bias but does not provide 3DFlex with information about the direction nor types of molecular motions present in the data. Rather, 3DFlex must still learn a non-linear non-rigid deformation from scratch across all mesh nodes jointly during training. Whether using a regular or custom mesh, there is substantial latitude in specifying the mesh. Where motions are smooth, the size and shape of mesh elements and their precise locations are not critical since they only serve to ensure the deformation is smooth, and the flow generator is able to displace the mesh elements (including changing their size or shape) during deformation. Likewise for custom meshes, the separation of subdomains does not need to be “exact” as the canonical voxel density values and structure within each region of the mesh are still learned from the data by 3DFlex. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#rigidity-weights) Rigidity Weights Along with the mesh topology, 3DFlex also defines rigidity weights for the mesh. The rigidity weight for each cell denotes the relative strength of the rigidity prior that should be applied to that cell. The overall strength of the prior is also a parameter (set at training time) but the relative rigidity is part of the mesh definition. For example, empty space between two subunits should not be very rigid and should be able to compress/expand allowing the subunits to move apart, while high density core parts of a subunit are more likely to remain rigid during deformation). By default, the 3D Flex Mesh Prep job will automatically generate rigidity weights based on the amount of density within each cell in the input consensus (rigid) refinement map. It is also possible to modify rigidity weights or provide custom rigidity weights to 3DFlex. [See this example **cryosparc-tools** notebook](https://tools.cryosparc.com/examples/3dflex-custom-latent-trajectory.html) . ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#fully-custom-meshes) Fully Custom Meshes It is possible to create and input fully custom meshes for 3DFlex using **cryosparc-tools**. [This example notebook](https://tools.cryosparc.com/examples/3dflex-custom-latent-trajectory.html) includes more details about how a mesh is defined and how to provide your own vertices, cells, tetra index map, and rigidity weights. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#parameter-tuning) Parameter Tuning ------------------------------------------------------------------------------------------------------------------------------------------------- Several parameters of the 3DFlex algorithm must be tuned for each dataset in order to give the best results. Details about parameter tuning are in the tutorial video. The important parameters to tune are: * 3D Flex Mesh Prep: * `Base num. tetra cells` controls the fineness of the tetramesh. Finer meshes allow for more detailed motion but reduce the regularization and with poor quality data or small particles can lead to overfitting. Note that currently 3D Flex Mesh Prep cannot create a mesh with `Base num. tetra cells` greater than 40. * `Segmentation` and `Rigidity weighting` see Mesh Generation above. * 3D Flex Training * `Number of latent dims` usually best to start with 2, and increase if the data appears to have more complex motions (and sufficient signal to resolve more motions) * `Number of hidden units` can be reduced to e.g., 32 to limit the capacity of the flow generator model for cases with simpler motion or where overfitting is a concern. * `Rigidity (lambda)` controls the overall strength of the rigidity prior. This should be tuned carefully through empirical tests. When too high, the model will ignore more detailed motions in the data. When too low, the model may learn unrealistic motions due to noise in the data. * `Noise injection stdev` controls the noise injected during latent inference. Higher values encourage more smoothness of the latent conformational landscape (i.e., nearby latent positions will encode similar conformations) but higher values also reduce precision in latent inference, potentially limiting how precisely flexible parts are aligned. * `Latent centering strength` controls the strength of a prior that tries to ensure that latent coordinates are generally centered in the latent space and stay within the range (-1.5, 1.5). This must be tuned for each dataset if you see that latent coordinates are all close to zero or are all hitting the edge of the (-1.5, 1.5) domain. It does not have impact on the results or capacity of the model and is simply a nuisance parameter. * 3D Flex Reconstruct * `Max BFGS iterations` is set to 20 by default. This can be increased for large box sizes or very high resolution. Also, in some cases it is possible for the FSC curve after 3DFlex Reconstruct to not drop off to zero at high resolutions or appears clearly artefactual, which is an indication that the BFGS optimization has not fully converged. In these cases, it can help to increase this parameter to 40. * `Load all particles in RAM` is a new option in CryoSPARC v4.4 that is off by default, meaning that particle images will be read from the project directory or from SSD cache during iterations of reconstruction, rather than being first pre-loaded into CPU RAM at the start of the job. Keeping this parameter off substantially reduces the CPU RAM requirements of the job, allowing for larger box size reconstructions. Turning the parameter on may improve speed. * `Cache particle images on SSD` is a new option in CryoSPARC v4.4 that is on by default, causing particle images to be cached at the start of the job. Turning this off will cause particles to be read directly from project directories instead of being copied to the cache. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#limitations) Limitations --------------------------------------------------------------------------------------------------------------------------------------- 3DFlex is an advance in modelling continuous heterogeneity but it does have several limitations. The most important are listed here: * Compositional heterogeneity. Being a motion model, 3DFlex currently does not have a way to cleanly represent compositional heterogeneity. It is able to move density around, but cannot delete or add density (the opposite of density-based methods like 3DVA, cryoDRGN, etc.). As such, when presented with data that contain compositional heterogeneity, it may result in strange effects. For example, a domain that is partially occupied in the data may be modelled by creating a deformation that “expands” that domain over a wide space, thereby causing the density to drop, appearing like that domain has been erased. This is obviously not ideal behavior and the 3DFlex model will waste capacity modelling this compositional change rather than conformational changes. Improving 3DFlex in compositional cases is an area of development. Currently we suggest using 3D Classification and Heterogeneous Refinement jobs to ensure that discrete compositional heterogeneity is separated as much as possible before inputting particles into 3DFlex. * Intricate motions. Though 3DFlex does well in modelling motion even of relatively small parts of a particle, it is not yet capable of modelling highly intricate motions such as side chain or loop motion. These motions are far smaller than the setup of 3DFlex (e.g., using a tetramesh) can allow to be modelled. Furthermore, small motions and conformational changes are unlikely to even be statistically detectable in single particle data unless those motions happen in tandem with other larger changes in the molecule. * Intermediate states with no data. 3DFlex is strongly biased to modelling motion, and so when presented with data with discrete heterogeneity, it will likely learn a model that maps the multiple discrete states together under deformations that unite them. However, if the data is discrete, there will not be any signal about the actual conformational states of intermediate positions between the discrete endpoints of motion. 3DFlex will still model these transitions, but it will only be guided by its rigidity prior for intermediate states that are not actually seen in the data. * Interpretation of latent space. The interpretation of 3DFlex is also an interesting area for future work. It is unclear how one should relate the continuous probability distribution of particle images in the 3DFlex latent space to a physically meaningful notion of energy via a Boltzmann distribution. This is because the non-linear capacity of the flow generator means that relative distances and volumes (and hence probability density) in the latent space are arbitrary. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#computational-considerations) Computational Considerations ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 3DFlex is relatively computationally demanding. It is GPU accelerated. Memory: * GPU memory use is relatively limited during training time, but at reconstruction time the GPU must be able to fit at least 2x the size of a volume at the full resolution box size. We have not yet finely profiled memory usage so it may be more. * CPU memory in CryoSPARC v4.4+: * 3DFlex loads all particles into CPU memory at training time. This means you must have sufficient CPU RAM to fit the entire dataset (at the training box size). During 3D Flex Data Prep, you can limit the number of particles. During 3D Flex Reconstruction, particles are read from SSD cache by default and therefore do not need to all fit in CPU RAM. * CPU memory in CryoSPARC prior to v4.4: * 3DFlex loads all particles into CPU memory at training time and reconstruction time. This means you must have sufficient CPU RAM to fit the entire dataset (at the training box size for train time, and at the high resolution box size for reconstruction time). During 3D Flex Data Prep, you can limit the number of particles as well. * 3DFlex does not yet use the CryoSPARC particle caching system. It reads particles directly from project directories into CPU RAM at the start of processing. Speed: * Speed of 3DFlex training (and reconstruction) are primarily driven by two factors: the number of latent dimensions and the number of voxels (i.e. the volume) inside the solvent mask. Training time will increase approximately linearly with both of these factors. Therefore to speed up training, downsampling to a smaller size (while still retaining enough resolution for training to pick up secondary structure, etc.) is very helpful. Similarly, the solvent mask should not be made overly loose (though it should also be loose enough not to cut off any density in flexible regions that are not well resolved in the consensus rigid density). * Performance appears to be more strongly affected by GPU performance than other CryoSPARC job types. We have not yet extensively characterized performance but newer/faster GPUs appear to provide substantial benefits. [PreviousTutorial: 3D Variability Analysis (Part Two)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-two) [NextInstalling 3DFlex Dependencies (v4.1–v4.3)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement/installing-3dflex-dependencies) Last updated 1 month ago * [Overview](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#overview) * [Example Results](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#example-results) * [Installing 3DFlex (CryoSPARC v4.1–v4.3 only)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#installing-3dflex-cryosparc-v4.1-v4.3-only) * [Job Types](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#job-types) * [Tutorial Video](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#tutorial-video) * [Mesh Generation](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#mesh-generation) * [Mesh Topology](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#mesh-topology) * [Rigidity Weights](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#rigidity-weights) * [Fully Custom Meshes](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#fully-custom-meshes) * [Parameter Tuning](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#parameter-tuning) * [Limitations](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#limitations) * [Computational Considerations](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#computational-considerations) --- # Tutorial: CTF Refinement | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement.md) . [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#overview) Overview ------------------------------------------------------------------------------------------------------------------------- CTF Refinement includes two major components: local (per-particle) CTF refinement and global (per-group) CTF refinement. `Local CTF Refinement` adjusts each particle's defocus value to estimate the z-position of the particle in the sample/ice. `Global CTF Refinement` adjusts the higher-order CTF terms (beam-tilt, trefoil, spherical aberration, tetrafoil) across an entire group of images to find the optimum values, accounting for misalignment or aberrations in the microscope itself. In CryoSPARC, both local and global CTF refinement can be performed standalone (using aligned particles and a reference volume as input) or they can be performed on-the-fly during a 3D refinement, so that the values are iteratively optimized along with particle alignments. **New:** As of CryoSPARC v3.3+, `Global CTF Refinement` now supports the estimation and correction of anisotropic magnification present in the particle images. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#local-ctf-refinement-per-particle-defocus) Local CTF refinement (per-particle defocus) --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This is a relatively straightforward optimization process of finding the optimal per-particle defocus for each particle in a dataset. Per-particle defocus refinement has been previously proposed and implemented in many other software packages for single particle EM (cisTEM, RELION, Thunder, etc.). `Local CTF Refinement` in CryoSPARC requires aligned particle images and a 3D reference (two half-maps), ideally already at a high resolution. Experimental particle images are compared against the 3D reference from their half-set, from the best known pose, at various defocus levels, and the best defocus is selected. The optimal defocus ideally corresponds to the height of the particle in the sample/ice. Since each particle can be at a different height and ice thicknesses can be 10 times larger than the particle diameter in many cases, per-particle defocus refinement can often make a large difference in the accuracy of CTF correction for each particle. However, it generally works best for larger, highly rigid, high quality samples that already reach relatively high resolutions (better than 4A). In general, it is a good idea to try local CTF refinement on every dataset, and to use a homogeneous (gold-standard) refinement to check whether the overall resolution increased or decreased. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#run-local-ctf-refinement) Run Local CTF Refinement Create a `Local CTF Refinement` job using the job builder and connect particles from a previously-run refinement (the particles must have `alignments3D` defined). Also connect the refined volume from the _same_ refinement job. You can optionally connect a separate mask input, otherwise the `mask_refine` that is included in the volume input from the previous refinement will be used by default. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNdpUVEBpY4h3lKwHjX%252F-MNdqPytWo-Pksh92U8E%252FCTF_ctf-refinement-1.png%3Falt%3Dmedia%26token%3Dbf8e8804-b2a5-489c-b6c7-0f08314ffabc&width=768&dpr=3&quality=100&sign=a6939c54&sv=2) Job builder for Local CTF Refinement The most important parameters to adjust are: * Minimum fit res (A): controls the minimum resolution used for fitting. Generally CTF refinement should be done only with medium to high resolution signal, as low resolution signal can throw off CTF fits. For smaller particles, change this to a higher resolution. * Maximum fit res (A): controls the maximum resolution used for fitting. Higher resolution signal is better for CTF refinement, until there is too much noise present in the half-maps. Leave this blank to have the maximum resolution automatically determined via FSC between the two input half-maps. * Defocus search range: controls how far above and below the current defocus to search for the optimal defocus of each particle. If you used `Patch CTF Estimation` previously in cryoSPARC, this value can be made relatively small, about the same as the thickness of ice you expect to have in the sample, since the input defocus values will already be fairly accurate. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNdrX9GPRc9p3u4zG0g%252F-MNdsjFEbHK7GaiJQv6V%252FCTF_ctf-refinement-2.png%3Falt%3Dmedia%26token%3D225cef45-1ca2-4ed9-9ada-5b135a3e99b4&width=768&dpr=3&quality=100&sign=1337c4a1&sv=2) Once the job is run, several diagnostic plots will be created that show the progress of CTF refinement. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNdrX9GPRc9p3u4zG0g%252F-MNdsmAkUDORNsWUlDTQ%252FCTF_ctf-refinement-3.png%3Falt%3Dmedia%26token%3D98fe9fc0-1283-4724-8aed-9d9438257fbc&width=768&dpr=3&quality=100&sign=b978baaa&sv=2) Plots of per-particle defocus error landscapes show the change in log-likelihood across the range of tested defocus values. The curves should like like these above, showing a clear minimum near 0 change in defocus. The X-axis is in units of Angstroms. The Y-axis is in log units, so each change of 1 unit corresponds to a change of 1/e^1 = 0.367 in probability. Therefore, plots with a minimum that is hundreds of units deep indicate that we are highly confident about the optimal defocus value. On the other hand, plots with very shallow minima (tens of units) indicate uncertainty in the optimal defocus. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNdrX9GPRc9p3u4zG0g%252F-MNdsptcmzFmozYxebiE%252FCTF_ctf-refinement-4.png%3Falt%3Dmedia%26token%3D8f8d6459-ad2e-4c3a-b06d-db3fb49a6878&width=768&dpr=3&quality=100&sign=81112ed6&sv=2) Histograms showing the change in per-particle defocus across all the particles in the half-set indicate the total amount of deviation from the input defocus parameters that was achieved by CTF refinement. The histogram should generally be very peaked near zero and should not have heavy tails. Heavy tails, or the presence of many particles having optimal defocus values at the ends of the search range indicates that defocus refinement was not very confident or accurate. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#global-ctf-refinement-per-exposure-group-beam-tilt-trefoil-spherical-aberration-tetrafoil-and-anisot) Global CTF Refinement (per-exposure-group beam-tilt, trefoil, spherical aberration, tetrafoil, and anisotropic magnification) ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Ultra high resolution cryo-EM structures require correcting for electron-optical aberrations and microscope misalignments that result in nuanced "high-order" terms in the Contrast Transfer Function (CTF). These higher order terms (corresponding with beam tilt, trefoil, spherical aberration, tetrafoil) can only be detected at very high resolution, and cannot easily be estimated by straightforward measurements on the microscope. Therefore, the strength of each of these aberrations must be estimated from single particle data itself, by refining the corresponding CTF parameters against a high-resolution reference map. This process of high-order aberration estimation and correction was pioneered by (Zivanov et al. 2019) in RELION 3.1. While microscope misalignments can result in higher order terms affecting the CTF, microscopes occasionally show magnification anisotropy. The result of this anisotropy is that micrographs are slightly distorted by a linear transformation (or "stretch") in the image plane. Unlike higher order aberrations, anisotropic magnification cannot be corrected by better microscope alignment, and must either be estimated using the diffraction pattern of known crystalline samples, or by projection-matching using a high quality reference map. As of CryoSPARC v3.3+, the latter method of anisotropic magnification estimation and correction is now supported, which also follows closely the developments made by Zivanov et al. High-resolution signal is typically required to estimate any anisotropy, and unless the anisotropy is extreme, correcting for it will typically only improve maps that have already reached a fairly high resolution. Furthermore, errors in defocus and astigmatism due to magnification anisotropy are also corrected when fitting the magnification matrix. CryoSPARC v2.12+ contains a GPU accelerated implementation of high-order aberration estimation and correction. In all cases, estimation is done by directly maximizing the likelihood of observing the experimental images given a 3D reference map, using _LBFGS_. Images collected on a given microscope generally will have related CTF parameters for higher-order aberrations and anisotropic magnification. The images that are related (same grid, same image shift position, etc.) can be grouped into "exposure groups" so that they can all be refined at once, with more signal. Creation and management of exposure groups is explained in the next section. Like local CTF refinement, `Global CTF Refinement` generally works best with larger, more rigid particles. However, `Global CTF Refinement` does use signal from all the particles in an exposure group, and so can detect beam tilt and other aberrations even with smaller/flexible structures. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#run-global-ctf-refinement) Run Global CTF Refinement Create a `Global CTF Refinement` job using the job builder and connect particles from a previously-run refinement (the particles must have `alignments3D` defined). Also connect the refined volume from the _same_ refinement job. You can optionally connect a separate mask input, otherwise the `mask_refine` that is included in the volume input from the previous refinement will be used by default. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MdIgzBqtqSpXU9spoAd%252F-MdIwrGz50JE1RuBPa1w%252Fglobal_ctf_refine_builder.png%3Falt%3Dmedia%26token%3D945ac331-4690-4a77-9c73-24ca7a03b13d&width=768&dpr=3&quality=100&sign=98005157&sv=2) Job builder for Global CTF Refinement The most important parameters to adjust are: * Number of iterations: controls the number of iterations of CTF refinement that are done. It is important that the number of iterations is at least 2 when anisotropic magnification is being estimated together with the other aberrations. This is to allow the aberrations to be fit to the data while accounting for anisotropic magnification. If only aberrations are being estimated, 1 iteration is usually sufficient. * Minimum fit res (A): controls the minimum resolution used for fitting. Generally global CTF refinement should be done only with medium to high resolution signal, as low resolution signal can be unreliable. For smaller particles, change this to a higher resolution. * Maximum fit res (A): controls the maximum resolution used for fitting. Higher resolution signal is better for CTF refinement, until there is too much noise present in the half-maps. Leave this blank to have the maximum resolution automatically determined via FSC between the two input half-maps. * Fit Tilt/Trefoil/Spherical Aberration/Tetrafoil: Select which higher-order aberrations should be refined. Tilt and Trefoil are 3rd order and require less high resolution signal to accurately detect, compare to spherical aberration and tetrafoil which are 4th order. In some cases, optimizing the 4th order terms can be detrimental, especially if per-particle defocus or the 3rd order terms are not yet correctly refined. Note: as of v4.0, only the third-order aberrations (Tilt and Trefoil) and fit by default, whereas the fourth-order aberrations (Spherical Aberration and Tetrafoil) are _not_ fit by default. * Fit Anisotropic Magnification: Activate to enable the estimation of anisotropic magnification. Note that this is _inactive_ by default, since significant anisotropic magnification is a relatively rare phenomenon compared to beam tilt and other aberrations. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNdrX9GPRc9p3u4zG0g%252F-MNdt0BLYoavbgO9fjYH%252FCTF_ctf-refinement-6.png%3Falt%3Dmedia%26token%3Dca6aa07f-aa1c-4c5c-b74c-ce89f5d0c8b6&width=768&dpr=3&quality=100&sign=1930db1e&sv=2) Once the job is run, several diagnostic plots will indicate the phase delay and fit diagnostics of each type of aberration. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNdrX9GPRc9p3u4zG0g%252F-MNdt304-EDwtidHJcN1%252FCTF_ctf-refinement-7.png%3Falt%3Dmedia%26token%3D0e76fd9b-7db2-4d5b-9840-f358a5d69da8&width=768&dpr=3&quality=100&sign=5b4a1526&sv=2) For each order of aberration (odd and even), three plots are made. The first shows the phase error data that is measured from all the particles in aggregate. On the left is the full phase error, on the right are the masked out terms that will be used for fitting. For the odd terms, the aberrations appear as anti-symmetric patterns of delay (blue) and advancement (red) of the diffracted beam. The second plot shows the fit predicted values of the phase delay, after refining CTF parameters. The third plot shows the residual phase error between the data and fit, which should only contain noise indicating a good fit. Similar plots are made for the even terms in the CTF. Note that odd terms are optimized from zero each time the `Global CTF Refinement` job is run, meaning that the plots will always show aberrations in the measured data (first plot). Even terms, on the other hand, are optimized starting from their current input values. Therefore if `Global CTF Refinement` is run twice, the second time, the even terms will show nearly zero aberration in the measured data (since the input CTF parameters are already nearly correct). Note that in the output log of `Global CTF refinement` the units of each aberration parameter are printed. Beam-tilt is internally parameterized in Angstroms rather than radians, as converting to the latter requires a non-zero spherical aberration coefficient. Values in milli-radians are printed in cases where the spherical aberration is non-zero. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#anisotropic-magnification) Anisotropic Magnification If magnification correction is enabled, three plots are also made akin to that of the aberration plots. The first plot shows the predicted displacement per-pixel based on an unconstrained least-squares fit, which should typically show a linear trend. The absence of a trend indicates that the anisotropy is not significant, and a non-linear trend could indicate that the anisotropy is severe (so that multiple iterations are necessary to converge), or that there are other systematic effects in the data. The second plot shows the fitted displacement values based on the current estimate of the magnification matrix, and the third plot shows the residual (i.e. the unconstrained displacements minus the linear fit). Note that two sets of plots are made, showing the displacements in the x direction and the y direction separately. Similar to the even aberrations, anisotropic magnification is optimized from its current values at the start of the iteration. This is done because unlike the refinement of odd and even aberrations, the refinement of anisotropic magnification involves approximations to the log likelihood objective function, and this approximation improves as the magnification matrix converges. As well, all high-order CTF parameters are fit to the current values of the magnification matrix. For these reasons, it is recommended to perform at least 2 iterations of CTF refinement when fitting aberrations together with anisotropic magnification. After two iterations, the residual anisotropy should typically be very small. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MdI5RPpweybR2MSTVKE%252F-MdIYjiJkgrcsJcGoTHv%252Fanisomag_plot_final_iter.png%3Falt%3Dmedia%26token%3D241b7205-cfe9-4147-a488-4101550db9a8&width=768&dpr=3&quality=100&sign=6b031ad7&sv=2) Anisotropic magnification plot for EMPIAR-10395. Above is an example of the anisotropic magnification plot from the first iteration, for [EMPIAR-10395](https://www.ebi.ac.uk/pdbe/emdb/empiar/entry/10395/) . On the left are the displacement plots for the Y coordinate from the first iteration, indicating that there is a significant linear trend in the predicted displacements at each voxel. On the right are the plots from the final iteration, showing a residual with no linear trend, hence no fit. The absence of a fit in the final iteration plot indicates that the anisotropic magnification matrix has converged. This job was run with three iterations. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#ewald-sphere-correction) Ewald Sphere Correction ------------------------------------------------------------------------------------------------------------------------------------------------------- Both Local and Global CTF Refinement may be run with Ewald Sphere correction enabled. This means that estimation of per-particle defocus and high-order aberration parameters can be done while accounting for the curvature of the Ewald Sphere. Generally, this does not significantly impact the outcome unless previous reconstructions have shown that Ewald Sphere correction results in a measurable resolution increase. To use this feature in either Local or Global CTF Refinement jobs, activate the `Account for EWS curvature` parameter and make sure to set the `EWS curvature sign` to the correct value of curvature determined from previous reconstructions with Ewald Sphere enabled. For more information on how to obtain these reconstructions and the curvature sign, please refer to the Ewald Sphere Correction tutorial for a detailed walkthrough. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#on-the-fly-ctf-refinement-in-homogeneous-refinement) On-the-fly CTF refinement in homogeneous refinement --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In CryoSPARC v2.12+, both `Local CTF Refinement` and `Global CTF Refinement` can be run as standalone jobs. However, since the refinement of these parameters is very fast, they can also be run on-the-fly during iterations of `Homogeneous Refinement`. In the new `Homogeneous Refinement` job in v2.12+, there are new parameters to enable local and/or global CTF refinement. CTF refinement is carried out iteratively with refinement of 3D poses and the 3D map, starting once the initial refinement is converged. The new `Homogeneous Refinement` job in v2.12+ will create plots similar to the standalone CTF refinement jobs, and the final CTF parameters after refinement will be outputted along with the 3D alignments of particles. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#non-uniform-refinement-with-high-order-ctf-correction) Non-uniform refinement with high-order CTF correction ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- In CryoSPARC v2.12+, the `Non-uniform Refinement` job has been updated to use the new GPU code that supports higher-order CTF correction, but this is **NOT** enabled by default. You must turn on the `Enable higher-order CTF` parameter in `Non-uniform refinement`. Please also note that legacy refinement jobs will not support the correction of high-order CTF aberrations or anisotropic magnification. On-the-fly CTF refinement cannot be done during a `Non-uniform Refinement`, so particles should be processed through the standalone `Local CTF Refinement` then `Global CTF Refinement` jobs first. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#exposure-groups) Exposure Groups --------------------------------------------------------------------------------------------------------------------------------------- In CryoSPARC v2.12+, particles, movies, and micrographs are organized into "Exposure Groups", which allow images with the same microscope configuration (beam tilt, image shift, etc) to have their CTF refinement done independently in a streamlined manner. This section describes the tools in CryoSPARC to create and manage exposure groups. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#at-import-time) **At Import time** When you import a dataset (movies, micrographs or particles) in CryoSPARC v2.12+, the set of imported data is automatically set with a new "exposure group ID". This ID is **unique within a project** (the group ID increments with each import job, starting from zero) unless overridden using the `Override Exposure Group ID` parameter. Using this method, you can import your datasets separately based on their beam tilt groups, or any other groups where you would like to use, and the grouping of imports will be retained even if the datasets are merged later on in processing. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#using-the-exposure-groups-utilities) **Using the** `**Exposure Groups**` **utilities** If you have a dataset that was imported prior to v2.12, or a dataset that contains multiple exposure groups and you would like to separate each of the groups in the dataset, you can use the [`Exposure Group Utilities` Job](https://cryosparc.com/docs/reference/jobs#exposure-group-utilities) . This job allows you to view, split, and combine datasets into one or more exposure groups. To split a dataset into exposure groups, can select which file path attribute of the dataset will be used to identify unique groups. For example, in EPU, when capturing multiple images per hole, each shift position should be separated as a separate group. The groups can be identified by the first section of numbers right after the the word "Data" in the file path, as outlined below: FoilHole\_21256428\_Data\_**21254194**\_21254195\_20190622\_0517\_Fractions.mrc Knowing this, we can separate our exposure (or particle) dataset into unique exposure groups. Input your data into the `Exposure Group Utilities` job, and select the `split` mode. Use the parameters `Field to use to split Dataset`, `Start Slice Index`, and `Number of characters to Consider` ([more information here](https://cryosparc.com/docs/reference/jobs#exposure-group-utilities) ) to create unique tokens out of the file paths available. The job automatically creates and sets exposure groups for these tokens: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNdrX9GPRc9p3u4zG0g%252F-MNdtD4as3nWvWzaXx9x%252FCTF_ctf-refinement-8.png%3Falt%3Dmedia%26token%3Ddfd810ec-7f1f-4cdc-9d1d-f2ea480f7b34&width=768&dpr=3&quality=100&sign=72573b48&sv=2) You can choose to output each exposure group separately by using the `Split Outputs by Exposure Group` parameter. You can also combine multiple exposure or particle datasets by connecting them all into the respective input slot in the Exposure Groups Utilities job. Using the `combine&set` mode and `Set Exposure Group Value` parameter, you can combine all connected datasets and set their exposure group to the same value. Note that when this happens, the job will check that the CTF values across the exposure group are consistent- you can decide what the job will do if it finds inconsistent values using the `Combine Strategy` parameter. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#for-advanced-users) For advanced users Another way to modify the exposure group ID for a dataset is to Export the job that created the dataset (or create a .csg file manually) [(view data management tutorial)](https://cryosparc.com/docs/tutorials/data-management) and modify the .cs file directly. You will have to modify the field `ctf/exp_group_id` (and `mscope_params/exp_group_id` for movie/micrograph datasets or `location/exp_group_id` for particle datasets) for all items inside the dataset. You can set these columns with the desired group identifiers, which do not need to be sequential but do need to be unique. If your dataset does not have this result slot (which may be the case for jobs **not** processed by Patch CTF Estimation or imported prior to v2.12), you will have to first add the field, then modify the fields. See the python (2.7) example below. You can then re-import this dataset using the Import Result Group job. [PreviousTutorial: Maximum Box Sizes for Refinement](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/performance-metrics) [NextTutorial: Ewald Sphere Correction](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ewald-sphere-correction) Last updated 2 years ago * [Overview](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#overview) * [Local CTF refinement (per-particle defocus)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#local-ctf-refinement-per-particle-defocus) * [Run Local CTF Refinement](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#run-local-ctf-refinement) * [Global CTF Refinement (per-exposure-group beam-tilt, trefoil, spherical aberration, tetrafoil, and anisotropic magnification)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#global-ctf-refinement-per-exposure-group-beam-tilt-trefoil-spherical-aberration-tetrafoil-and-anisot) * [Run Global CTF Refinement](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#run-global-ctf-refinement) * [Ewald Sphere Correction](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#ewald-sphere-correction) * [On-the-fly CTF refinement in homogeneous refinement](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#on-the-fly-ctf-refinement-in-homogeneous-refinement) * [Non-uniform refinement with high-order CTF correction](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#non-uniform-refinement-with-high-order-ctf-correction) * [Exposure Groups](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#exposure-groups) * [At Import time](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#at-import-time) * [Using the Exposure Groups utilities](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#using-the-exposure-groups-utilities) * [For advanced users](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement#for-advanced-users) Copy #import the modules from cryosparc_compute import dataset from cryosparc_compute import common #load the dataset particle_dataset = dataset.Dataset.load() #add missing fields (this example is for particle datasets) particle_dataset = common.create_missing_fields_in_dataset(particle_dataset, 'ctf', 'particle.ctf') #set the exposure group id particle_dataset['ctf/exp_group_id'][:] = 2 --- # Tutorial: EPU AFIS Beam Shift Import | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import.md) . [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import#introduction) Introduction --------------------------------------------------------------------------------------------------------------------------------------------- Thermo Fisher’s [EPU data collection software](https://www.thermofisher.com/ca/en/home/electron-microscopy/products/software-em-3d-vis/epu-software.html) is a commonly used data acquisition software for single particle analysis. Typically, SPA data has been collected via manually moving the stage around, placing different regions within the hole at the optical center of the microscope. This is done in order to avoid the strong aberrations that result from off-axis use of the objective lens. However, collecting data in this manner induces delays from moving the stage and waiting for the stage to settle. Thus, advances such as [EPU’s Aberration-Free Image Shift (AFIS) collection mode](https://assets.thermofisher.com/TFS-Assets/MSD/posters/MM2019-poster-advances-SPA-data-acquisition.pdf) have allowed for targeting multiple holes without stage movement in-between each hole. Importantly, AFIS and associated microscope calibration service allow for targeting holes that don’t lie at the optical center of the microscope **without** inducing severe artefacts, and this significantly speeds up data collection. For many datasets collected via AFIS mode, it is still worthwhile to estimate residual higher-order aberrations such as coma via the Global CTF Refinement job: if there are any residual aberrations, correcting them may lead to improved structures. However, doing so requires grouping movies into subsets ([Exposure Groups](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-refinement/job-exposure-group-utilities) ) based on similar optical conditions, which includes the amount of applied beam shift. Since refinement of higher-order CTF aberrations is done separately for each exposure group, the assignment of movies into exposure groups can have significant impacts on the aberration values, which will impact the resolution achieved by subsequent refinements. In CryoSPARC v4.4+, we have integrated the import of beam shift values from EPU sessions collected in AFIS (Aberration-free Image Shift) mode to allow for Exposure Group assignments based on applied beam shift. The following tutorial covers: * how to import movies/micrographs with beam shift values * how to assign movies/micrographs into exposure groups based on beam shift * merging beam shift values into pre-v4.4 movie/micrograph datasets, without re-processing from scratch * continuing processing data from live [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import#use-case-1-clustering-movies-into-exposure-groups-at-import-time) Use case #1: Clustering movies into exposure groups at import time ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This use case covers the situation where dataset processing starting in CryoSPARC from scratch, i.e., all processing steps post-motion correction (including particle picking) have not been done yet. For existing CryoSPARC exposure datasets, datasets processed in CryoSPARC Live, or datasets with existing particles, please refer to the subsequent use case #2. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import#import-movies-or-import-micrographs) Import Movies or Import Micrographs For movie or micrograph datasets collected via EPU’s AFIS mode, beam shift values can now be imported along with other metadata. As an example, here is a screenshot of an output data directory from a data collection session using EPU. Note that the movies we would like to import are in `.eer` format, and the associated files containing the beam shifts are in `.xml` format. Each EER movie has a corresponding `xml` file. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FFh9Kp2T6wjTwMJrVJMWv%252Fv4-4-0-beamshift-afis-xml-example-filedir.png%3Falt%3Dmedia%26token%3D39c4122e-2353-44bf-b642-632cebbdc75f&width=768&dpr=3&quality=100&sign=5ff9042f&sv=2) In CryoSPARC’s Import Movies and Import Micrographs jobs, you will now notice an “XML Import” section that allows specification of an absolute path wildcard to the directory containing the XML files. This path is in addition to the wildcard expression pointing to the raw movie files. For the above example, we have set the two wildcard expressions to the following: * Movies data path: `/bulk9/data/EPU_apof_JLR/20230212a_T64/Images-Disc1/GridSquare_11564642/Data/*.eer` * EPU XML metadata path: `/bulk9/data/EPU_apof_JLR/20230212a_T64/Images-Disc1/GridSquare_11564642/Data/*.xml` Here they are the same paths with different file extension filters. There are 4 additional parameters that assist in finding correspondences between the `.eer` and `.xml` files. These parameters specify the number of characters to cut off of the beginning and end of the movie and XML filenames in order to match them to each other, one-to-one: * **Length of movie filename prefix to cut for XML correspondence:** Use this field to specify the number of characters to cut off the prefix of the imported movie filename, to match with the XML filename. * **Length of movie filename suffix to cut for XML correspondence:** Use this field to specify the number of characters to cut off the suffix of the imported movie filename, to match with the XML filename. * **Length of XML filename prefix to cut for movie correspondence:** Use this field to specify the number of characters to cut off the prefix of the XML filename, to match with the imported movie filename. * **Length of XML filename suffix to cut for movie correspondence:** Use this field to specify the number of characters to cut off the suffix of the XML filename, to match with the imported movie filename. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FCmj0RAEF7bvb0RzGhuwr%252Fv4-4-0-afis-filename-trim-1.png%3Falt%3Dmedia%26token%3D90d99942-bb34-4530-a8de-bd5044542954&width=768&dpr=3&quality=100&sign=6db94b8b&sv=2) Example illustrating how to determine the number of characters to cut for the movie and XML filenames. In this case, we need to trim the eight `_EER.eer` characters off of the end of the movie filename, as well as the four `.xml` characters off of the XML filenames. Thus, we’ll set the movie suffix parameter to 8, the XML suffix parameter to 4, and leave the rest empty. After inputting the parameters and running the job, a scatter plot with the beam shift values will be displayed in the event log if the XML import was successful. The event log will also print an example of the movie and XML paths after applying the prefix/suffix trim, in order to ensure that these are aligned and match in structure. If any of the XML files are absent, corrupt, or missing beam shift values, they will be flagged as having missing beam shifts; in the example image, two exposures are missing beam shift values. **Be sure to check the event log to see if the majority of exposures had successfully read beam shift values**; if this is not the case, a warning will be displayed in orange highlight. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fj7nskS3LlYevsFS0Vjum%252Fv4-4-0-beamshift-xml-import-scatterplot.png%3Falt%3Dmedia%26token%3D857b09dd-615a-40c8-87a6-5fa35d94ac58&width=768&dpr=3&quality=100&sign=702fb461&sv=2) ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import#pre-processing) Pre-processing Next, exposures must be pre-processed via motion correction (applicable to movies) and CTF estimation. CTF estimation is required to cluster exposures by the applied beam shift. The recommended motion correction job is Patch Motion Correction, and the recommended CTF estimation job is Patch CTF Estimation. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import#clustering-via-exposure-group-utilities) Clustering via Exposure Group Utilities The next step is to cluster exposures into groups based on the applied beam shift. The main purpose of clustering is to ensure that exposures with similar beam shift values are placed into the same exposure group. This can be done via running Exposure Group Utilities in the `cluster&split` mode. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FzFEHTlsxcwV0FMcHL1j4%252Fv4-4-0-afis-import-usecase1-treeview.png%3Falt%3Dmedia%26token%3D9e9d4101-952e-40f4-b649-b857bdd14c7c&width=768&dpr=3&quality=100&sign=800e8844&sv=2) Example of a tree view for the workflow (starting from importing micrographs) until exposure group utilities. First, connect the outputted exposures from the Import Movies or Import Micrograph job above. Then, specify the “Input Selection” as `exposure`, and the “Action” as `cluster&split`. Finally, set the number of clusters. In this case, based on the beam tilt scatter plot above, we counted 61 clusters, which correspond to the 61 unique “rings” (each ring corresponding to 8 different collection sites arranged in a circle on one hole). Note that it is not necessary to ensure that the number of clusters matches the number of holes precisely. Indeed, depending on the layout and orientation of holes on the grid, the beam shift distribution may not form neat clusters and may appear more continuous. In any case, the following should be noted when choosing the number of clusters: * With too few clusters, there will be greater intra-exposure-group variability in the beam shift, possibly leading to less accuracy when fitting the higher-order aberrations * With too many clusters, there will be fewer exposures and particles per exposure group, possibly limiting the precision of the fit higher-order aberration values. In extreme cases, too few particles per exposure group could impact the stability of the Global CTF Refinement aberration fitting algorithm, as there is a minimum cumulative amount of signal in each exposure group that is needed to fit the aberration parameters. This is important to keep in mind, as aberration estimation is done _independently_ for each exposure group. The “Clustering method” may also be tweaked. The most important factor when clustering exposures is that clusters are reasonably uniform in both: * the number of exposures they contain, and * the range of beam shift values they span The default of `agglomerative` clustering works well on a variety of datasets, but we also enable `kmeans`. K-means clustering works better when exposures’ beam shift values form isotropic clusters with most points located close to the mean, or when the spread over beam shifts is more “continuous” and doesn’t form neat discrete clusters. In these cases, k-means will ensure that clusters remain relatively uniform in the range of beam shift values that each cluster spans. Agglomerative clustering may perform better when clusters form more irregular shapes, such as the “rings” in this example. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F7Bbee4SEfXFbepck0ByC%252Fv4-4-0-beamshift-egutils-jobbuilder-clustering-params.png%3Falt%3Dmedia%26token%3Df2493688-7aa1-4a37-ade9-491f7df3a2d5&width=768&dpr=3&quality=100&sign=51414b76&sv=2) Once the number of clusters is chosen, queue and run the job. At the first checkpoint, the exposure group clustering result is shown. If any exposures are missing beam shift values, they will be placed into their own separate exposure group, and the number of exposure groups outputted by the job in total will be one larger than the parameter value. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FMCdUKl642vk179rkJiwc%252Fv4-4-0-beamshift-egutils-scatter-clustering-results.png%3Falt%3Dmedia%26token%3Da9d32f2f-22ec-4f05-a20e-4c1e691b2292&width=768&dpr=3&quality=100&sign=9e3c82aa&sv=2) The output exposures are now ready for downstream processing, including motion correction, CTF estimation, and particle picking. Be sure to experiment with Global CTF Refinement to see if clustering particles into exposure groups helps obtain better resolutions. Note that only exposure groups with an adequate number of particles should have their aberrations refined, as Global CTF Refinement depends on having enough signal across the particle images in the exposure group. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import#use-case-2-importing-beam-shifts-and-clustering-exposures-after-the-original-movie-import) Use case #2: Importing beam shifts and clustering exposures _after_ the original movie import ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This use case applies when beam shifts were not imported at the time of movie import, for example when: * The mode of exposure import did not support the simultaneous import of beam shifts, such as the _Import Movies_ job in a CryoSPARC version older than 4.4 or exposure import or pre-processing in CryoSPARC Live. * One did not enable the _Import Beam Shift Values from XML Files_ option when importing movies in CryoSPARC version ≥ 4.4. In this case, the following steps (outlined below) allow for re-clustering of exposure groups: * Running an Import Beam Shifts job in order to retrieve the exposures’ beam shift values; * Clustering the movies/micrographs into exposure groups via Exposure Group Utilities, with input particles provided to the job If you are importing fresh movies or micrographs into CryoSPARC v4.4+, Use case #1 covers the basic import case, which is recommended to read first. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import#import-beam-shift) Import Beam Shift Navigate to the job builder, locate the new “Import Beam Shift” job under the imports section, and build an Import Beam Shift job. This is a new job created to add beam shift information to existing exposures datasets in CryoSPARC, without need for re-importing the movies/micrographs from scratch. Next, **connect the existing movies or micrographs dataset from CryoSPARC as input to the Import Beam Shifts job**. This may be a movie dataset exported from CryoSPARC Live, or a movie dataset processed in regular CryoSPARC. Ensure that the entire movie dataset is inputted to the job (i.e. if any exposure curation had filtered out some exposures, ensure to use exposures from upstream to that job). When connecting movies as input to the Import Beam Shifts job, the jobs will use the existing movies’ UIDs rather than generating new UIDs, like the other import jobs. These existing UIDs are required when updating particles’ exposure group assignments in Exposure Group Utilities, to match particles to the exposures that they came from. As in use case #1, provide the XML directory wildcard expression, that points to the directory containing the original XML files. These parameters are identical to those in Import Movies, and the instructions can be followed in [use case #1 instructions](https://www.notion.so/T-2825-Tutorial-EPU-AFIS-Beam-Shift-Import-7799197a2beb41c58fc307024a9556b2?pvs=21) . If needed, specify the four “Length of movie/XML filename prefix/suffix…” parameters to correctly match movie filenames to XML filenames. Examples of the trimmed file-paths will be printed to the event log to help determine the number of characters. The values of these parameters is most quickly determined by running the job with all defaults, and observing the event log. For example, when connecting movies that were previously imported to CryoSPARC and running the Import Beam Shift job, the event log shows the following messages: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FlhPdBZKbb7K2I4mII2vn%252Fv4-4-0-beamshift-usecase2-importbeamshift-sourcequery-nomatch.png%3Falt%3Dmedia%26token%3D9013796a-5026-49f3-9406-dba056e1200f&width=768&dpr=3&quality=100&sign=20eec040&sv=2) Here, the example movie/mic filename is the same as the XML filename, except for the trailing `_EER.eer`. Due to these extra characters, the beam shift import was not successful, and CryoSPARC warned that it did not find the beam shifts associated with any of the 2797 exposures. To fix this, we can set the “Length of movie filename suffix to cut for XML correspondence” parameter to 8 to cut off the trailing 8 characters and find proper matches between the XML and movie files. Re-running the job, we see that the XML files were found for all but two exposures, which happen to be missing from this dataset: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fg4kThIv3vNP36r96nBGM%252Fv4-4-0-beamshift-usecase2-importbeamshift-sourcequery-successful.png%3Falt%3Dmedia%26token%3D1a6a1a59-11be-4387-84d4-a94868762317&width=768&dpr=3&quality=100&sign=7a23b1c3&sv=2) Finally, if the XML import was successful and beam shifts were present in the XML files, a beam shift scatter plot will be displayed in the event log as in use case #1. The UIDs and all input slots (e.g. motion correction or CTF estimation results) will have been pulled from the input dataset, meaning we do not have to repeat these steps if they have already been done. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import#clustering-via-exposure-group-utilities-1) Clustering via Exposure Group Utilities If particles were already picked, we also do not have to repeat particle picking and can instead assign particles to exposure groups based on which exposures they came from. This can also be done via the Exposure Group Utilities job. In this case, we can [use Exposure Group Utilities as described above](https://www.notion.so/T-2825-Tutorial-EPU-AFIS-Beam-Shift-Import-7799197a2beb41c58fc307024a9556b2?pvs=21) , with the following (**bolded**) modifications: * Connect the output exposures from “Import Beam Shift” **and the existing particle dataset** to Exposure Group Utilities; * As in use case #1, Set the “Input Type” to `exposure`, specify the “Action” as `cluster&split`, and specify the number of clusters and clustering method; * Activate the “**Correspond particles to exposures and enforce consistency of exposure group IDs**” parameter * **If particles were previously split into more than one exposure group, set the “Combine strategy” to** `**take_mode**` * This ensures that when particles from different exposure groups are combined into the same group, the aberration values for the entire group will be set to the mode (most common value) amongst particles in the group. Since exposure group clustering is done with the purpose of re-running Global CTF Refinement, aberrations will be re-refined and this is not a point of concern. In our case, particles were previously from only one exposure group, so we don’t need to change the combine strategy. Thus, we’ll run the job with the following input parameters: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FqJ1SF4taXsVCXshLVXxd%252Fv4-4-0-beamshift-usecase2-egutils-jobbuilder-clustering-params-correspond-prtcls2egs.png%3Falt%3Dmedia%26token%3Dbe55b286-e559-4dcc-8a3b-bca37b947837&width=768&dpr=3&quality=100&sign=6bbb4b10&sv=2) Checkpoints 1 and 2 will show the exposure and particle datasets’ exposure group information prior to clustering, respectively, in a table format in the event log. In most cases, particles and exposures will initially be all pooled into one exposure group, unless they were assigned different exposure group IDs upon import. Checkpoint 1 will also show the beam shift scatter plot labelled by the assigned exposure groups. Checkpoints 3 and 4 will show the exposure and particle datasets’ exposure group information after clustering. If “**Correspond particles to exposures**” was activated, the particles and exposures datasets should be consistent. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import#next-steps) Next Steps ----------------------------------------------------------------------------------------------------------------------------------------- Once picked particles are obtained and a relatively high-resolution structure has been obtained, use Global CTF Refinement to fit higher-order aberration values. If you followed use case #2, it’s possible to do an apples-to-apples comparison of resolutions before and after clustering particles into exposure groups. This can be done by using two Global CTF Refinement jobs and two Homogeneous Reconstruction Only jobs along with a fixed mask. In our example dataset, the resolution improvements obtained via exposure group clustering were rather modest, indicating that the microscope was quite well calibrated. However, [examples](https://discuss.cryosparc.com/t/beam-tilt-refinement-by-image-shift-groups-for-datasets-acquired-with-the-leginon-appion-suite/10682) of more significant improvements have been previously documented on the forum. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FmbPhdtOaFa8RbeYo1Njw%252Fv4-4-0-beamshift-treeview.jpg%3Falt%3Dmedia%26token%3Dc14ee928-eb31-472b-babc-5ef67506c19b&width=768&dpr=3&quality=100&sign=f7327b0a&sv=2) Example of processing movies exported from live. Movies were processed through “Import Beam Shifts” to tag them with beam shifts, then together with particles were passed through Exposure Group Utilities to cluster them based on beam shift. Global CTF Refinement was performed twice, once on the initial exposure group assignments (J135: all particles in one group) and once on the new assignments (J136). Two final reconstructions were done, with fixed poses and identical masks, to compute FSCs. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FPIM1tTWr2RM1LSz4kOo4%252Fv4-4-0-beamshift-fsc-comparison-direct.png%3Falt%3Dmedia%26token%3D9bec2cff-cd1b-4ccc-bf2c-c3ba094c09e4&width=768&dpr=3&quality=100&sign=7e3ea869&sv=2) A modest increase in resolution (1.53 Å) induced by exposure group clustering into 61 clusters, compared to the baseline of 1.54 Å associated with keeping all exposures in one cluster. Note that poses and masks were identical, thus the only variables changed between these two reconstructions were the fitted high-order aberrations from Global CTF Refinement. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import#references) References ----------------------------------------------------------------------------------------------------------------------------------------- * Dustin Morado’s [EPU\_group\_AFIS](https://github.com/DustinMorado/EPU_group_AFIS) repository for clustering strategies, as well as his [detailed forum post](https://forum.scilifelab.se/t/creating-optics-groups-from-epu-afis-data-and-more/122) describing the motivation for exposure group clustering when collecting in AFIS mode. [PreviousTutorial: EER File Support](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-eer-file-support) [NextTutorial: Patch Motion and Patch CTF](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-patch-motion-and-patch-ctf) Last updated 1 month ago * [Introduction](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import#introduction) * [Use case #1: Clustering movies into exposure groups at import time](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import#use-case-1-clustering-movies-into-exposure-groups-at-import-time) * [Import Movies or Import Micrographs](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import#import-movies-or-import-micrographs) * [Pre-processing](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import#pre-processing) * [Clustering via Exposure Group Utilities](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import#clustering-via-exposure-group-utilities) * [Use case #2: Importing beam shifts and clustering exposures after the original movie import](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import#use-case-2-importing-beam-shifts-and-clustering-exposures-after-the-original-movie-import) * [Import Beam Shift](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import#import-beam-shift) * [Clustering via Exposure Group Utilities](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import#clustering-via-exposure-group-utilities-1) * [Next Steps](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import#next-steps) * [References](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import#references) --- # Tutorial: 3D Variability Analysis (Part One) | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one.md) . [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#id-3d-variability-analysis-tutorial-part-one) 3D Variability Analysis Tutorial: Part One ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This is part one of a two-part series on 3D variability analysis in CryoSPARC. This Part One explains the background, basics, and usage of the new tools, while [Part Two](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-two) covers several case studies, interpretation of results, and advanced usage. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#introduction) Introduction --------------------------------------------------------------------------------------------------------------------------------------------------- 3D Variability Analysis (3DVA) is a powerful tool in CryoSPARC v2.9+ for exploring both discrete and continuous heterogeneity in single particle cryo-EM data sets. It is based on a different fundamental idea from previous methods like 2D/3D classification, focused refinement, multibody refinement, etc. In a standard 3D cryo-EM refinement, individual particle images are used to reconstruct a single 3D structure, under the assumption that there is only one rigid conformation present in the data. In 3D variability analysis, individual particle images are instead used to reconstruct a _continuous family_ of 3D structures. Together, this family captures the multiple discrete and flexible conformations that are present in the data. Intuitively, the family can be thought of as the set of all possible 3D conformations that appear in the sample. There may be multiple dimensions (intuitively: attributes) needed to characterize the members of the family. For example, a subunit may associate/dissociate (first attribute), while another subunit bends (second attribute), while a ligand binds/unbinds (third attribute). Each single particle image is an image of one particular member of the family, with the position of each particle along each attribute of family being known as a "reaction coordinate", "embedding coordinate", or "latent representation". 3D variability analysis in CryoSPARC solves for the continuous family of 3D structures as well as the assignment of each particle to a position within the family (reaction coordinates). As such, it is a helpful way to understand and visualize the heterogeneity and flexibility in a data set. In practice, 3DVA can be used in multiple ways. Most simply, it provides visually interpretable insight into the conformational space of a molecule, in the form of 3D videos. In more complex workflows, it can be used to seed subsequent heterogeneous refinement (especially when discrete heterogeneity is present) by creating versions of the molecule at different conformational positions, according to the family of structures solved. It can also be used to determine the direction and amount of continuous flexibility in a molecule, and thereby help in the selection of masked regions for local refinements. Finally, it can directly provide information about the energy landscape of the molecule, by inspecting the distribution of reaction coordinates among particles. Note that 3DVA was released as BETA in CryoSPARC v2.9, and is a work in progress - stay tuned for further updates and improvements! ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#publication) Publication To read more about 3D Variability Analysis, please see: Punjani, A. and Fleet, D.J. 3D Variability Analysis: Resolving continuous flexibility and discrete heterogeneity from single particle cryo-EM images. Journal of Structural Biology, Volume 213, Issue 2, 2021. [https://doi.org/10.1016/j.jsb.2021.107702](https://doi.org/10.1016/j.jsb.2021.107702) ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#example-tri-snrnp-empiar-10073) Example: Tri-snRNP ([EMPIAR-10073](http://www.ebi.ac.uk/pdbe/emdb/empiar/entry/10073/) ) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNutc6u7JNTTwscNy9b%252F-MNuuXU0nVxK3WKUqJSt%252F3dva_p1_1_gif.gif%3Falt%3Dmedia%26token%3D6e7b5cc8-09cd-4be3-a8e7-c8381f7a2acf&width=768&dpr=3&quality=100&sign=ec576df8&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNutc6u7JNTTwscNy9b%252F-MNuu_sIigQAec1CWqfn%252F3dva_p1_2_gif.gif%3Falt%3Dmedia%26token%3De10a25b6-bf15-4934-ad2a-c48a3565d2bd&width=768&dpr=3&quality=100&sign=99f075ee&sv=2) [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#background) Background ----------------------------------------------------------------------------------------------------------------------------------------------- 3D variability analysis is based on a widely explored idea: to try and compute the _eigenvectors of the 3D covariance_ of a set of particle images. This is also known as principal Components Analysis (PCA). A "principal component" is the same as an "eigenvector of the covariance". Eigenvectors of the 3D covariance are, literally, linear directions in the space of 3D volumes in which there is significant variability within the data set. They can be thought of as "trajectories" in the space of 3D structures along which the molecule exhibits conformational variability. Moving along a particular eigenvector yields all the different 3D conformations of a molecule that are captured by that particular eigenvector. Multiple orthogonal eigenvectors can be solved simultaneously, each corresponding to a different type of variability. Together they yield a multi-dimensional subspace that defines the _family_ of conformations in the dataset. In CryoSPARC, we have developed and implemented an algorithm and new fast GPU implementation for computing high-resolution 3D eigenvectors of the covariance, using particles from all viewing directions simultaneously, while also accurately correcting for the CTF. The algorithm can be used by running the new `3D Variability` job in cryoSPARC v2.9+. The algorithm, as mentioned above, solves for the eigenvectors as well as the position of each particle along each eigenvector - referred to here as reaction coordinates. The number of eigenvectors to solve, _K_, is a user specified parameter. Each eigenvector itself is a 3D volume, containing positive and negative values at each voxel. These describe the specific change that is happening to the molecule along the corresponding dimension of variability. Below is an example of what three orthogonal slices of a single typical eigenvector may look like: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNurbg84VKm82VpIxs3%252F-MNusmImNGYFoE-A_inS%252F3dva_p1_33dva-1.png%3Falt%3Dmedia%26token%3D2a1903f4-b4e4-447e-8685-81a89d41221d&width=768&dpr=3&quality=100&sign=935f33fc&sv=2) Reaction coordinates are a set of numbers, for each particle. The number of reaction coordinates is equal to the number of eigenvectors that are solved. The distribution of reaction coordinates across particles in a dataset gives a picture of the energy landscape of the molecule, by showing how many particles in the population take on a specific position in the conformational space spanned by the eigenvectors. Below is a sample reaction coordinate plot for the EMPIAR-10028 80s ribosome: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNurbg84VKm82VpIxs3%252F-MNuspcDAtFTnTOyjHNM%252F3dva_p1_4_3dva-2.png%3Falt%3Dmedia%26token%3D49d7abae-dac1-4e5f-b657-6341a0970c69&width=768&dpr=3&quality=100&sign=bc2d870&sv=2) The 0, 1, and 2 axes correspond to the three different components (eigenvectors) that were solved using 3DVA. The left plot shows coordinates for 0 vs 1, and the right plot for 1 vs 2. There is a small "bimodal" split along coordinate 0 (which by inspecting the eigenvector, is determined to correspond to ratcheting of the small subunit), while 1 and 2 correspond to a more continuous flexing of the molecule. In the case of discrete heterogeneity, we expect to find multiple "clusters" in reaction coordinate space. Each cluster corresponds to a discrete conformation of the molecule, where there is more probability of finding particles. The spaces between clusters may be either non-physical intermediate states that are never occupied (eg. if the clusters represent conformations where a subunit associates/dissociates, the in-between state of a half-density subunit is not realistic) or they may represent intermediate states that are occupied with very low probability (eg. if the clusters represent the opening and closing of a channel, the in-between state of a half-open channel may by physically realistic, but almost never observed). It is important to note that 3D variability analysis starts with fixed alignments of each particle, which are estimated using a standard refinement. As such, 3DVA assumes that the conformations present in a dataset are not so different that a refinement of particles against a consensus structure would yield grossly incorrect alignments. In other words, 3DVA will work well for finding discrete heterogeneity only when the different conformations present have enough shared density structure that particles from any conformation can be reliably aligned to an average structure of all the conformations. This is generally the case for conformational changes induced by eg. ligand binding, opening/closing of channels, or association/dissociation of subunits. It is generally _not_ the case when there are multiple entirely distinct molecules in the sample - for this case, it is recommended to start with a heterogeneous _ab-initio_ reconstruction to separate grossly different conformations. On the other hand, in the case of non-discrete or continuous flexibility, we expect to find a single cluster with significant variability in one or more dimensions. Those components of variability will correspond to the flexibility itself, and particles along the corresponding dimension of the reaction coordinates are in varying states of "flex". The flexibility itself can be visualized as a "movie" of volumes along the variability dimension. In this case, intermediate states all have significant probability and are physically realistic. It is important to note that in this case, the eigenvectors of the covariance, representing linear transformations of density, can not perfectly represent motion. They are, however, a good approximation to motion at resolutions that are approximately equal to the extent of motion. This means that for example, the motion of an alpha helix moving by 5Å can be well approximated by an eigenvector that is limited in resolution to 5Å. Likewise, the motion of a subunit by 40Å is well approximated by an eigenvector limited to 40Å resolution. This subtle point will be discussed further in the second part of this tutorial series. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#usage) Usage ------------------------------------------------------------------------------------------------------------------------------------- We recommend using the `3D Variability` job when you suspect or are interested in investigating conformational variability in your dataset. Through this tutorial and the example results below, we can conclude that many molecules analyzed through single particle cryoEM contain at least some observable 3D variability. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#consensus-refinement) Consensus Refinement 1. If commencing with a new particle stack, try and clean up the data set as much as possible using `2D Classification`. The goal is to remove obvious "junk" particles that may later affect the quality of reconstruction and refinement, but not to try and separate conformations. 2. Using the cleaned particle stack, run a multi-class `Ab-initio Reconstruction` (i.e., `Number of Ab-initio classes` ≥ 2) in order to find and exclude grossly different structures that may be present in the data set (e.g., if you have two completely different molecules present). You may also observe some heterogeneity that is conformational variability of a single molecule (e.g,. a subunit moving/flexing or portions dissociating, etc.) - this is fine. 3. Combine all particles that have some amount of protein density in common (i.e. all conformational states) and perform a `Homogeneous Refinement`. **NB.** For now, you will need to set `Refinement box size (Voxels)` to the full resolution/box size of the particles. If this is very large, it is better to use the `Downsample Particles` job first to downsample the particles, do the consensus refinement again, and then apply 3D variability analysis. During the refinement, optionally enter a value to enforce `Symmetry` (e.g., C4) if known. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#id-3d-variability-job) 3D Variability Job 1. Create a new `3D Variability` job and connect both the **particles** and the **mask** from the previous refinement. 1. Alternatively, if only interested in conformational variability of a specific region of the protein, import a different mask and connect this one to the `3D Variability` job to perform "focused variability". **NB.** For membrane proteins it is particularly helpful to mask out the micelle because otherwise, almost all the observable variability will be of the micelle and not the encapsulated molecule. Similarly, for very flexible proteins, it is helpful to further dilate the mask from a refinement to ensure that the mask is wide enough to contain all the flexed versions of the molecule. This can be done using the Volume tools job in CryoSPARC. 1. `Number of modes to solve`: This is the number of components/eigenvectors that will be solved for. The default value is 3, and the maximum is limited by GPU memory and the box size. Generally more modes will take (linearly) more time to solve. One subtle but important point is that the algorithm will generally solve these in order of importance, so asking for more modes will not reduce the quality of the first, most important modes of variability. **NB.** in v2.9 there is a known bug where sometimes, due to numerical instability, using more modes can result in "streaking" and other artefacts in the results. We are working on fixing this. 2. `Filter resolution (Å)`: This is the resolution at which eigenvectors are filtered during optimization. We recommend setting this to a value lower than the resolution of the consensus refinement performed previously (e.g., you may set it to 5A for a 3A refinement, but it can be experimented with). Using too high of a resolution will allow noise to dominate the determination of the eigenvectors. **NB.** Currently, `3D Variability` (as well as the preceding refinement) must be run at the full resolution/box size of the particles. You can downsample particles before processing, or else downsample the results before downloading outputs, in the next step. Also, you cannot currently enforce symmetry during this job. 3. Run the job. It should take 30 mins to 1 hour for 3 components and 100K particles on 1 GPU. 2. In the meantime, create a `3D Variability Display` job, connecting the particles and volumes from the `3D Variability` job. 1. `Output mode`: There are several output modes that can be used for different purposes. **NB.** in v2.9.0 only the simple mode is exposed as the others are experimental. **Simple mode:** This will output a simple linear "movie" of volumes along each eigenvector. You can control the number of frames, and the starting/ending position as a percentile of the reaction coordinate distribution. The output will be a set of "volume series" which are essentially just a set of numbered `.mrc` files that can be opened with Chimera (see below). 2. `Number of frames`: How many frames (per dimension) or clusters to output. 3. `Downsample to box size`: Downsample the volumes before writing, to save disk space. 4. `Crop to size (after downsample)`: Crop the volumes after downsampling, to save disk space. 5. `Only use these components`: Select a subset of variability components to use for visualization or clustering. This can be entered as a comma-separated, 0-indexed list (e.g., '0,2,3'). Leave as None to use all the components computed by the 3DVA job. 3. Once completed, `3D Variability Display` will output a series of frames for each component of the variability detected in the dataset. The outputs can be used to either download and view as movies in Chimera, or can be used to provide initial models and particle subsets for further classification and refinement. See the [Visualization](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#visualization) section below for detailed instructions about how to display the results in Chimera. Also see the second part of this tutorial series for more advanced usage and interpretation. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#example-results) Example Results --------------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#id-80s-ribosome-empiar-10028) 80s Ribosome ([EMPIAR-10028](http://www.ebi.ac.uk/pdbe/emdb/empiar/entry/10028/) ) The 80S Ribosome dataset contains 100K particles. It is a perfect example of a molecule that has many types of variability that are simultaneously occurring in the dataset. The results here were obtained by first performing a consensus refinement of the dataset reaching 3Å resolution. Next, the particles and mask were used in a 3D variability job with 5 modes. This yielded 5 different eigenvectors and the reaction coordinates of particles along each one: The figure below shows a projection of each of the 5 modes of variability (eigenvectors) that were solved. Each one contains both black (negative) and white (positive) regions that correspond with density changes. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNurbg84VKm82VpIxs3%252F-MNut1J2JVnJwpIK4CMD%252FEDva_p1_53dva-3.png%3Falt%3Dmedia%26token%3Dd6471bd6-41eb-4acc-a315-a3847e3ce3bc&width=768&dpr=3&quality=100&sign=38d69e2c&sv=2) The figure below shows the reaction coordinate distribution of particles, as scatter plots between adjacent pairs of components (0 vs 1, 1 vs 2, 2 vs 3, etc). Clearly, the first component (0) has the most variability. This corresponds to the ratcheting motion of the small subunit. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNurbg84VKm82VpIxs3%252F-MNut5VNbzSNYOjWDX34%252F3dva_p1_6_3dva-4.png%3Falt%3Dmedia%26token%3D094ad2c8-4886-467c-94b1-ddf102f1d372&width=768&dpr=3&quality=100&sign=cd17abb2&sv=2) The following videos show, from the same view, the first two eigenvectors of variability in the 80S ribosome dataset. These correspond with ratcheting of the small subunit, and rotation of the head region of the small subunit in a transverse direction. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNutc6u7JNTTwscNy9b%252F-MNuufikfnRfNY2JThbN%252F3dva_p1_7_gif.gif%3Falt%3Dmedia%26token%3De3192ad2-9a16-4c9b-8bdd-dd2ffd4ce4f1&width=768&dpr=3&quality=100&sign=b855dbeb&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNutc6u7JNTTwscNy9b%252F-MNuujKIkeqS6Sj1bieV%252F3dva_p1_8_gif.gif%3Falt%3Dmedia%26token%3D8105c822-7494-46ac-bc68-8f61b906b4ef&width=768&dpr=3&quality=100&sign=e16f03e3&sv=2) ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#nav-1.7-channel-empiar-10261) Nav 1.7 Channel ([EMPIAR-10261](http://www.ebi.ac.uk/pdbe/emdb/empiar/entry/10261/) ) The Nav1.7 channel is an example of a dataset where the first component of variability (pictured below) is a flexible conformational change. Notably, the resolution of this result is high, and the motion of individual helices can be seen at a resolution where side-chains and helical pitch is identifiable. Note that this result was computed using particles from an already-classified subset of the data, containing only the active state of the channel. This was done to emphasize the flexible degrees of variability rather than the discrete channel opening-closing variability. Note also that this result was computed using a mask that excludes the micelle. 3 modes were solved in this case. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNutc6u7JNTTwscNy9b%252F-MNuun3ephFzJYnkkDts%252F3dva_p1_9_gif.gif%3Falt%3Dmedia%26token%3D65480575-7124-42ff-9120-1ec2ca21176d&width=768&dpr=3&quality=100&sign=a109a935&sv=2) ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#t20s-proteasome-empiar-10025) T20S Proteasome ([EMPIAR-10025](http://www.ebi.ac.uk/pdbe/emdb/empiar/entry/10025/) ) Some molecules that have always been assumed to be quite rigid, like the T20S Proteasome, actually do have flexible variability as well! The video below shows the first mode of variability of the proteasome data, computed from 50K particles. In this mode, there appears to be a slight lengthening and twisting of the proteasome molecule. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNutc6u7JNTTwscNy9b%252F-MNuuqNgZc4SxM5NErJS%252F3dva_p1_10_gif.gif%3Falt%3Dmedia%26token%3D484ecca0-1ab1-4d06-9e5b-2bd5b87cbcea&width=768&dpr=3&quality=100&sign=af394965&sv=2) [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#visualization) Visualization ----------------------------------------------------------------------------------------------------------------------------------------------------- 1. Download each of the volume series from the `3D Variability Display` job (`.zip` files). Unzip the files and open the volume series in [UCSF Chimera](https://www.cgl.ucsf.edu/chimera/) . 1. Tools → Volume Data → Volume Series. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNurbg84VKm82VpIxs3%252F-MNutBpsLUfPdVbKHcXk%252F3dva_p1_11_.png%3Falt%3Dmedia%26token%3Dda1cef13-e358-49dc-84c9-f584cd3b8dc6&width=768&dpr=3&quality=100&sign=dacbb6e2&sv=2) 2\. In the volume series tool, click "Open..." and multi-select the series of `frame_XXX.mrc` files from a single component. This will open them all together as a series. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNurbg84VKm82VpIxs3%252F-MNutFNBB3U934pn0inb%252F3dva_p1_12_volume-series-2.png%3Falt%3Dmedia%26token%3D76835ed0-ed0c-463e-a436-d092437a1732&width=768&dpr=3&quality=100&sign=47bdda16&sv=2) 3\. Now in the Volume Series tool there are some options for displaying. Use these options: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNurbg84VKm82VpIxs3%252F-MNutJ466cBAIUIWDsEq%252F3dva_p1_13_volume-series-3.png%3Falt%3Dmedia%26token%3D8acc58ed-5d77-402b-a950-cd5c006a9e56&width=768&dpr=3&quality=100&sign=24c9ee97&sv=2) * When you set the Data cache size, you have to press "Enter" after typing 1024.0 in the box, then check if it worked by clicking "Current use" to see how much space is allocated for cache. * Make sure the number of cached renderings is at least as many as the number of frames (20 by default). * Oscillate mode will make the frames loop back and forth, best way to show the motion. * The "Normalize threshold levels" option is nice because it adjusts the threshold for each frame so that the total volume enclosed by the mesh from each frame is the same. * Once you have these settings set, adjust the step and threhsold in the volume viewer so that the initial frame looks good: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNurbg84VKm82VpIxs3%252F-MNutMxaSq4zw3qVsHFZ%252F3dva_p1_14_volume-series-4.png%3Falt%3Dmedia%26token%3Dac1b5c29-0112-469b-bd77-6b1444448a77&width=768&dpr=3&quality=100&sign=f57c9655&sv=2) * Then you can press "Play". The frames will play slowly at first, but after the first pass through they will speed up a lot because the 3D mesh is cached. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#optional-record-movie) Optional: Record Movie We recommend the following settings: `set bgColor white` and a colour of `#70d470d4b333` ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNurbg84VKm82VpIxs3%252F-MNutSHM22omKSrRkjH6%252F3dva_p1_15_chimera-rendering-1.png%3Falt%3Dmedia%26token%3D3198429c-f04a-422c-8a6d-321a12e86305&width=768&dpr=3&quality=100&sign=6c6b7cc0&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNurbg84VKm82VpIxs3%252F-MNutVKaijLCzC01A4Xw%252F3dva_p1_16_chimera-rendering-2.png%3Falt%3Dmedia%26token%3D0553cdf0-9c06-45e5-9ef5-a8b03ef86912&width=768&dpr=3&quality=100&sign=96e4abcd&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNurbg84VKm82VpIxs3%252F-MNutYJIBRvwFX4zI6Vi%252F3dva_p1_17_chimera-rendering-3.png%3Falt%3Dmedia%26token%3Dc86d98ed-e068-446d-9650-62b44da75a15&width=768&dpr=3&quality=100&sign=f7223aab&sv=2) To record a movie use these commands: [PreviousTutorial: 3D Classification](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification) [NextTutorial: 3D Variability Analysis (Part Two)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-two) Last updated 3 years ago * [3D Variability Analysis Tutorial: Part One](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#id-3d-variability-analysis-tutorial-part-one) * [Introduction](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#introduction) * [Publication](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#publication) * [Example: Tri-snRNP (EMPIAR-10073)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#example-tri-snrnp-empiar-10073) * [Background](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#background) * [Usage](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#usage) * [Consensus Refinement](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#consensus-refinement) * [3D Variability Job](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#id-3d-variability-job) * [Example Results](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#example-results) * [80s Ribosome (EMPIAR-10028)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#id-80s-ribosome-empiar-10028) * [Nav 1.7 Channel (EMPIAR-10261)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#nav-1.7-channel-empiar-10261) * [T20S Proteasome (EMPIAR-10025)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#t20s-proteasome-empiar-10025) * [Visualization](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#visualization) * [Optional: Record Movie](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one#optional-record-movie) Copy windowsize 1200 1200 vseries open *.mrc vol #0 step 1 movie record vseries play #0 direction oscillate loop false normalize true cacheFrames 30 wait 77 vseries stop #0 movie stop movie encode ~/Desktop/3DVA/result.mpg quality higher --- # Tutorial: Helical Processing using EMPIAR-10031 (MAVS) | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs.md) . This page will focus on the application of the various tools for helical processing, to the [EMPIAR-10031](https://www.ebi.ac.uk/pdbe/emdb/empiar/entry/10031/) dataset (Mitochondrial antiviral signalling (MAVS) filaments). Here, we will cover the workflow from particle picking to reconstruction and refinement. Prior to following this case study, it is recommended to read the page detailing [helical symmetry in CryoSPARC](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/helical-symmetry-in-cryosparc) , for more information on how helical symmetry is treated during reconstruction. If you are new to CryoSPARC, it is also strongly recommended to complete the [T20S Proteasome tutorial](https://cryosparc.com/docs/tutorials/t20s) before following along with this case study. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#preprocessing) Preprocessing ---------------------------------------------------------------------------------------------------------------------------------------- Before we pick particles, we must import the raw movies, motion correct the movies, and perform CTF estimation. This can be done using any of the motion correction and CTF estimation jobs within CryoSPARC. For this dataset, we use the following preprocessing steps: * **Import Movies** job with pixel size of 1.05 Å, accelerating voltage of 300 kV, spherical aberration of 0.01 mm, and exposure dose of 35 electrons per square Angstrom (parameters from the [publication](https://elifesciences.org/articles/07546) ) * **Patch Motion Correction (multi)** with maximum alignment resolution of 3 Å * **Patch CTF Estimation (multi)** with minimum search defocus of 25000 Å (parameters from the [EMDB entry](https://pdbj.org/emnavi/quick.php?id=emdb-6428) ) For this case study, we also use the **Manually Curate Exposures** job to remove exposures with CTF fit scores under 6 Å. After these steps have been done, we are ready to move on to particle picking. Below is the workflow tree up until this point. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNenkTpvzhjYeCSldTN%252F-MNenqRzoCTJlIJHgqmp%252F2_MAVS_Screen%2520Shot%25202020-11-11%2520at%25202.59.34%2520PM.png%3Falt%3Dmedia%26token%3D44432d4c-cdc7-494f-aa42-63fbbb55b341&width=768&dpr=3&quality=100&sign=a9d7a460&sv=2) Preprocessing workflow tree. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#particle-picking) Particle Picking ---------------------------------------------------------------------------------------------------------------------------------------------- There are three main methods in CryoSPARC for particle picking on helical datasets. One can choose to first manual pick a subset of micrographs, and generate templates from 2D classification. Using templates, one can do either template-based [**filament tracing**](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-picking/job-filament-tracer-beta) , or standard **template picking**. Alternatively, one can avoid manual picking by launching the filament tracer without providing templates, and enable _template-free tracing_ by setting the minimum and maximum filament diameter parameters. Finally, one can also use any of the [**deep picker**](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/deep-picking) jobs available, including [**Topaz**](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/deep-picking/topaz) . For more details on these picking options, please refer to the linked job pages. Note that if you use pickers other than the filament tracer or template picker, you may need to set certain additional parameters such as the number of times to apply helical symmetry. This detailed in the [helical refinement](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-helical-refinement-beta) job page. Relative to the template or blob pickers, the advantages of the filament tracer are that it allows the specification of a fixed inter-box distance, the detection of individual filaments, and the rejection of filaments that are too highly bent. The disadvantages are that the filament tracer assumes that all filaments are roughly cylindrical (at least at low resolution), and depends on two extra hysteresis thresholding parameters. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#manual-picking) Manual Picking For this case study, we will manually pick a subset of micrographs, generate templates, and then use the [template based filament tracer](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-picking/job-filament-tracer-beta) . To start, launch a **Manual Picker** job and manually pick around 150 - 200 particles from at least 10 different micrographs, with varying defoci. Manual picking for filaments works the same as for globular proteins, and you can pick overlapping segments of the filament. Try to avoid areas with crowding, intersection points, and picks near the edge of the micrograph.At this stage, the distance between particle picks does not matter, as we will later specify a constant inter-box distance between picks during filament tracing. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNenkTpvzhjYeCSldTN%252F-MNenx65fkdgPjOzvr7s%252F3_MAVS_P108_J2115_extracted_coordinates_on_j748motioncorrected8229739760171009629_falcon_2014_11_05_16_27_32_0_patch_aligned_doseweightedmrc.jpg%3Falt%3Dmedia%26token%3D1f703f38-b780-4a46-ad78-d288df09e120&width=768&dpr=3&quality=100&sign=b7cc047a&sv=2) Example of manual picks on one micrograph. Once the manual picking is complete, launch a **2D classification** job and connect the manually picked particles to the 2D classification job. Set: * `Number of 2D classes` to 5, and * `Force Max over poses/shifts` to true. We don't need many classes as all views are very similar, and the micrographs are too noisy to produce high resolution templates. After the 2D classification is complete, launch a **Select 2D** job to select all of the good classes that show a clear bright filament against a dark background, with as little noise as possible. Here, we selected three of the five classes to serve as templates for filament tracing. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNenkTpvzhjYeCSldTN%252F-MNeo2K4hZGelmfqYkog%252F4_MAVS_P108_J2118_selected_3_classes.png%3Falt%3Dmedia%26token%3D6c49d64e-a61a-46cf-b2b0-499f5b6fe3e3&width=768&dpr=3&quality=100&sign=82905f03&sv=2) Selected classes for initial template-based filament tracing. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#filament-tracing) Filament Tracing Next, we build a [Filament Tracer (BETA)](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-picking/job-filament-tracer-beta) job. To this job, we connect the templates from the previous Select 2D job, and the micrographs from the Manually Curate Exposure job. The filament tracer needs two parameters to be set: * `Filament diameter (A)`: The estimated diameter of the filament, in Angstroms * `Separation distance between segments (diameters)`: The distance between adjacent picks along a filament, in terms of multiples of the filament diameter We choose a **diameter of 90 Å**, and a **separation distance of 0.25** (corresponding to 90/4 = 22.5 Angstroms). Note that there are also various advanced parameters in the filament tracer, giving finer control over the ridge detection filter and the thresholding parameters. For more information and tips on adjusting these parameters, refer to the [Filament Tracer (BETA)](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-picking/job-filament-tracer-beta) job page. For this dataset, we can leave all parameters as defaults, and launch the job. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#inspect-picks) Inspect Picks Once the filament tracer is complete, we should inspect filament picks to remove contaminant picks and highly bent filaments. Often, removing highly bent filaments can make a significant difference in the reconstruction quality. In addition to the normalized cross-correlation (NCC) score and the local power, the filament tracer tracks two dataset fields that measure how bent a filament is: * `curvature`: This is the estimated curvature (1/Å) of the filament at the pick location. Local curvature is useful to prune out the most bent locations along a filament, without removing all picks from that filament * `sinuosity`: This is the ratio between the actual filament contour length, and the straight line start-to-end distance. Filament sinuosity is useful to remove entire filaments that may correspond to contaminants or aggregated filaments Launch an **Inspect Picks** job, and connect the outputs of the filament tracer to it. In the Inspect Picks job, you will see sliders for NCC score and local power (as usual), in addition to the **local curvature** and **filament sinuosity** sliders and their accompanying histograms. In addition to the curvature and sinuosity thresholds, adjusting the **power threshold** is often very useful to filter out contaminants, or ice/carbon picks. For this dataset, we stringently remove a large portion of curved picks, as well as picks with too high power score. The specific thresholds we changed are listed below, however with different templates, ideal values will likely differ. * `Local Power`: under 1296550 * `Curvature`: under 0.0004 * `Sinuosity`: under 1.08 ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNenkTpvzhjYeCSldTN%252F-MNeo9alZ-NGm8SqJkv5%252F5_MAVS_Screen%2520Shot%25202020-11-11%2520at%25204.47.50%2520PM.png%3Falt%3Dmedia%26token%3Da0bbdff8-53a1-47d0-bf00-a383d9e139c3&width=768&dpr=3&quality=100&sign=4fc9a248&sv=2) Screenshot of the inspect picks interface with the additional filament sliders. Now, we can click "Done Picking! Output Locations" to complete the job. The particle picking workflow up until this point is shown in the tree below. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNenkTpvzhjYeCSldTN%252F-MNeoGhz5iLKpKsHgeye%252F6_MAVS_helical_tree_1.png%3Falt%3Dmedia%26token%3D26b78a19-fdd2-467b-ba56-07fe2d7dcb08&width=768&dpr=3&quality=100&sign=2ec5f337&sv=2) Workflow tree from exposure curation to filament tracing and inspect picks. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#extraction) Extraction The final step we need to complete is extraction. Build an **Extract From Micrographs** job, and connect the outputs of the Inspect Picks job to it. Here, we use a **box size of 300**. Once the job is complete, we can move onto 2D classification. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#id-2d-classification) 2D Classification --------------------------------------------------------------------------------------------------------------------------------------------------- **2D classification** remains largely the same for helical proteins as with globular proteins. The main change in 2D classification is the vertical alignment of filament classes, which helps when visually comparing classes to each other during Select 2D. Note that vertical alignment is not enabled by default, and must be activated by turning on the `Align filament classes vertically` parameter in any 2D classification job. For this dataset, we run a 2D classification from defaults and adjust the following parameters: * `Number of 2D classes`: 100 * `Align filament classes vertically`: True * `Remove duplicate particles`: False * `Batchsize per class`: 400 (since these filaments are rather small) Note that `Remove duplicate particles` in 2D Classification is activated by default for globular proteins. Since particle picks for filaments are often intentionally very dense, this parameter should generally be _deactivated_ for processing of filaments. [Helical Refinement](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-helical-refinement-beta) can account for the proximity of adjacent picks by constructing gold-standard splits that prevents overlapping particles from being randomized to different half-sets. Poor classes are those that include filament crossings, breaks, or end points, or otherwise have little or no high resolution detail. Good classes are those are straight and have high resolution detail, all the way out to the box edges (near where the filament touches the circular window). With this dataset, we obtained 21 good classes and 79 poor classes, with a total yield of 88,033 particles. A subset of the good and poor classes are shown in the image below. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNenkTpvzhjYeCSldTN%252F-MNeoL0n4cO3UQ4NMXIB%252F7_MAVS_P108_J2122_2d_classes_for_iteration_20.png%3Falt%3Dmedia%26token%3D952581a1-92fa-4666-81c7-90b9aaa26258&width=768&dpr=3&quality=100&sign=8b9cc3cc&sv=2) A subset of 5 good classes (top) and 5 poor classes (bottom). It should be noted that using these high quality 2D classes, the results from filament tracing can often by significantly improved by feeding the good classes back into the filament tracer, and running inspect picks, extract from micrographs, and 2D classification one more time. In particular, low SNR micrographs and small filaments benefit the most from the use of higher quality templates during picking. From here, we can either refine the model with imposed symmetry, or we can attempt to reconstruct without imposing any symmetry. Note that many helical datasets are not amenable to reconstruction without symmetry estimates, and it's currently an open research problem to characterize the symmetry of helical assemblies without prior knowledge. Thus, if symmetry estimates are already present for your dataset, you can skip down to the [symmetric helical refinement](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#symmetric-helical-refinement) section. On the other hand, if you do not have initial symmetry estimates for your dataset, you can attempt a similar approach to the one presented in the [asymmetric helical refinement](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#asymmetric-helical-refinement) and [symmetry search](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#symmetry-search) sections. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#helical-refinement) Helical Refinement -------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#asymmetric-helical-refinement) Asymmetric Helical Refinement Next, we will reconstruct the filament from the particles without applying symmetry. To do this, build a **Helical Refinement (BETA)** job, and connect the input particles from the select 2D job. The helical refinement job will use the standard maximum likelihood optimization done during any standard refinement, and it will not apply helical symmetry unless you provide initial estimates for the helical twist and rise. Note: In some datasets, prior estimates of the helical symmetry parameters are necessary to obtain a correct structure. These can often be informed from prior similar structures, or [Fourier-Bessel indexing](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4944209/) of high quality 2D class averages. In these challenging datasets, asymmetric refinements (or refinements with incorrect symmetry imposed) may result in incorrect structures, and thus the outputs from a symmetry search utility job will not be useful. For generating power spectra that may be used for Fourier-space symmetry determination methods, refer to the [Average Power Spectra](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-average-power-spectra) job in CryoSPARC v4.0+. The tools presented here do not circumvent this issue for all datasets. Regardless, we hope that these tools can be useful for many datasets in the exploratory phase of data processing, as well as the high-resolution refinement stage. The helical refinement job also doesn't require an input volume. If a volume isn't provided, it will generate an initial density using the `filament` in-plane rotation estimates, which are written to either during filament tracing or 2D classification. Note that this generates a "cylindrical" density from the input particles – for filaments with highly oblong cross sections (e.g. amyloid filaments), **Ab-Initio Reconstruction** may produce a better initial model. We have generally seen that for filaments that are approximately cylindrical, and have constant diameter, directly running a helical refinement from the particles is more successful than running an ab-initio reconstruction job. Conversely, for filaments that are distinctly _not_ cylindrical, ab-initio reconstruction can take advantage of the diversity of views along the helical axis, and often results in a better initial model. For this dataset, we leave all parameters as default and we run the asymmetric helical refinement. Below shows the slice plots from the final iteration. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNenkTpvzhjYeCSldTN%252F-MNeoPsVnjrRUeB5Lc70%252F8_MAVS_P108_J2125_real_space_slices_iteration_012.png%3Falt%3Dmedia%26token%3D2efc026b-bdda-494c-a49d-59c046188442&width=768&dpr=3&quality=100&sign=21b9c0be&sv=2) Slice plots from the final iteration of asymmetric helical refinement. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#symmetry-search) Symmetry Search Next, we can use the **symmetry search utility (BETA)** job to take a look at the symmetry that is present in the reconstruction. Before running this job, ensure you have read [the page](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/helical-symmetry-in-cryosparc) for more information on how helical symmetry is treated during reconstruction, as well as the job information for the [symmetry search utility](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-symmetry-search-utility-beta) . Since the reconstruction was done without applying symmetry, analyzing how well the volume fits different symmetry parameters can give us an "unbiased" look into the symmetry present in the dataset. To start, build a symmetry search utility job, and connect the volume and mask from the asymmetric helical refinement. We also must tell the job over what parameter ranges it should search. We can either define a 2D grid of rise/twist values, or a 2D grid of pitch/number-of-subunit values. Here, we choose the latter option, with pitch values between 5 Å and 50 Å, and number of subunits per turn ranging from 3 and 8. In many cases, the best search ranges can be obtained by manually inspecting the asymmetric reconstruction in UCSF Chimera, and looking for clear signs of helical symmetry. Once complete, this job will output various plots and tables that display the mean squared error (MSE) associated with all of the different symmetry parameters that were tested. By default, it will search over both right and left handed helical symmetry parameters, where checkpoint 1 shows MSE values over right handed parameters, and checkpoint 2 shows MSE values left handed parameters. For each hand searched over, the job will produce: * 2D plots of the error surface, with the global optimum highlighted by intersecting horizontal/vertical gray dashed lines; * 1D plots of the error, evaluated along the horizontal and vertical dashed lines; * A table with the first 20 local minima in the MSE surface, listed in order of increasing error Below is the 2D error surface plots for this asymmetric helical refinement, one for left-handed parameters and the other for right-handed parameters. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNenkTpvzhjYeCSldTN%252F-MNeo_EQ2zW1_ChdtYz7%252F9_MAVS_mavs_asym_plots.png%3Falt%3Dmedia%26token%3D228065d6-0a5d-4857-bcfd-2fea24c8cde5&width=768&dpr=3&quality=100&sign=8799e34&sv=2) 2D error surface plots for asymmetric refinement of the MAVS dataset. If the symmetry is indeed discernible from the input volume, the global minima of MSE values should provide the best estimate of symmetry parameters. In some cases, the correct symmetry parameters may only be local optima of the MSE surface. For this reason, the job prints out a table of local minima, which are also produced as outputs of the job in the `symmetry_candidates.cs` file. For the MAVS dataset, asymmetric refinements tend to give accurate estimates for initial symmetry parameters. Across both plots, the global minima is at a MSE value of ~1217 with right handed symmetry parameters printed below. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#symmetric-helical-refinement) Symmetric Helical Refinement When estimates for symmetry parameters are known, it is always recommended to run a helical refinement with the estimated twist (º) and rise (Å) as parameters to the job. This is because enforcing symmetry has two important roles: * boosting the effective amount of signal in the dataset, and * imposing a strong structural constraint on the reconstruction The first point is important, as it can increase the resolution of the reconstruction. For the MAVS dataset, since we already picked particles with fairly small inter-box distance, each image is used up to 4 times by default. For helices with particularly small rises, such as the Tobacco Mosaic Virus (EMPIAR-10022) with helical rise of ~1.41 Å, each image can be used 20 or 30 times in reconstruction, providing a dramatic increase in resolution. The second point is especially important for helical reconstruction, as it is often necessary to prevent the refinement from falling into a "local maxima" in the likelihood landscape. We now build a helical refinement job. From the symmetry search utility, we plug in a twist of +101.436º, and a rise of 5.088 Å, as input to the initial symmetry parameters. We also connect the same set of particles into the refinement, and the volume from the asymmetric refinement. By default, optimization of the symmetry parameters will begin when the GSFSC cutoff resolution exceeds 5 Å, however, this threshold can be changed in the **Helical Symmetry Search** parameter section. Here, we leave it as default, and queue the job. When the refinement is complete, we should always download and inspect the map to ensure that secondary structure is apparent. If you have experience interpreting cryo-EM density maps, you may notice that the alpha helices present in the refined map are left handed! Generally, left handed alpha helices are unexpected, and indicates that the handedness of our overall reconstruction is inverted. This can happen because the handedness of a cryo-EM map is ambiguous – particles can equally well reconstruct a density map as they can with its mirror image. In helical reconstruction, this ambiguity is linked to the value of the **helical twist**. Specifically, inverting the sign of the helical twist is equivalent to flipping the hand of the reconstruction. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNenkTpvzhjYeCSldTN%252F-MNep1eKQJo1W7iN8WJp%252F10_MAVS_Screen%2520Shot%25202020-11-12%2520at%25205.16.20%2520PM.png%3Falt%3Dmedia%26token%3D24f71f58-c7b2-4f9f-aca5-14a9a0f98462&width=768&dpr=3&quality=100&sign=f6b30b50&sv=2) Left handed alpha helices. To correct this, we can build a **Volume Tools** job, connect the output of the helical refinement to the volume tools job, and set the **Flip Hand** parameter as true. In order to obtain a final reconstruction with the correct hand, as well as a final set of aligned particles, we may want to run one last helical refinement with the corrected hand. To do this, we can take the output volume from the volume tools job, and connect it as an input to one final helical refinement. Similarly, connect the particles from the original select 2D job to the helical refinement. Be sure to input the symmetry parameters with the same rise, but inverted twist (-101.436º). Finally, we can optionally activate the [Non-Uniform refinement](https://www.nature.com/articles/s41592-020-00990-8) switch, which will invoke an adaptive regularization process that often leads to higher map quality. Our final helical refinement (with non-uniform refinement enabled) reached a resolution of 3.6 Å. Below is the output sharpened and symmetrized map, `map_sym_sharp.mrc` The workflow, from particle extraction to symmetric refinement, is shown in the tree below. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNenkTpvzhjYeCSldTN%252F-MNepA6UzpUcnYHPdG4O%252F11_MAVS_helical_tree_2.png%3Falt%3Dmedia%26token%3D813fec1d-9e4c-466b-8cfc-e1f9291c4a40&width=768&dpr=3&quality=100&sign=d61a4a16&sv=2) Workflow tree from particle extraction and 2D classification to symmetric refinement. [PreviousTutorial: Blob Picker Tuner](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-blob-picker-tuner) [NextTutorial: Maximum Box Sizes for Refinement](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/performance-metrics) Last updated 3 years ago * [Preprocessing](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#preprocessing) * [Particle Picking](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#particle-picking) * [Manual Picking](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#manual-picking) * [Filament Tracing](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#filament-tracing) * [Inspect Picks](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#inspect-picks) * [Extraction](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#extraction) * [2D Classification](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#id-2d-classification) * [Helical Refinement](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#helical-refinement) * [Asymmetric Helical Refinement](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#asymmetric-helical-refinement) * [Symmetry Search](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#symmetry-search) * [Symmetric Helical Refinement](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs#symmetric-helical-refinement) Copy Showing the 20 best local minima. No. | p (A) | n | dz (A) | dphi (deg) | mse ------------------------------------------------------------------------- 00 | 018.059 | 003.549 | 005.088 | +101.436 | 1216.663 --- # Tutorial: 3D Flex Mesh Preparation | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation.md) . It can often be important to the success of 3D Flexible Refinement that a mesh with the proper topology is created prior to training. The mesh defines what types of motion are “allowable”. In this article, we first highlight the theoretical grounding for 3D Flex meshes and why custom meshes may be necessary in many cases. We then walk through the process of creating an example mesh. More detail is available in the original 3D Flexible Refinement publication ([Punjani and Fleet, 2023](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#references) ). The material covered in this page is also available in video form: [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#summary) Summary --------------------------------------------------------------------------------------------------------------------------------- 3D Flex Meshes are used to regularize the 3D Flex deformation model during training. Custom meshes are most useful when discontinuous motion, such as separation or sliding, are expected. When generating a custom mesh, special care must be taken when deciding on a fusion strategy to allow the model to capture the full range of target motion. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#id-3d-flex-mesh-overview) 3D Flex Mesh Overview ---------------------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#why-model-deformation-with-a-mesh) Why model deformation with a mesh? At its core, CryoSPARC’s 3D Flex is attempting to determine the conformation of a given single particle using just one image. The algorithm treats conformational change as movement of electron potential from one voxel to another. An algorithm to directly solve this problem by checking various mixtures of density in various voxels would be both slow and sensitive to overfitting. We have therefore implemented 3D Flex to instead model movement of the vertices of a mesh. This reduces the number of degrees of freedom in the model and imposes smoothness on the resulting flexibility because nearby regions of the map must move together. The reduced number of degrees of freedom also result in increased speed during training. To explain how the use of a deformation mesh achieves these goals, consider a theoretical 1D particle. This particle can stretch or squish to various degrees along its length. Rather than directly try to model the movement of intensity from one pixel to another, we instead create a mesh along the particle, like so. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FLHLR2c3psNBRomi7MrVU%252F1d-resting.png%3Falt%3Dmedia%26token%3Dad5d5d15-efc5-4c02-8821-ccf6230ae742&width=768&dpr=3&quality=100&sign=7e43bdb2&sv=2) Then, rather than applying a deformation to each pixel individually, we instead move just the points of the mesh. Everything between the mesh points is squished or stretched linearly as the points on either side move closer or further apart: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FJARdeXQvA4GRYi1BWco4%252F1d-expansion.png%3Falt%3Dmedia%26token%3D1c1efff4-44fa-45a9-8db5-581f62bc30c2&width=768&dpr=3&quality=100&sign=bf6d97a9&sv=2) The same principle can be applied in higher dimensions — the important point is that the algorithm is only modeling deformation of the _mesh_. The deformation of the underlying volume is, in turn, a function of the deformation of the mesh. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F6NwLpL9K9aHabWZVarxl%252Flogo-deform.png%3Falt%3Dmedia%26token%3Dbb48fe0f-ffa4-4128-a314-f73b10ad188b&width=768&dpr=3&quality=100&sign=91d87bfd&sv=2) Meshes are created with the [3D Flex Mesh Prep](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#mesh-generation) job. This job requires a consensus volume and a solvent mask. When you run this job, a number of tetrahedral cells (hereafter just “tetras”) will be created to span the entirety of the box. The resulting tetra mesh can then be used by a [3D Flex Training](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement#parameter-tuning) job to model the movement of the particles. It is during the training job that the tetra cell vertices are moved to model deformation — 3D Flex Mesh Prep jobs produces the vertices in their evenly-spaced “starting” positions. This is an important point — both vertices and tetras are important in 3D Flex. When a vertex moves, the tetra deforms. This deformation will move any voxels inside the tetra in a linear fashion. For a simpler example, consider again the two-dimensional case. We start with a consensus reconstruction, a solvent mask, and a set of “tetras” (triangles in 2D). When we move individual vertices along the right-hand side of our mesh, the rectangular particle is deformed linearly within each tetra. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Flkazo0knDF7lTHbiTKpe%252Fmesh-deformation-rectangle.png%3Falt%3Dmedia%26token%3D99fab1a1-62d0-4746-bd0e-84540e8d128b&width=768&dpr=3&quality=100&sign=908d987c&sv=2) Using this technique we can model most forms of **continuous** motion as movement of the vertices which make up the mesh (or, equivalently, the expansion and contraction of tetras). Note, however, that **moving a vertex deforms all adjacent tetras in the same way**. All of these tetras share their vertices, so they cannot move independently from each other. We thus cannot model discontinuous motion (i.e., domains which slide past each other or move apart from one another) using a single tetra mesh. We therefore must _segment_ the mesh into separate parts which can move independently. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#discontinuous-motion) Discontinuous Motion Consider the case in which we have two domains which “slip” past each other: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fm8hX5R4TgDwsRln8Bh8Z%252Fbase.png%3Falt%3Dmedia%26token%3Defb1f641-8925-48d5-ae4c-7605d3ea3f81&width=768&dpr=3&quality=100&sign=f0b5f303&sv=2) To model the movement of the purple arrow downward we could move the vertices enclosing it downward. Similarly, to move the orange arrow upward we could move the vertices surrounding the orange arrow upward. However, this creates a problem at the interface between the two arrows — the vertices there want to move downward and upward at the same time. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FeEwTVAkyXQ76kiGTp7rx%252Fsingle-mesh-undeformed.png%3Falt%3Dmedia%26token%3De558abf6-b208-45f5-83c3-9a21252e283e&width=768&dpr=3&quality=100&sign=90ca4603&sv=2) This is not possible with a single mesh — the vertex must move either up or down. Therefore, the sliding movement of these two arrows cannot be modeled with this mesh topology. To resolve this conflict, we instead create two overlapping meshes and assign each arrow to its respective mesh. These meshes _start_ with vertices and tetras in the same position, but there are now _multiple vertices and tetras_ where the meshes overlap. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FiwOjuCtXI95B2hWXu0Hn%252Fdual-mesh.png%3Falt%3Dmedia%26token%3Dbd1b7c86-d358-4218-912b-74af48e143ee&width=768&dpr=3&quality=100&sign=67b53100&sv=2) In this image, we have segmented our starting mesh into two distinct meshes, one for each arrow. As before, each mesh is defined by a set of vertices, indicated with blue circles. The vertices are annotated with a symbol corresponding to the mesh to which they belong. At the border between the two arrows, where the two meshes meet, there are two vertices which start in the same position. These vertices are thus annotated with both an up- and down-arrow symbol. This vertex duplication allows each mesh to deform the same region of space in its own direction, since it models that region with vertices and tetras which are independent of the other mesh. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F9c9Bunkiaxcqnj8HNzZt%252Fdual-mesh_deform.png%3Falt%3Dmedia%26token%3D19496893-a9ba-4e0f-b33d-710ee506b781&width=768&dpr=3&quality=100&sign=f8ebe010&sv=2) Note that the formerly-overlapping vertices are no longer in the same position. The orange mesh moved its vertices up and to the right, while the purple mesh moved its vertices down and to the left. We have successfully modeled sliding movement of the arrows, even though they are positioned in a region of space which fits inside a single tetra. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#when-is-a-custom-mesh-required) When is a custom mesh required? -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- As demonstrated above, domains which are expected to slide past each other or move apart from each other _require_ custom meshes. Other types of flexibility, or specific targets, may benefit from a custom mesh even in the absence of sliding or separating movements. For instance, proteins with a micelle may benefit from a custom mesh in which the micelle is segmented into a separate mesh which is then marked as rigid, so as not to waste latent space modeling uninteresting deformation of the micelle. In the remainder of this page, we walk through the process of segmenting and creating a mesh. How does CryoSPARC assign voxels to one or the other overlapping mesh?[](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#how-does-cryosparc-assign-voxels-to-one-or-the-other-overlapping-mesh) When you create a segmented 3D Flex mesh, two objects are actually created. The first is the mesh (or set of meshes), which itself comprises a **list of vertices** and the **connections between those vertices**. The connections are listed in out in sets of four, with each set therefore defining a tetrahedral cell. Each of these tetra cells are then assigned to a mesh. The second object is a **tetra mask**, which is a 3D volume the same size as the input volume used for segmentation and training. Each voxel of the tetra mask stores an integer value corresponding to the mesh to which that voxel “belongs”. During training and reconstruction, the density in a particular voxel is flowed along the vector described by the mesh indicated by its coordinate in latent space and the tetra to which it is mapped. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FrVSJESDcRowleQSWGVca%252Ftetra-mapping.png%3Falt%3Dmedia%26token%3Da999ea86-002f-4684-b4ff-69b4730ec3f5&width=768&dpr=3&quality=100&sign=4f954540&sv=2) In this way, tetra meshes can overlap while still only moving map density from “their own” voxels, and voxel-level segmentation is retained while the tetras themselves are significantly larger than a single voxel. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#creating-a-custom-mesh) Creating a custom mesh --------------------------------------------------------------------------------------------------------------------------------------------------------------- Creating a custom mesh from prepared data comprises three main steps: 1. Create the segmentation 2. Decide on a mesh fusion strategy 3. Run the 3D Flex Mesh Prep job This tutorial will walk through the details of each of these steps, working through the process of creating a custom 3D Flex Mesh for NaV 1.7 (EMPIAR 10261). The data used in this tutorial were originally collected and processed by [Xu and colleagues (2019)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#references) . ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#creating-a-segmentation) Creating a segmentation 3D Flex Mesh Prep jobs need a way to assign each voxel in the consensus map to a particular tetrahedron. In the default case, there is only one tetra segment which covers the entire map. Assignment is thus trivial — voxels belong to the lone tetra that encloses that voxel. However, in the case of the custom mesh, voxels which are on the border between two segments may be enclosed by two or more tetra cells, so their assignment is uncertain. The segmentation we create in this step explicitly assigns every voxel inside the mask to one and only one tetra. **Segmenting the mesh using another tool** We recommend creating a segmentation using the Segger tool in ChimeraX, and we cover that technique in this tutorial. If you prefer another tool, 3D Flex Mesh Prep jobs also accept an `.mrc` file with an integer value in each voxel corresponding to that voxel’s segment. The segmentation `.mrc` file should have the same dimensions as the map from 3D Flex Data Prep. In each voxel, store an integer value to indicate the segment ID for that voxel. All voxels of a given segment should contain the same segment ID value. The segment IDs you use should start from zero and be contiguous (i.e., for `S` segments, use the IDs `0, 1, 2,..., S-1`) . Indicate solvent voxels (i.e., "No segment") with `-1`. When identifying segments for fusion, use the integers in the `.mrc` file instead of using ChimeraX segment ID numbers. First, download the consensus volume from the 3D Flex Data Prep job. This volume has already been cropped and downsampled to match the training box size. For Nav 1.7, the consensus volume looks like this: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FEHsvtxWIDbS6p7CulFGl%252Fconsensus-overview.png%3Falt%3Dmedia%26token%3Dba19b549-13e8-4a81-abb6-7bd1ebb48d35&width=768&dpr=3&quality=100&sign=c5e699b5&sv=2) From the lower contour (transparent in this image) it is clear that the C-terminus of the channel (bottom of the image) is poorly aligned, perhaps due to flexibility. However, it should never slide along or separate from the well-aligned transmembrane domain (TMD), so the TMD and C-terminus should be one segment. Two fabs are bound to the channel at the top of the image. A 3D Variability Analysis job shows that these Fabs flex toward and away from each other, so they should be separated into individual mesh segments. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FgZ0XH90qcY0RNwhUUCx4%252Ffullsize-3dva.gif%3Falt%3Dmedia%26token%3D0c0c9591-3032-4e5f-bb1e-1885fbf2a67a&width=768&dpr=3&quality=100&sign=31832875&sv=2) 3D Variability Analysis of the NaV 1.7 channel. Finally, the micelle is visible in the low contour. There is not a clear best method of treating the micelle in 3D Flex Meshes. We generally recommend starting by segmenting the micelle into its own mesh and marking it as rigid, but there are several alternate treatments of the micelle which may produce better results with some targets. See the [Micelles, fusion, and rigidity](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#micelles-fusion-and-rigidity) section of this page for more discussion of this topic. The custom mesh for this target will therefore have four mesh segments: 1. The “root” segment of the TMD and C-terminus 2. The micelle, which should be treated as if it is rigid 3. The “left” Fab 4. the “right” Fab Following the guidance in the [Mask Creation tutorial](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#method-one-volume-segmentation) , the Segger segmentation for such a mesh looks like this: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FxWoPGhRKoRxqWLrxjgFb%252Fsegmentation.png%3Falt%3Dmedia%26token%3Ddfdb95a7-32e8-4cb4-9fab-5bf018c4fc26&width=768&dpr=3&quality=100&sign=42a881ee&sv=2) At this point you should make note of the segment numbers for your segmentation. You can find the segment number by hovering over a region of the segmentation and reading the number out of the tooltip. In this example, the “left” Fab is segment `570`, the “right” fab is segment `569`, the micelle is segment `572`, and the TMD/C-terminus segment is number `571`. The hover tooltip displays both the segment number and ChimeraX’s model ID. **The segment number is the one you will enter into CryoSPARC** and usually appears first. It does not have a prefix and is typically a larger integer, e.g., `570`. The ChimeraX model ID, which is not what you want, is instead prefixed with a `#` and is typically two small integers separated by a `.`, e.g., `#2.2`. Save the segmentation as a `.seg` file using File > Save segmentation in the Segger pane and upload the resulting file to the CryoSPARC system. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#decide-on-a-fusion-strategy) Decide on a fusion strategy #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#why-fuse-segments) Why fuse segments? Recall that the ultimate goal of this job is to create separate mesh segments with overlapping tetra cells to allow for discontinuous movement of domains which are not physically attached to each other. When you import your segmentation, the creation of these independent meshes is automatic, since they are fully defined by the base tetra grid and the segmentation file. However, there is no way to know from this information alone which vertices should be physically coupled and which should be allowed to move freely. For instance, using the above segmentation, the mesh creation job will create four tetra mesh segments. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FQkhdKeL3UcS5PDOUGXU0%252Fraw-meshes.png%3Falt%3Dmedia%26token%3D056bf906-66de-424f-ad76-7dd361366a00&width=768&dpr=3&quality=100&sign=c61004a&sv=2) In these images, tetras are colored by which mesh segment they belong to. They have also been slightly scaled inward for visibility — the true tetra cells share edges with no gap between them. Each of these mesh segments comprise individual tetras which use the same vertex grid. Additionally, each segment contains some tetra cells which enclose the same physical space as other grids. For instance, the green and yellow grids both contain tetra cells which enclose the interface between the two fabs. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FAtjof4Ej094b7THaTSFy%252Ffab-interface.png%3Falt%3Dmedia%26token%3Dd5df47c8-e1bb-4762-9d77-b070a4ca9e9a&width=768&dpr=3&quality=100&sign=da143a1f&sv=2) We can show this in another way by marking the vertices with symbols based on which grids use that vertex, as we did for the 2D example above: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FHhVfEFL7sc2vqwXsi04u%252Fverts-slice.png%3Falt%3Dmedia%26token%3Dcd7e3273-820b-43b8-ac59-35c1bd918dd6&width=768&dpr=3&quality=100&sign=2a00bc24&sv=2) In this image, the base tetra mesh vertices are marked with transparent spheres. If a tetra mesh contains any tetra cells which use a vertex, that vertex is marked with a symbol corresponding to that mesh. Vertices between the two fabs are used by both the green and yellow meshes, so they are marked with a green cube and a yellow cone. As with the 2D example, **the green and yellow meshes each have their own “copy” of these vertices** that they can move independently during training and reconstruction — they only overlap now because both meshes start in their consensus, undeformed state. Additionally, each Fab also has overlapping tetra cells with the TMD/C-terminal segment: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FRZhVA4ge7cmsnaKPUmv8%252Fthreeway-mesh-overlap.png%3Falt%3Dmedia%26token%3D50cbad21-d0f8-44b2-bfcf-c45933f788c9&width=768&dpr=3&quality=100&sign=b185e670&sv=2) The Fabs are bound tightly to the TMD, and so the 3D Flex model should not be allowed to move voxels which belong to the fab away from voxels which belong to the TMD. However, if each Fab is in its own mesh, there is no reason for the model to keep them attached to the TMD. We must impose this mesh fusion on the model. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F6knSvattmV1o1EnNqAtq%252Ffusion-cartoon.png%3Falt%3Dmedia%26token%3D1d604e0b-51d8-47f2-b3a7-80d978bd2198&width=768&dpr=3&quality=100&sign=521742bb&sv=2) Consider the 2D cartoon above. If each mesh is allowed to move independently (top), the arrows can separate from each other instead of sliding. If, however, the overlapping tetras are removed and the border vertices are forced to occupy the same position, the orange arrow can no longer move away from the purple arrow. The process of: 1. Deleting one copy of tetras shared by two meshes, then 2. Fixing the position of the border vertices to be the same in both meshes is called a mesh fusion. Tuning these fusions is an important component of generating a custom mesh. Typically, several fusion strategies should be tried to find the optimal parent/child order and final mesh topology. In this example, we will fuse each fab and the micelle to the TMD and no other fusions. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#mesh-fusion-and-rigidity) Mesh Fusion and Rigidity It is important to be careful when choosing a mesh fusion strategy when some segments will be marked rigid. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F9TGuN2lTaZ35LffgzMLo%252Fmesh-rigidity.png%3Falt%3Dmedia%26token%3D6e31b931-54a0-4874-a18a-33b949f6a7b0&width=768&dpr=3&quality=100&sign=1a2b7cb9&sv=2) In this cartoon example, the blue segment is fused to yellow, yellow to orange, etc. The pink segment is the only rigid segment. It may be obvious that the vertices that purple and red share with pink will be rigid, since those mesh segments are fused. However, the central vertex of _all_ of the mesh segments will be rigid, since blue cannot move its central vertex without moving that of yellow, etc., until we consider that red’s central vertex cannot move without moving the rigid vertex of pink. Although this fact may be readily apparent in this simple example, results of segment fusion can be less obvious when considering a three-dimensional mesh. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#running-the-job) Running the Job Create the 3D Flex Mesh Prep job and input the prepared solvent mask and consensus volume and enter the path to the `.seg` file. In the example segmentation, the TMD/C-terminus segment is `571`, the two Fabs are `569` and `570`, and the micelle is `572`. The `Segment connections` field (which controls segment fusions) should therefore read `571>569,571>570,571>572` to produce our desired topology of TMD > Fab1, TMD > Fab2, TMD > micelle. Currently, 3D Flex Mesh Prep cannot create a mesh with a base num. tetra cells greater than 40. Finally, we set the micelle segment fully rigid by entering `572` in `Rigid segments`. This allows the 3D Flex Train job to correctly model the micelle without trying to move density in or out of this region. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#results) Results --------------------------------------------------------------------------------------------------------------------------------- ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F6vegnbNB6gr1VPkIISyW%252Frigid-micelle.avif%3Falt%3Dmedia%26token%3D42799faa-100d-4a4e-8e67-e4dab8957705&width=768&dpr=3&quality=100&sign=e74f253a&sv=2) Trajectories along each of the three latent space coordinates when the micelle is marked rigid and fused to the TMD. The three components of the 3D Flex Train results are displayed above. The constant regions (at the top of the image) of the fabs are able to move closer together and further apart with the custom mesh than the default mesh, because they are not fused together. Note, however, that the TMD is held relatively steady because of the rigidity of the micelle segment. Directly investigating the mesh can be overwhelming, but provides insight into how tetra cells are allowed to deform when one or more of their vertices are fused with other tetra segments. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FrAtFAxuIvYDgYYrHoWLK%252Fmesh-deform_optimized.gif%3Falt%3Dmedia%26token%3Deb056cd8-301d-41d9-85a6-d16420586dc4&width=768&dpr=3&quality=100&sign=704d93ea&sv=2) [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#micelles-fusion-and-rigidity) Micelles, fusion, and rigidity ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Consider the motions detected by 3D Variability again, paying close attention to the transmembrane domain: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FgZ0XH90qcY0RNwhUUCx4%252Ffullsize-3dva.gif%3Falt%3Dmedia%26token%3D0c0c9591-3032-4e5f-bb1e-1885fbf2a67a&width=768&dpr=3&quality=100&sign=31832875&sv=2) 3DVA modeled a significant flexing motion of the TMD, but the 3D Flex results captured only a slight deformation of this region. While it is possible that 3D Flex more accurately modeled the data and the channel truly is rigid, it is also possible that this rigidity is an artifact of marking the micelle segment rigid. As discussed above, if a vertex belongs to two fused meshes, it must occupy the same position in both meshes. This means that the edges of the TMD are treated as rigid, since they must be in the same position as the rigid micelle. Additionally, because the mesh is extended to fill the mask, the C-terminus moves in our custom mesh example than the default mesh, because it is also fused to the rigid micelle. It is undesirable for the micelle to hold parts of the target in a single rigid conformation. However, there is not yet an obvious best treatment of micelles in 3D Flex. Below we present a non-exhaustive collection of alternate fusion strategies, each of which have their own advantages and drawbacks. Aside from the fusion parameter, these training jobs all used the same parameters. Optimization of the other training parameters for each fusion strategy may further improve results. Which of these strategies (or a strategy of your own creation) works best depends on the sample. The best results typically require trying a few different fusion strategies, especially for new targets with as-yet unknown motion. Note that it is not currently possible to make a one-to-one comparison between different 3D Flex training jobs, as the latent spaces are almost certainly different. For these examples we display a trajectory along each coordinate (the default behavior of 3D Flex Generate). ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#alternate-strategy-1-do-not-mark-the-micelle-as-rigid) Alternate strategy 1: Do not mark the micelle as rigid Another approach would be to simply allow the micelle to move as much as any other segment. This approach allows the Flex model to “see” micelle density and so prevents it from filling the micelle with density from other regions, and allows nearby regions like the TMD to move freely. However, in some cases, the model will dedicate an undesirable amount of focus to modeling micelle noise rather than biologically-relevant flexibility. Here is the result of using the same mask and mesh as above, but without setting the micelle as rigid. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FKSxGtexjowRqaAF0sFHi%252Fnon-rigid-micelle.avif%3Falt%3Dmedia%26token%3D4da7b2b4-9b61-4f5b-b3d2-747999e78d06&width=768&dpr=3&quality=100&sign=ad5dd463&sv=2) Trajectories along each of the three latent space coordinates when the micelle is fused to the TMD without being marked as rigid. The flexibility of the fabs remains clear, but the movements of the TMD and C-terminus have also been captured, unlike the rigid micelle example. Clearly, fusing a rigid micelle to the TMD had a detrimental effect on the ability of 3D Flex to model those domains. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#alternate-strategy-2-mask-out-the-micelle-entirely) Alternate strategy 2: Mask out the micelle entirely One obvious approach is to mask the micelle out entirely, as one would for 3DVA or Local Refinement. In some cases, this does produce good results. In others, the model moves density which belongs to the helices to fill the empty space where the micelle should be. This happens because no matter how we mask the volume, the micelle is still present in the images, and the model has no way of knowing which parts of the image we may want it to ignore. Here is the result of using a mask during the mesh prep job that does not include the micelle, then creating a custom mesh with three segments: one for each fab and one for the channel. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FsbPxpKecSzEb0t3n5azS%252Fmask-out-micelle.avif%3Falt%3Dmedia%26token%3D1f58bbc5-8c8a-4205-9ff8-75180d1bbd7d&width=768&dpr=3&quality=100&sign=acee4190&sv=2) Trajectories along each of the three latent space coordinates when the micelle is masked out of the consensus volume prior to training. This mesh produced a model in which the C-terminus and TMD are both highly flexible — more than either the the rigid or non-rigid micelle models. However, the movement of the fabs toward and away from each other is no longer clear. Unlike all other forms of refinement currently available in CryoSPARC, 3D Flexible Refinement performs volume reconstruction in real space rather than Fourier space. This can introduce surprising and significant artifacts when the mask cuts through significant map density. If large streaks of map density are observed, we recommend increasing the size of the mask. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F0AenNj53RtPvy19Su7E4%252Fmasking-artifacts.png%3Falt%3Dmedia%26token%3D5c225dd9-7dc7-47c6-8bd0-569a46cc47af&width=768&dpr=3&quality=100&sign=56b98a34&sv=2) Masking artifacts in 3D Flexible Refinement. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#alternate-strategy-3-fuse-the-rigid-micelle-to-a-less-flexible-region) Alternate strategy 3: Fuse the rigid micelle to a less flexible region Recall that when we fuse regions, we do not necessarily need to follow the physical linkages of the sample. In the case of the NaV 1.7 channel, the micelle truly is fused (in some sense) to the TMD — but for the purposes of 3D Flex, we could fuse it to one of the Fabs instead. This would allow the TMD to move independently of the micelle, and could allow rigid body movement of the entire channel/fab complex relative to the micelle. Below is an example of this fusion strategy. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FjYeJr64yEaFzlP5U9Bje%252Ffuse-to-fab.avif%3Falt%3Dmedia%26token%3D7bb2845c-09ab-44b4-9998-3abd113d4576&width=768&dpr=3&quality=100&sign=42d339ed&sv=2) Trajectories along each of the three latent space coordinates when the micelle is marked as rigid and fused to a fab (the fab on the right in these maps). Treating the micelle in this way also failed to capture the flexibility of the TMD, and also may have held the Fabs more rigid. However, the model is able to move the C-terminus of the channel significantly more in this topology than when the micelle was fused to the ion channel. Recalling that each mesh segment is expanded to fill the mask this result is not too surprising, since the rigid micelle and C-terminus share a border. The movement of the C-terminus in this example may therefore also be captured if a fifth mesh segment was added surrounding the C-terminus and fused to the TMD. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#references) References --------------------------------------------------------------------------------------------------------------------------------------- 1. Ali Punjani and David J. Fleet, “3DFlex: Determining Structure and Motion of Flexible Proteins from Cryo-EM,” _Nature Methods_ 20, no. 6 (June 1, 2023): 860–70, [https://doi.org/10.1038/s41592-023-01853-8](https://doi.org/10.1038/s41592-023-01853-8) . 2. Hui Xu et al., “Structural Basis of Nav1.7 Inhibition by a Gating-Modifier Spider Toxin,” _Cell_ 176, no. 4 (February 7, 2019): 702-715.e14, [https://doi.org/10.1016/j.cell.2018.12.018](https://doi.org/10.1016/j.cell.2018.12.018) . [PreviousInstalling 3DFlex Dependencies (v4.1–v4.3)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement/installing-3dflex-dependencies) [NextWebinar Recordings](https://guide.cryosparc.com/processing-data/webinar-recordings) Last updated 1 month ago * [Summary](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#summary) * [3D Flex Mesh Overview](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#id-3d-flex-mesh-overview) * [Why model deformation with a mesh?](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#why-model-deformation-with-a-mesh) * [Discontinuous Motion](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#discontinuous-motion) * [When is a custom mesh required?](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#when-is-a-custom-mesh-required) * [Creating a custom mesh](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#creating-a-custom-mesh) * [Creating a segmentation](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#creating-a-segmentation) * [Decide on a fusion strategy](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#decide-on-a-fusion-strategy) * [Running the Job](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#running-the-job) * [Results](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#results) * [Micelles, fusion, and rigidity](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#micelles-fusion-and-rigidity) * [Alternate strategy 1: Do not mark the micelle as rigid](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#alternate-strategy-1-do-not-mark-the-micelle-as-rigid) * [Alternate strategy 2: Mask out the micelle entirely](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#alternate-strategy-2-mask-out-the-micelle-entirely) * [Alternate strategy 3: Fuse the rigid micelle to a less flexible region](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#alternate-strategy-3-fuse-the-rigid-micelle-to-a-less-flexible-region) * [References](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation#references) --- # Tutorial: Tips for Membrane Protein Structures | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures.md) . Membrane proteins are an increasingly important class of targets for cryo-EM in academia and industry. These targets are often small (<100kDA in molecular weight), flexible, and have a large micelle in the transmembrane region. Here, we list a few suggested tips for working with these targets in CryoSPARC, sorted by the different stages of processing. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FuaHA7G33ExcvQJLmTE9m%252Fv4-1-0-membrane-3D-GPCR.png%3Falt%3Dmedia%26token%3D291a19e7-7ca5-47bf-882d-c9b49f7e4429&width=768&dpr=3&quality=100&sign=624f58b&sv=2) One type of membrane protein: the Cannabinoid Receptor 1-G GPCR complex, (Kumar et al., 2019). Data from EMPIAR-10288. Density shown at two different thresholds to illustrate the micelle regions. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fqtyq8gHU4cCglJFnAjY4%252Fv4-0-0-membrane-pipeline.png%3Falt%3Dmedia%26token%3D9b5828d6-9957-4427-b67a-b75c8711b387&width=768&dpr=3&quality=100&sign=4a5df3b9&sv=2) ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures#pre-processing) Pre-Processing Generally, pre-processing steps remain unchanged from other nominal cryoEM pipelines—namely, we recommend the use of the [**Patch Motion Correction**](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/motion-correction/job-patch-motion-correction) and [**Patch CTF**](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-estimation/job-patch-ctf-estimation) jobs with no salient modifications to the parameters. We find that per-particle CTF refinements (post-3D refinement) rarely improve final structures due to the low amount of signal present per-particle in the micrograph for small membrane targets. Nevertheless, per-particle CTF refinement may be useful to try once a sufficiently detailed structure is refined. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures#particle-picking) Particle Picking Particle picking can be one of the most challenging parts of working with membrane proteins. [**Blob Picker**](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-picking/job-blob-picker) **/** [**Template Picker**](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-picking/job-template-picker) For crowded micrographs, the following two parameters can substantially affect picking performance: * `Particle diameter` * `Min. separation dist` * Reducing this for crowded datasets may help pick out more true particles. Other potentially useful picking jobs: * Neural-network-based particle picking techniques such as [**Topaz**](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/deep-picking/topaz) or [**Deep Particle Picker**](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/deep-picking/deep-network-particle-picker) can be useful when a large portion of particles are difficult to identify visually * [**Blob Picker Tuner**](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-picking/job-blob-picker-tuner) can also be quite useful for crowded micrographs. Be sure to choose approximately 100 manual picks, focusing on picks that are ‘clumped together’ and originating from micrographs that span a wide range of defocus values. [**Extract from Micrographs**](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/extraction/job-extract-from-micrographs) We suggest using an extraction box size that is approximately 2-3 times larger than the particle diameter. To account for signal displacement caused by the CTF, Rosenthal and Henderson (2003) suggest a box size of: D+2R\=D+2(λΔF/d),D + 2 R = D + 2 (\\lambda \\Delta F / d),D+2R\=D+2(λΔF/d), where DDD is the diameter of the particle, λ\\lambdaλ is the electron wavelength, ΔF\\Delta FΔF is the defocus value and ddd is the resolution. Note that the radius of displacement, RRR, is not a function of the particle diameter, and therefore this formula may result in a box size that is 4-5 times larger than the particle diameter when DDD is relatively small (as is typically the case for membrane proteins). Many of the CryoSPARC algorithms (e.g., 2D classification, ab-initio), however, are tuned for particle images with a box size that is 2-3 times larger than the extent of the particle. Furthermore, substantial computational savings can be achieved by using a smaller box size at early stages of processing where there are potentially many particles (millions). With small membrane proteins at reasonable concentrations, it is common to have many particles (several hundred) per micrograph and therefore very large particle sets in initial classification. To address this, a suggested pipeline is: 1. Extract particles with smaller box size (1.5X - 2X particle extent), 2. Perform multiple rounds of 2D classification, ab-initio, 3D classification, and initial (heterogeneous) refinements, selecting the best particles to carry forward 3. Re-extract surviving particles with a larger box size (2X - 3X particle extent) to reasonably account for all the information spread due to the CTF, and finally 4. Perform high resolution refinement(s). ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures#particle-curation) Particle Curation During 2D classification, a number of parameter changes can help improve performance for membrane targets: [**2D classification**](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/job-2d-classification) * `Force Max over poses/shifts` * By turning this off, 2D classification will automatically marginalize over the poses and shifts of each particle. For small particles, the uncertainty over poses and shifts can be substantial, and account for this through marginalization over these unknowns can be beneficial. Marginalization will add computational cost, but can help improve classification results in general when SNR is low. When this option is used, 2D classes will appear more “radially blurred” with less streaky or noisy artefacts towards the periphery. * `Number of iterations` * Increasing the default value of 20 may help improve classes. * `Batch size` * Empirically, users have found that doubling the initial value to 400 is sometimes beneficial. * `Circular mask diameter` * This can help account for crowding by masking out any information outside of a circular region in each particle image. For small particles with a lot of crowding, this can be necessary to ensure classification is based on view/conformation rather than arrangement of neighbours. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures#reconstruction-and-refinement) Reconstruction & Refinement #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures#a-note-about-masks) A note about masks First and foremost, all masks applied during 2D-to-3D processing should be smooth (i.e., contain no sudden 'cliffs' where the mask drops from a value near 1 to a value near 0) to avoid ringing effects. This is because sharp masks, when applied to half-maps during refinement jobs, can increase the likelihood of overfitting by introducing artifactual signal that is common to both half-maps. If you are generating masks using Chimera(X) (e.g., by following our [tutorial](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera) ), be sure to use the [Volume Tools](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools) job to add a sufficient soft padding width. As noted by the [mask generation tutorial](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#importing-and-processing-the-masks-in-cryosparc) , a useful rule of thumb is to keep the mask padding width proportional to the achieved resolution in Angstroms. As long as the _soft padding width_ is sufficient, and the mask covers the desired region of structure (while "cutting" through minimal density), the _threshold value_ and _dilation radius_ may be set as needed in order to generate a mask of the desired size. Furthermore, it is especially important for membrane target masks to not be overly 'tight' to the structure. For such small proteins, a tight mask can more easily lead to a situation where a refinement 'overfits' to junk/noise (cf. [Common Failure Modes](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures#common-failure-modes) ). In general, avoid creating a mask that is similar in shape to the secondary structure of the protein, and err on the side of loose (but nevertheless smooth) masks for all processing. [**Particle subtraction**](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta) In general, we find that particle subtraction can only help in very specific situations. Namely, if your structure contains two very rigid subunits, one large and one small. In this case, subtracting the larger subunit can improve the resolution of the smaller unit, if particle alignments are sufficiently well resolved for this subtraction to accurately remove the larger subunit signal. We strongly recommend avoiding the subtraction of micelles -- these structures are generally disordered, and it is very difficult to subtract them from particle images without removing other useful signal. Instead, consider the use of [local refinement](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement) with non-uniform refinement and marginalization turned on. [**Ab initio**](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-reconstruction/job-ab-initio-reconstruction) * `Initial / maximum resolution` * For smaller membrane proteins, it is often useful to set the initial and maximum resolutions to smaller numerical values (e.g., 9Å and 7Å). This is because smaller particles appear as featureless blobs at lower resolutions and there will not be enough information to align particles and recover the structure. [**Non-uniform refinement**](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-non-uniform-refinement-new) Non-uniform refinement can significantly improve refinements for targets that contain micelles and for smaller proteins. Consider the following two modifications if refinement results are poor: * `Initial lowpass` * Empirically, increasing this resolution (e.g., to a lower numerical value such as 15Å) may improve results for smaller targets. * `Static masking` * For small, low-SNR particles, dynamic masking may perform poorly. Instead, supplying a soft, static mask may improve the final refinement. [**Local refinement**](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement) Local refinement can also be quite useful for membrane targets. Note that local refinement masks **must** be softly padded, especially when cutting into density (even a micelle). A few salient parameters to consider: * `Rotation/Shift search extent` * When using smaller masks, tighter orientation search extents generally produce better results. * `Marginalization` * (Default on) Marginalization over poses and shifts can greatly improve alignments for smaller targets. * `Non-uniform refine enable` * (Default on) Non-uniform refinement can help account for disordered regions (such as micelles and flexible/floppy appendages). * `Rotation/Shift gaussian prior widths` * In cases of small molecules, small masks, or poor SNR, local refinement may benefit from the introduction of gaussian priors around each particle's initial orientation parameters. The utility of these priors is commensurate with the quality of the initial alignments. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures#heterogeneity-analysis) Heterogeneity Analysis CryoSPARC includes a wide assortment of tools for assessing and separating heterogeneous datasets. For high-resolution refinement of any protein, it is critical to ensure that the dataset is as homogeneous as possible; this often entails both particle curation (junk removal) and pruning of heterogeneity. In addition to 2D Classification and Ab-Initio Reconstruction, several other job types for heterogeneity analysis are highlighted below, along with important parameters to consider. [**Heterogeneous Refinement**](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-heterogeneous-refinement) * `Force hard classification` * Hard classification can improve results for low-SNR particles, especially when the target contains a static (well-resolved) domain connected to a flexible/heterogeneous domain (such as a micelle). [**3D Classification**](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-3d-classification-beta) * `Force hard classification` * Similar to Heterogeneous Refinement, force hard classification can help isolate regions of heterogeneity. * `RMS convergence criterion` * For low-SNR particles, the standard class switching criterion may lead to more F-EM iterations than necessary and cause processing to take longer. Consider turning this secondary criterion on to save computational cost. [**3D Variability**](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-3d-variability) 3D Variability (3DVA) analysis can be an especially important tool for heterogeneity analysis of small membrane targets. The 3DVA publication includes results on the Cannabinoid Receptor 1-G GPCR, which show that 3DVA can resolve two different bending motions of the 53kDa transmembrane region of the protein. When running 3DVA, be sure to supply a soft solvent mask to ensure that the job does not resolve variation due to the micelle. It is often advantageous to use a mask that _excludes_ the micelle, nanodisc, or other disordered regions, in order to force the algorithm to focus only on variability within the ordered region. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures#common-failure-modes) Common Failure Modes * "Spiky" densities like the one shown below are often a sign that there are many junk particles in the dataset — this can be especially prevalent in membrane protein datasets where particle picking is difficult. In these cases, it is often helpful to further “purify” the dataset, by either: * performing additional 2D classification rounds, or * running ab-initio reconstruction with multiple classes, then using the resulting volumes (including junk classes) to initialize heterogeneous refinement or 3D classification jobs and processing all the particles. Particles that fall into intact classes where the protein density is strong can be used for further refinements and particles falling into other classes can be discarded. This “junk-sorting” in 3D can often separate junk particles more effectively than 2D classification. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FN4CdlSInFDBe324dbDpI%252Fv4-1-0-membrane-E11030-spikes.png%3Falt%3Dmedia%26token%3Dcf1bc981-b6c0-482c-bfe2-825c9c7d37cb&width=768&dpr=3&quality=100&sign=cc2f5949&sv=2) A ’spiky’ hyaluronan synthase (the same density shown at two different thresholds for clarity) resolved from one class of a 3D classification job. Data from EMPIAR-11030 (Maloney et al., 2022). ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures#useful-discussion-threads) Useful Discussion Threads * [https://discuss.cryosparc.com/t/cannot-align-small-protein-complex-particles/4674/14](https://discuss.cryosparc.com/t/cannot-align-small-protein-complex-particles/4674/14) * [https://discuss.cryosparc.com/t/ab-initio-yield-better-map-than-nu-refinment-on-a-small-membrane-protein-complex/6778/2](https://discuss.cryosparc.com/t/ab-initio-yield-better-map-than-nu-refinment-on-a-small-membrane-protein-complex/6778/2) * [https://discuss.cryosparc.com/t/strange-reconstruction-failures-in-heterogeneous-refinement/8802/6](https://discuss.cryosparc.com/t/strange-reconstruction-failures-in-heterogeneous-refinement/8802/6) ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures#citations) Citations Kumar, Kaavya Krishna, et al. "Structure of a signaling cannabinoid receptor 1-G protein complex." _Cell_ 176.3 (2019): 448-458 Maloney, Finn P., et al. "Structure, substrate recognition and initiation of hyaluronan synthase." _Nature_ 604.7904 (2022): 195-201. Rosenthal, Peter B., and Richard Henderson. "Optimal determination of particle orientation, absolute hand, and contrast loss in single-particle electron cryomicroscopy." _Journal of Molecular Biology_ 333.4 (2003): 721-745. [PreviousCase Study: Exploratory data processing by Oliver Clarke](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-exploratory-data-processing-by-oliver-clarke) [NextTutorial: Common CryoSPARC Plots](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots) Last updated 1 month ago * [Pre-Processing](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures#pre-processing) * [Particle Picking](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures#particle-picking) * [Particle Curation](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures#particle-curation) * [Reconstruction & Refinement](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures#reconstruction-and-refinement) * [Heterogeneity Analysis](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures#heterogeneity-analysis) * [Common Failure Modes](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures#common-failure-modes) * [Useful Discussion Threads](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures#useful-discussion-threads) * [Citations](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures#citations) --- # Tutorial: Mask Creation | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera.md) . [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#what-is-a-mask) What Is a Mask? ------------------------------------------------------------------------------------------------------------------------------------------------------------ In many circumstances it is important to specify a region of a 3D volume. Whether this is to indicate a sub-volume to refine in Local Refinement, a volume to subtract for Particle Subtraction, or simply the region in which to calculate the GSFSC, we use a mask to select the volume. A mask is another 3D volume with the same box and pixel size as the volume to which it will be applied. The mask has a value of `0.0` outside the region to select and a value of `1.0` inside. When the volume is multiplied by the mask, the result will be a box that is empty except for in the region of interest, which will have the same information as the original volume. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FO5DHZjdoBXi8hLx6xOuc%252Fmask-creation_how-masks-work.png%3Falt%3Dmedia%26token%3Db82a1851-ac11-4c1a-b74d-fafb46d85873&width=768&dpr=3&quality=100&sign=14ab885d&sv=2) To mask a region of a volume, the original volume is multiplied by a separate volume (the mask) which contains 0.0 outside the region of interest and 1.0 within the region of interest. In almost all cases, masks for cryoEM must be “softened” by adding a smooth transition between the `1.0` values inside the mask and the `0.0` values outside the mask. This softening prevents [ringing artifacts](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#why-do-masks-need-a-soft-edge) which severely degrade alignment. A softer mask introduces less severe artifacts, but includes more information from outside the region of interest. The [Volume Tools](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools) job can be used to create a binarized (`1.0` inside and `0.0` outside) mask with a soft edge all in one step from an input map or mask. The ideal softness for a given dataset and subvolume typically must be determined empirically, but we recommend a minimum soft padding width of 5×resolutionapix5 \\times \\frac{\\mathrm{resolution}}{\\mathrm{apix}}5×apixresolution​ where **resolution** is the GSFSC resolution in Å and **apix** is the pixel size in Å. In some cases, it can be beneficial to add a wider soft edge or even an expansion of the 1.0 values (i.e., making the mask “wider”, called dilation). It is often best to try a few different combinations of dilation and padding and determine which produces the best results. Why do masks need a soft edge?[](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#why-do-masks-need-a-soft-edge) When we view particles, we view them in the spatial domain. However, particles are aligned in the frequency domain. To convert an image to its frequency domain representation we must take the 2D Fourier Transform. This is where the difficulty with hard-edged masks arises. A sharp transition in the spatial domain (the “normal” way of looking at images) becomes an infinitely fluctuating wave in the frequency domain. This wave artifact is known as “ringing". When we align images with ringing artifacts, we run the risk of lining up the artifact from our sharp mask instead of the information from the image itself. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fg5A9SsokwYD0iRUgvpdo%252Fmask-creation_spatial-vs-freq.png%3Falt%3Dmedia%26token%3D5d0bf952-5363-42bc-bc39-8101ac5e17d1&width=768&dpr=3&quality=100&sign=6cf00e14&sv=2) A comparison of various wave shapes in the spatial and frequency domains. As the spatial wave gets a softer edge, ringing artifacts in the frequency domain disappear. The softer the edge is in the spatial domain, the less ringing we observe. However, the softer the mask’s edge, the more of the volume outside our region of interest we include. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#common-pitfalls) Common Pitfalls ------------------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#mask-too-tight) Mask too tight One very important thing to consider is that masks which are too tight or too high-resolution introduce shared information into both half-maps, which breaks independence between the two half sets and artificially inflates the GSFSC curve. The magnitude of this effect is directly related to how much “ringing” is present in the mask’s frequency domain; thus sharper masks produce more severe artificial correlations. In jobs which produce GSFSC curves, be wary of results in which the Tight FSC curve does not closely follow the Corrected curve. A mismatch between these curves typically indicates an over-tight mask. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FwlyJe2V6H9ysOFK22aoT%252Fmask-creation_mask-tightness.png%3Falt%3Dmedia%26token%3D47fb2796-bb8a-4450-be5d-0703633146b0&width=768&dpr=3&quality=100&sign=989ff91c&sv=2) As a mask gets tighter and tighter, it introduces more and more information to the volume itself. Because the same mask is applied to each half-set, the FSC curve is affected. This results in the tight and corrected FSC curves not following each other --- a telltale signal of a too-tight mask. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#mask-too-small) Mask too small One common application of masks is [Local Refinement](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-new-local-refinement-beta) . Local Refinement is a powerful technique for improving the quality of smaller sub-volumes of a map. However, when the masked sub-volume is too small, not enough information remains for reliable image alignment. This can result in overfitting, characterized by noise, shells, or “blips”, especially surrounding the edge of the mask. If a small subdomain must be masked out for Local Refinement, we recommend applying a Gaussian prior to reduce overfitting. More information on Gaussian Priors is available on the job page. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#mask-base-creation) Mask Base Creation ------------------------------------------------------------------------------------------------------------------------------------------------------------------- A video version of this section of the tutorial. The same three techniques are covered in the article and the video. The first step in all mask creation workflows is the creation of a mask base. We focus here on the three most common workflows: * erasing regions of an existing map * using volume segmentation on an existing map * creating a mask using a molecular model For all three techniques we will use ChimeraX, a molecular visualization tool developed by the Resource for Biocomputing, Visualization, and Informatics at UCSF. ChimeraX is free for academic, government, nonprofit, and personal use and can be licensed for commercial use. The visuals for mask creation use data from EMPIAR-10073. This dataset was originally collected and processed by [Nguyen et al](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#references) . **Mask Bases** Throughout this page we make a distinction between mask bases and masks. **Mask bases** are created by a user to specify the region the mask ought to cover. They may have a hard edge and may or may not be binarized. **Masks**, on the other hand, are ready for use in CryoSPARC. They have been binarized, dilated, and padded. Typically a **mask base** is plugged into a Volume Tools job to create a **mask**. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#method-one-volume-segmentation) Method One: volume segmentation This technique uses watershed segmentation to split a volume into regions. Masks are specified by deleting regions outside the desired volume. This technique remains relatively simple while also allowing for the construction of complex masks and is the method we recommend for most purposes. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#step-1-open-and-blur-the-input-volume) Step 1: Open and blur the input volume Blurring a volume before creating your mask achieves two aims. Practically, it is significantly easier to select a region of interest when high-frequency noise has been attenuated with a blurring operation. Theoretically, it is important that the mask does not introduce high-frequency correlations between the two half maps. Only building blurred masks helps reduce the chance of this happening. 1. Open the volume in ChimeraX. This example uses the results of a non-uniform refinement. For all commands in this example, **the base volume is volume #1**. 2. Blur the volume using a Gaussian filter: `volume gaussian #1 sDev 2`. Increasing the value for `sDev` will make the map more blurry. This command creates a new, blurred volume. **The blurred base volume is volume #2**. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2Flh7-us.googleusercontent.com%2Fioiz_IVRSTJfYWZsw8MT2BaBgT8iWS-dBZG26ARd5PVMnsd5JZw2ytCih5jDpLOTTHwyMiQJA1ImVZW5R4BBsziT2eE-_vvBkjhiLqV3L0EfKFwX-RSZX-yfAlbwUUVm709utiOoT9aBN0RkBUMMN3I&width=768&dpr=3&quality=100&sign=b87295ae&sv=2) Comparison of a map before (left) and after (right) application of a Gaussian blur with standard deviation 2. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#step-2-segment-the-volume) Step 2: Segment the volume In this step, the [Segger](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#references) tool in ChimeraX splits the blurred volume into several regions. 1. Contour the blurred map until no noise is visible and the region which will ultimately form the mask has the desired topology. 2. Open Segger via clicking Tools > Volume Data > Segment Map ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2Flh7-us.googleusercontent.com%2Fy6bexV_y_jnVGzpeOZiOaG3T5uFLjXctdZq_cjj5B3FVMseVYPZclvgDBaJBYpSvhCvd2h45RLT2r9_MZ3roaeTsLUdoMaFhgSH9MWmVHvuwrDemh70f7S9KSQBMT4nIVK853or50YL1O-VYwxtj39w&width=768&dpr=3&quality=100&sign=eb582f25&sv=2) The Segment Map tool can be found in Tools > Volume Data > Segment Map. 1. Click "Segment" in the Segment Map pane to produce the segmentation. In this case, **the segmentation is model #3**. This segments the map into several regions, each of which is a distinct color. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2Flh7-us.googleusercontent.com%2FfS9krLQcxcZ-7psNeC67R1vtjwYhjS-W1yqAwyHSKgMmbWoS9kbaRTfIqcwoKoSM4dQUYnKkVxCJ_Yf3t-WPu7hhWyJn1trjdWBONmcifLnYrqm9N3XKeOkdR2UsqFK3PL3WNLqCuxlQ576kMUhrnkk&width=768&dpr=3&quality=100&sign=6d6a5b2&sv=2) A segmented volume. Each region is a distinct color. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#step-3-hide-unwanted-regions) Step 3: Hide unwanted regions In this step, we build our mask around the region of interest by hiding regions we wish to exclude from the mask. Because **there is no undo operation** when working with segmented maps, we recommend that users only hide the regions rather than outright deleting them. This also makes it easier to generate a complementary volume for Particle Subtraction, which we will do later on. 1. Open the "Shortcuts Options" dropdown in the Segment Map pane. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2Flh7-us.googleusercontent.com%2FSyf3AyM4mIxtih3OyI-qXUrk4II9qIe1toywPuTudMG9o-U8W64UsTwx_K9ofNZp2xFuxOda1qU4_N2vMtOgqi1xiRHDsVemxMPtnXF1rbixux9uFD5TBiq_ZoTSiWKfwNyXFeaG368uH7A_GwD66G8&width=768&dpr=3&quality=100&sign=889f7e1a&sv=2) The Segment Map pane with the Shortcuts Options panel open. 1. **Control-click** a region you wish to exclude from the mask to select it. The region should be surrounded by a light-green outline. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2Flh7-us.googleusercontent.com%2F_EOzTnxzFplTmjomhNyVBE-07MOu6jFwT5ZhXBilDz7Fre1DbbxYxQts165MT2eabKN4AbYvAViaXHBI13EHitnOfUiiOmuVZFkGKIK16jk15uqoHEJq1NOKi5QALT4CkoXGZJiY9qMGfja-zhqQN8w&width=768&dpr=3&quality=100&sign=b643a375&sv=2) The grey region in the top-left of the map is selected. 1. Click the Hide button in Shortcuts Options to hide the region. The hidden region remains selected until a new region is clicked, so clicking “Show” will return the region to a visible state. 2. Proceed to hide all regions which are to be excluded from the mask. * **Control-click-and-drag** selects multiple regions in a box * **Control-shift-click** adds or removes a region from the current selection * If a region contains parts of the map you wish to keep and parts you wish to exclude, select only that region and click “Ungroup” in Shortcuts Options. This will break the region into smaller subregions. Repeat the process until you can isolate the regions you wish to exclude. When this process is completed, the segmentation model should have all regions outside your desired mask hidden and all regions in the mask shown. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2Flh7-us.googleusercontent.com%2Fzq-mG3LVmAHkKhP3_KaJtR2TFYMhRpjDOhbat3xbajM6lM9oSmPQT7QPtRU-G6yXZ3X9RiFbpDogUDrTGnV4iBj5rfCKPHZLetxzOxTUjK1kdfU_eclCKdvhzC6wobceh78NqrCqJxWueabIm1vbuA4&width=768&dpr=3&quality=100&sign=47662d67&sv=2) A mask base segmentation for the tri-snRNP foot domain. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#step-4-create-the-mask-base) Step 4: Create the mask base CryoSPARC cannot accept Segger segmentations, so they must first be converted to `.mrc` format. 1. Control-click and drag over the entire segmentation to select all visible regions. 2. In the Segment Map pane, click File > Save **selected** regions to .mrc file. The filename you choose at this stage is not important, as the resulting .mrc file has some issues we will fix in the next step. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2Flh7-us.googleusercontent.com%2FQQJ_2Ev4QNmc5UtMMWt8rjDsvz_i8kv2je4848vSZKpLPTql8_JJDE9vRf3JUwnP7vDK5ArrSc4drSm0PpQJbBFurQCta_GFg4llIIw6wuyDmeVtRGOSLgvtasFqRHy9Hp3Mw19BHfPQNX2JA_Dzaa4&width=768&dpr=3&quality=100&sign=a912a359&sv=2) The segmentation can be converted to a .mrc file using File > Save selected regions to .mrc file in the Segger pane. This step generates a new volume, the mask base. In this example, **the mask base is volume #4**. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#step-5-optional-generate-the-complementary-mask-base-for-particle-subtraction) Step 5 (optional): Generate the complementary mask base for Particle Subtraction If you are performing Local Refinement, it can be helpful at this stage to create the complementary mask for Particle Subtraction. To do this, we delete all the regions we used to create the mask base, then save another `.mrc` file with the remaining regions. 1. With the regions used to create the mask base selected, click “Delete” in the Segment Map pane. 2. In the Segment Map pane, click “All” next to “Show regions:”. This reveals the regions you hid during Step 3. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2Flh7-us.googleusercontent.com%2FgUe5b5EN23seUoFZEi5pggTQuzXt7BcKKXDgHcVKeCzZg5wyzAn6ycIjqMybquGeRNR-1Dfrt0Zt_zSJcSCVQbPXko4dBXEpcKts2O17SDOHN66hdQVIcy6cCSyDpqtEGAGeQbk4BS_6b2omKs4egws&width=768&dpr=3&quality=100&sign=11b52df3&sv=2) The complementary mask to be used for Particle Subtraction. 1. Select all regions and save an .mrc file as in Step 4. In this example, **the particle subtraction mask base is volume #5**. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#step-6-fix-box-size) Step 6: Fix box size When Segger saves the regions to an `.mrc` file, it crops the box size to perfectly fit the mask base. This results in a box size that is different from the map’s box size, meaning CryoSPARC will not know where to position the mask. To resolve this problem, we must first resample the mask base onto the original map’s box. When using commands in ChimeraX, ensure that you are using the correct numbers for your maps, as they may differ from those printed here if you did not take the optional step 5, or your ChimeraX session already had maps or models loaded into it prior to starting this tutorial. 1. Use the command `volume resample #4 onGrid #1` to resample the mask base onto the original map’s box. Note that the resulting maps are positioned in the same region of space and contain the same information, but the box sizes are different. In this case, the volume saved by Segger has a box size of `96 x 94 x 129` voxels, while the map and resampled volumes both have box sizes of `380 x 380 x 380`. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2Flh7-us.googleusercontent.com%2F1w9x878QutRJ8ySLoOckbErUNRJieidj_cUl18__j4eOJDeWgV5sdTj3A1IrjMc8Be1aFU2vv6erwTmKXff7dE16_qMaHhErpkU7-98197E4lPVSxLi7ldxDX4-TneABaL205tPj4PszUTq23OS8PaI&width=768&dpr=3&quality=100&sign=f2cbbde3&sv=2) The resampled volume (cyan) and the volume created by Segger (red) have the same topology but different box sizes. 1. Save the **resampled volume** (in the case of this example, volume #5 if a particle subtraction volume was not created and volume #6 if a mask subtraction volume was created). This is the mask base, so we recommend using an informative name. 2. _(If creating a particle subtraction mask)_ repeat the above steps to resample the Particle Subtraction mask base onto the map volume. In this case, the necessary command would be `volume resample #5 onGrid #1`. This is the completed mask base for Particle Subtraction. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#step-7-upload-to-cryosparc) Step 7: Upload to CryoSPARC Both of the mask bases are now resampled and ready for import to CryoSPARC, via the Import 3D Volumes job. Once imported, the mask bases can be converted to masks via thresholding, dilation, and padding in the Volume Tools job. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#method-two-volume-eraser) Method Two: volume eraser For very simple masks, this technique is much faster than Method One. However, it can be susceptible to creating undesired noise and care must be taken when analyzing the resulting masks. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#step-1-open-and-blur-the-input-volume-1) Step 1: Open and blur the input volume Blurring a volume before creating your mask achieves two aims. Practically, it is significantly easier to select a region of interest when high-frequency noise has been attenuated with a blurring operation. Theoretically, it is important that the mask does not introduce high-frequency correlations between the two half maps. Only building blurred masks helps reduce the chance of this happening. 1. Open the volume in ChimeraX. This example uses the results of a non-uniform refinement. For all commands in this example, **the base volume is volume #1**. 2. Blur the volume using a Gaussian filter: `volume gaussian #1 sDev 2`. Increasing the value for `sDev` will make the map more blurry. This command creates a new, blurred volume. **The blurred base volume is volume #2**. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2Flh7-us.googleusercontent.com%2Fioiz_IVRSTJfYWZsw8MT2BaBgT8iWS-dBZG26ARd5PVMnsd5JZw2ytCih5jDpLOTTHwyMiQJA1ImVZW5R4BBsziT2eE-_vvBkjhiLqV3L0EfKFwX-RSZX-yfAlbwUUVm709utiOoT9aBN0RkBUMMN3I&width=768&dpr=3&quality=100&sign=b87295ae&sv=2) Comparison of a map before (left) and after (right) application of a Gaussian blur with standard deviation 2. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#step-2-create-copies-of-the-blurred-volume) Step 2: Create copies of the blurred volume Since the volume eraser tool directly modifies the volume it operates on, you must create another copy of the blurred volume if you plan on creating a mask for Particle subtraction. 1. Copy the map with `volume copy #2` #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#step-3-erase-the-region-outside-the-desired-mask) Step 3: Erase the region outside the desired mask In this step, we will erase the regions outside the mask using the Volume Eraser tool. This tool creates a sphere and allows you to erase everything either inside or outside the sphere. There is **no undo function** for this tool, so be careful when erasing volumes. You can create copies of the volume as you go if the process is long and complicated using `volume copy` as above. 1. Open the volume eraser tool: Right Mouse ribbon menu > Erase. You should see a sphere appear. Holding down right-click and dragging your mouse moves the sphere. Erasing inside or outside the sphere is accomplished with the buttons in the Map Eraser pane. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2Flh7-us.googleusercontent.com%2F04qv9G8YU2x-K6VRa6oWLvm2rd_dN0GHmbINRIn4pFqyYdpuxIQgzswXKgpRskx0OJKEumZu3f1lrhv4ElSRiwix0vpDh2dJPvsqBPk5TZGjmz_rnszR7IZd9KECHSbdy9SU23ZqkCr4XM-9l7afF7s&width=768&dpr=3&quality=100&sign=93eabf0a&sv=2) The Volume Eraser pane controls the size of the eraser (pink sphere) and allows for erasing (i.e., setting to 0) all values inside or outside the eraser. 1. Using the sphere, erase all regions outside your desired mask. Perform a close inspection of your final volume, being careful to notice small regions left behind by imprecise eraser placement: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2Flh7-us.googleusercontent.com%2FDvZhYdkLwD4y6-NJuVnmh6ikHttq72Rcm22J3RxV8hfVkQ1DnJ75Iryj28jKJu1bae4U2vWrEnibEM8Xm-BDo3ZEVGSCReVSqQFwmZmi4_9OH7yLIhVDvR94MMTIJ1Wrv2s7rnR2LPNkbd0uXIrWlDA&width=768&dpr=3&quality=100&sign=2313d50f&sv=2) Small fragments of the map may be left behind by the volume eraser. It is important to closely inspect the map after each eraser operation to remove this "dust". 1. Save the erased map as your mask base. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#step-4-optional-create-a-particle-subtraction-mask) Step 4: (Optional) Create a particle subtraction mask 1. Subtract the erased and blurred map (in this case, #2) from the unerased and blurred map (in this case, #3): `volume subtract #3 #2`. If the result has negative values in most voxels and an unexpected and noisy shape, the arguments were likely given in the wrong order. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2Flh7-us.googleusercontent.com%2FJSw8WIy42FpI97ShEJkDFd6Z42KpqmCc8NqObfARaa6LxnzSOts32RQnKyqYuVAB39HdP-Duu9iQEyUj9T5PsJc5VJy7piA27-CjY_60BdY7_nbsjINdWnz8fOjPP_wnTU90n4pQKZ1HFTly1Tfif2Q&width=768&dpr=3&quality=100&sign=77a48fe5&sv=2) Subtracting the local refinement mask base (#3) from the original blurred map (#2) creates the complementary particle subtraction mask (#5). #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#step-5-upload-the-mask-bases-to-cryosparc) Step 5: Upload the mask bases to CryoSPARC 1. Save the resulting mask bases to `.mrc` files and upload the files to CryoSPARC. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#method-three-molmap) Method Three: molmap ChimeraX can create volumes based on molecular models using a command called molmap. Generating mask bases using this technique is by far the simplest — using a single command, we can create a mask around (in this example) chain U: `molmap #2/U 16 onGrid #1`. If `onGrid` is left out of this command, a seemingly-correct mask base will be generated, but it will be on the wrong grid and so unusable! In this example, #2 is our molecular model, we generated a mask base with a resolution of 16 angstroms, and #1 is our map from CryoSPARC. Note that even though none of the information in the mask base comes from the map, you still must have it loaded so that the mask base is on the correct grid. In this command, resolution merely notes the level of detail in the resulting simulated map. It will not affect the quality of refinements using the mask. Note also that any resolution can be selected. ChimeraX is not simulating any electron microscopy process — it is simply generating a volume using the provided model and the specified resolution. We recommend that masks are never generated with a resolution better than (i.e., never a value lower than) 12 Å. Masks created using molmap are already on the correct grid and can immediately be saved and uploaded to CryoSPARC. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#converting-a-mask-base-to-a-mask) Converting a mask base to a mask ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- A mask base is converted to a mask by following steps: 1. The mask base is “binarized”. All values greater than a user-selected threshold are set to `1.0` and all values below this threshold are set to `0.0`. 2. The resulting binary volume is dilated. Additional pixels within a user-selected distance from the volume surface are also set to `1.0`. 3. The binary volume has a soft edge added (padding). This edge gradually decreases from `1.0` to `0.0` and has a user-specified width. All of these steps can be performed simultaneously via a [Volume Tools](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools) job, and more information about the necessary parameters are available on that job page. The binarization threshold will be different each time depending on the input mask base, and can be determined with a volume visualization tool like ChimeraX. A threshold should be selected such that there are no floating “specks” of density and the desired topology of the mask is preserved. The amount of dilation required also depends on both the dataset and the sub-volume. Generally adding a few pixels of dilation helps to prevent over-tight masks, and allows for the inclusion of newly-resolved density in the masked volume. Padding is an [essential component of masking](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#why-do-masks-need-a-soft-edge) in cryoEM to prevent ringing artifacts. We recommend a minimum padding width of 5×resolutionapix5 \\times \\frac{\\mathrm{resolution}}{\\mathrm{apix}}5×apixresolution​ where **resolution** is the GSFSC resolution in Å and **apix** is the pixel size in Å, but the optimal result can require significantly larger padding widths. We therefore recommend that users try a variety of mask dilation and padding combinations to find the ideal combination. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#next-steps) Next steps --------------------------------------------------------------------------------------------------------------------------------------------------- [Job: Local Refinement](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-new-local-refinement-beta) [Job: Particle Subtraction](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta) [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#references) References --------------------------------------------------------------------------------------------------------------------------------------------------- 1. Eric F. Pettersen et al., “UCSF ChimeraX: Structure Visualization for Researchers, Educators, and Developers.,” _Protein Science : A Publication of the Protein Society_ 30, no. 1 (January 2021): 70–82, [https://doi.org/10.1002/pro.3943](https://doi.org/10.1002/pro.3943) . 2. Thi Hoang Duong Nguyen et al., “Cryo-EM Structure of the Yeast U4/U6.U5 Tri-snRNP at 3.7 Å Resolution,” _Nature_ 530, no. 7590 (February 1, 2016): 298–302, [https://doi.org/10.1038/nature16940](https://doi.org/10.1038/nature16940) . 3. Grigore Pintilie and Wah Chiu, “Comparison of Segger and Other Methods for Segmentation and Rigid-Body Docking of Molecular Components in Cryo-EM Density Maps.,” _Biopolymers_ 97, no. 9 (September 2012): 742–60, [https://doi.org/10.1002/bip.22074](https://doi.org/10.1002/bip.22074) . [PreviousTutorial: BILD files](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-bild-files) [NextTutorial: Dynamic Masking in Refinements (v5.0+)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-dynamic-masking-in-refinements-v5.0) Last updated 2 years ago * [What Is a Mask?](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#what-is-a-mask) * [Common Pitfalls](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#common-pitfalls) * [Mask too tight](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#mask-too-tight) * [Mask too small](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#mask-too-small) * [Mask Base Creation](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#mask-base-creation) * [Method One: volume segmentation](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#method-one-volume-segmentation) * [Method Two: volume eraser](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#method-two-volume-eraser) * [Method Three: molmap](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#method-three-molmap) * [Converting a mask base to a mask](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#converting-a-mask-base-to-a-mask) * [Next steps](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#next-steps) * [References](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#references) --- # New Live Session: Start to Finish Guide | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide.md) . [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#start-to-finish-guide) Start to Finish Guide ---------------------------------------------------------------------------------------------------------------------------- This guide covers the end-to-end workflow for using CryoSPARC Live. In addition to the below, we recommend checking out the [CryoSPARC Live Walkthrough](https://guide.cryosparc.com/live/tutorial-videos) : Processing EMPIAR-10288 in CryoSPARC Live. [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-1.-create-new-session) 1\. Create New Session -------------------------------------------------------------------------------------------------------------------------------- > From the **CryoSPARC Live Sessions View,** create and configure new sessions and view a summary of existing sessions. ➡️ Navigate to the Sessions View by clicking on the CryoSPARC Live icon on the sidebar. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FH0Xp7igNVZL3gzpKebvw%252Fv4-0-0-app-browse-system-session-view.png%3Falt%3Dmedia%26token%3D67d8cceb-f895-405e-bf40-0746582e8a1e&width=768&dpr=3&quality=100&sign=a8178f21&sv=2) ➡️ Click on the **New Session** button in the header, which opens a panel in the sidebar. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FED7YGOVXwDDDcmCM7LQK%252Fv4-0-0-cryosparc-live-new-session-1.png%3Falt%3Dmedia%26token%3D999f8a4c-6bd8-4905-844c-0efebea5e23f&width=768&dpr=3&quality=100&sign=5bf0f7f7&sv=2) ➡️ Select or enter the CryoSPARC Project number where you would like this Session to be created. Enter a Title for the Session. **The project you select must already exist.** You may need to create a new Project in the CryoSPARC interface, if one does not already exist. **Recommended scenarios for creating new Projects/Sessions:** * **Collecting new data for the first time on a new target molecule:** Create a new Project and a new Session within it. * **Collecting data a second time on the same sample/target** (potentially the same or different grid from the same batch, potentially on a different day): Use the existing Project and existing Session where you processed the first set of images. In the existing Session, create a new exposure group (see Configure New Session, below) and start the Session again, causing it to read in the new set of images. * **Collecting data on a new sample/grid/preparation of the same target molecule:** Use the existing Project, but create a new Session. This allows easy re-use of 3D volumes, 2D templates, and easy combining of particle images downstream. You can create multiple Sessions within a project, for example if collecting/processing new data from a similar sample. The recommended workflow is to create a new Project for each new unrelated sample on which you are collecting data. **Only users who own a particular Project or have a Project shared with them can see CryoSPARC Live Sessions within those Projects.** To add a user to a Project, navigate to the Project Details Panel in your regular CryoSPARC instance and click `Share With Users` to select the user you wish to give access. ➡️ Click Create Session. This will open your new session. All new sessions by default are set to Paused status. [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-2.-configure-new-session) 2\. Configure New Session -------------------------------------------------------------------------------------------------------------------------------------- > On the **Configuration Tab**, enter required parameters (or load a saved Configuration Profile) and select compute resources. New Sessions will open on the **Configuration Tab** by default. In order to Start this New Session, you must first type in (or load from a profile - see [below](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#load-configuration-profile) ) a few required parameters in three sections (Configuration, Parameters, and Exposure Groups). Required parameters are outlined in red and are also summarized in the **Start Checklist.** ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNit3yYCmEg7qfGhXmM%252F-MNiw5jwzZ_86don4mKR%252FSTF_3_csl_stf_config_tab_new_1.png%3Falt%3Dmedia%26token%3Db8ddbe2f-296a-44c6-96d5-eaad6acc8698&width=768&dpr=3&quality=100&sign=f6acf1b8&sv=2) ### [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#configuration-section) Configuration Section #### [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#compute-resources) Compute Resources These are parameters relating to the GPU resources on which CryoSPARC Live jobs will run. For more details on hardware requirements for Live, please see: [Prerequisites and Compute Resources Setup](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FvgO3QPTYVKAXvhsjB8Ze%252Fimage.png%3Falt%3Dmedia%26token%3Db58b7318-c9d7-49f5-a308-72fde94bc4f7&width=768&dpr=3&quality=100&sign=7dfb9f4f&sv=2) ➡️ **Select/enter the following required parameters** * `Preprocessing Lane`: Select a Preprocessing Lane and a Number of Preprocessing GPU Workers. This is effectively the number of GPU workers that will carry out motion correction, CTF estimation, particle picking and extraction in parallel concurrently. It's possible to adjust the number of GPU workers allocated for the preprocessing stage throughout the lifecycle of the session. For example, at the start of the session, you can allocate four GPUs to quickly extract particles from exposures, then lower the number of workers to two or one when resources are needed for the reconstruction stage. For information on the minimum number of GPU workers you need to assign, see: [Prerequisites and Compute Resources Setup](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup) * `Reconstruction Lane`: Select a Reconstruction Lane where Streaming 2D classification and Streaming Refinement jobs will be launched. * `Auxiliary Lane`: Select an Auxiliary Lane where Ab-initio Reconstruction and Generate Templates jobs will be launched. * (Optional) `Use SSD`: If the disk you are using for the project directory is already an SSD, you don't need to copy the files to another SSD. Turn off `Use SSD` if you do not wish to copy files over (for larger datasets, it can take some time to write all files over to the SSD before processing can stream in new collected particles). * (Optional) `Priority`: Unless specified, all jobs in the Live Session will run according to the Session-level priority. For details on Priority Queuing in CryoSPARC, please see the [Priority Queuing Tutorial](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/tutorial-priority-job-queuing#cryosparc-live-session-priority) . * (Optional) (v5.0+) `Workers per GPU` : Specifies the number of Live preprocessing worker jobs to launch per GPU worker. On some systems, launching multiple preprocessing workers per GPU can improve throughput without requiring additional GPU hardware. #### [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#run-configuration) Run Configuration ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FiliABUPY3bClLk1rGJRp%252Fimage.png%3Falt%3Dmedia%26token%3D4a94008b-a22d-4e6b-b991-dc97a66baedb&width=768&dpr=3&quality=100&sign=4f7252bb&sv=2) * (Optional) `Exposure Processing Priority` : Determines the order in which exposures are processed or reprocessed. * Normal: Exposures are processed in ascending UID order (i.e. earlier exposures are processed first), however exposures are reprocessed in descending UID order (i.e. recent exposures are reprocessed first). * Oldest: The oldest exposure found will be (re)processed first. * Latest: The latest exposure found will be (re)processed first. * Alternate: Alternate between Oldest and Latest priority modes. This mode helps ensure that some recent exposures are always being processed, even if there is a backlog of older exposures that need to be reprocessed. * (Optional) (v5.0+) `Delay worker startup until ready` : Whether to wait until there is at least one exposure ready to process before queueing Live Worker jobs. Turning on this option allows you to start a Live session before any data has been collected, as long as you know where the movie files will be saved by the microscope control software. The Live session will not consume GPU resources or license tokens while waiting for data to appear. In this way, multiple Live sessions can be started in anticipation of data that will arrive later (e.g. from a multi-grid collection setup) and in conjunction with the next option (auto pause), data from multiple collections can be processed in an unattended fashion. * (Optional) (v5.0+) `Auto Pause Mode` : Automatically pause the session when there are no remaining exposures to process, and a sufficient amount of idle time has passed since the last processed exposure was found. * Standard: Pause the session immediately after the idle timeout has expired. This will kill any running 2D/3D streaming jobs at the time auto pause is triggered. * Graceful: Once the idle timeout has lapsed, also wait for 2D/3D streaming jobs to finish processing the available particles before pausing the session. * (Optional) (v5.0+) `Auto Pause Timeout` : The amount of idle time that must pass after the last found exposure before automatically pausing. While a session is running, jobs launched by Live can be viewed by clicking on the **'x Active Jobs'** button in the footer of the CryoSPARC Live application. You can also see Live jobs in the Resource Manager tab in your regular CryoSPARC instance. To quickly navigate to the Resource Manager, click on any job in the CryoSPARC Live application footer. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNit3yYCmEg7qfGhXmM%252F-MNiwB338-A_2X0hSoRR%252FSTF_5_csl_stf_footer_active_jobs_1.png%3Falt%3Dmedia%26token%3D03fb69cc-c38d-4baf-b0db-12cb4ca5abae&width=768&dpr=3&quality=100&sign=53f0466d&sv=2) This will open the Active Jobs modal. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNit3yYCmEg7qfGhXmM%252F-MNiwDuxU_j_EBVoWEys%252FSTF_6_csl_stf_active_jobs_modal_1.png%3Falt%3Dmedia%26token%3D1fe8308f-b049-4745-9821-7ee4403c4915&width=768&dpr=3&quality=100&sign=feb38fe4&sv=2) To adjust the Number of Preprocessing GPU workers or any other Compute Resources settings while a Session is Running, you will need to first Pause the Session, update the configuration and then Start the Session again. ### [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#parameters-section) Parameters Section These are data processing parameters that will be engaged across Microscope/Camera, Motion Correction, CTF Estimation, Blob Picker, Template Picker, and Particle Extraction once the Session is Started. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNit3yYCmEg7qfGhXmM%252F-MNiwH6IsFlAns6F8DVQ%252FSTF-7_csl_stf_parameters_section_1.png%3Falt%3Dmedia%26token%3D3e2db638-b63f-4f43-a422-e86aa5e50c80&width=768&dpr=3&quality=100&sign=82d22e9a&sv=2) ➡️ **Step 1: At a minimum, you need to enter the following 7 required parameters which are outlined in red.** * **Microscope/Camera Parameter** * `Raw pixel size (A)`: The raw pixel size of the input movie data. For super-resolution data, this should be the super-resolution pixel size (i.e. not the camera native pixel size). For EER data, this should be the camera native pixel size, and you should also modify the corresponding EER parameters for upsampling and dose fractionation. * `Accelerating voltage (kV)`: The accelerating voltage of the microscope collecting the data. * `Spherical abberation (mm)`: The spherical aberration of the microscope collecting the data. This should be zero if there is a Cs-corrector. * `Total exposure dose (e/A^2)`: This should be the total electron dose across each movie (i.e. not the per frame dose). * Blob Picker * `Minimum particle diameter (A)`: This can initially be set as the minimum dimension of the particle expected. This can be fine-tuned later on the Picking tab. * `Maximum particle diameter (A)`: This should be set equal to or slightly larger than the maximum dimension of the particle expected. This can be fine-tuned later on the Picking tab. * Particle Extraction * `Extraction box size (pix)`: This should be a box size in pixels for extracting particles. Generally, this should be set to about twice the particle diameter, and should be a set to a number that has 2,3,5,7 as its prime factors. Typical box sizes are between 256 and 640 pixels. Note that the box size is in pixels after Fourier-cropping (if enabled in motion correction). Mathematically good numbers are: If you would like to save micrographs or particles in [float16 format](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-float16-support) , you can enable the toggle titled "Save results in 16-bit floating point" under "Motion correction", "Particle extraction", or both. (CryoSPARC v4.4+) To view advanced parameters, click **Show Advanced**. You can also filter by a parameter name. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNit3yYCmEg7qfGhXmM%252F-MNiwKllgJwgdioY2_XA%252FSTF_8_csl_stf_show_advanced_2.png%3Falt%3Dmedia%26token%3D8bc1ea23-fd3e-40c9-bc82-a954a33cdbf2&width=768&dpr=3&quality=100&sign=c81032a2&sv=2) ➡️ **Step 2: Apply (i.e., Save) your parameters** * Click `Apply to All` to save your parameter entries for all exposures (existing and incoming) in the Session. **This is the recommended course of action when starting a New Session.** * If your Session is already running and you wish to have new parameter changes apply only to new exposures coming in after that point, then you may instead wish to click `Apply to Future`. ### [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#exposure-group-section) Exposure Group Section This section covers parameters that will tell CryoSPARC Live where to find new movie files, which files to read, whether to apply a gain reference and/or defect file (if available), and how to handle multiple Exposure Groups, if applicable. By default, there is always at least one Exposure Group in a Session. Exposure groups are collections of exposures that have the same optical parameters. You can use exposure groups for multiple purposes including: * A new exposure group per grid * A new exposure group for each beam-shift position in a template * A new exposure group for different squares on a grid * A new exposure group for a new data collection session, perhaps on a different day After enabling an exposure group, you can choose to continuously listen to the specified directory and filename wildcard filter for new exposures, or ignore the group if you no longer want to include a set of exposures for processing. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNit3yYCmEg7qfGhXmM%252F-MNiwO6liZwbhArYTJWi%252FSTF-9_csl_stf_exp_grp_1.png%3Falt%3Dmedia%26token%3Dec505c28-4a00-426b-a00e-fa62c84e0169&width=768&dpr=3&quality=100&sign=c42a83b1&sv=2) ➡️ **Step 1: Edit Parameters for Exposure Group 1** * `Enable continuous import`: Toggling this on will allow new exposures added to the directory, to be processed as they are found. Be default, this is enabled. If you are creating more than one exposure group and the initial exposure groups are not expected to contain any new images that Live has not already found, you can disable continuous import from the older exposure groups to save disk operations needed to check for new files periodically. * (Optional) `Ignore exposures from group`: This is useful only if you have multiple Exposure Groups configured, and you would like to ignore all exposures from a particular Group. This might be used if you find that all the images from a group are of poor quality, for example. * `Directory to watch`: The file system location where the exposures are being written or are already saved. This is a directory on the filesystem, **not** a wildcard path. * (Optional) `File name wildcard filter`: You can optionally filter by wildcard to select files within the Directory to watch, that match a specific extension, e.g., `*.mrc`. This is important if multiple types of files will be saved in the same directory. * (Optional) `Search recursively`: This option will traverse all subdirectories within the directory specified in `Directory to watch` that matches the `File name wildcard filter` value. This can be helpful if your data collection software writes files to multiple sub-folders, for example. Leaving this option turned off will tell the file engine to only search for files matching the `File name wildcard filter` value in the top level directory. If using Search Recursively, then in `Directory to watch`, ensure you choose the highest-level folder to which movies are being written for the data collection session. For example, for EPU, specifying `../Images_Disc1/Data` as the `Directory to watch` value and `FoilHole_*_Data_*_*_*_*.mrc` for the filter value when `Search Recursively` is enabled allows you to get around creating an exposure group for every grid square. * `Gain Reference Path`: Enter or select an absolute path to the gain reference, if the gain reference file is available. The gain reference should be in `.mrc` or `.gain` (for EER data) format. * `Defect File Path`: Enter or select an absolute path to the defect file, if the defect file is available. Note that zeros in the gain reference are also treated as defects and corrected in the same way. To add a new Exposure Group, click **'New'**, ➡️ **Step 2: Click Enable (i.e., Save) for each Exposure Group added** You must click Enable for each newly added Exposure Group. This saves the parameters you entered. Once the exposure group is enabled, its parameters are locked in and cannot be changed unless the Session is cleared. For more information on clearing a session, please see: [Live Jobs and Session-Level Functions](https://guide.cryosparc.com/live/live-jobs-and-session-level-functions) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNit3yYCmEg7qfGhXmM%252F-MNiwRmPwcWkDKxQop_b%252FSTF-10_csl_stf_config_filled_1.png%3Falt%3Dmedia%26token%3Db4105a70-a2e7-4931-b7e9-9c3775e10b90&width=768&dpr=3&quality=100&sign=baef5c20&sv=2) ### [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#load-configuration-profile) Load Configuration Profile CryoSPARC Live [configuration profiles](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#load-configuration-profile) are not backwards compatible when created in v5.0+: profiles created in previous versions will be retained when updating to v5.0, but new profiles created in v5.0 will be dropped if downgrading to v4.7 or older. To speed up the configuration a new Live session, you can save all or some of the parameters of an existing session into a configuration profile and apply the profile to future sessions. Configuration profiles can store: * Compute resource preferences * Exposure groups * Parameter sections such as microscope params #### [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#creating-a-profile) **Creating a profile** In any Live session that has been configured with all required parameters, you can choose to save a subset or all of that configuration as a profile. ➡️ Navigate to the 'Configuration' section. ➡️ Click the Profiles dropdown, then 'Save current configuration'. ➡️ Choose a title, and one or more sections to save to that profile. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiwTBlSuHtyBRk2iUZ%252F-MNiwqtlQojPywqtfxKW%252FSTF-11_csl_stf_conf_prof_1.png%3Falt%3Dmedia%26token%3D49b0d3a5-b34e-4f24-8205-48f3307804cc&width=768&dpr=3&quality=100&sign=181c6580&sv=2) #### [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#applying-a-profile) **Applying a profile** ➡️ Navigate to the 'Configuration' section. ➡️ Click the Profiles button to see a list of saved profiles. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiwTBlSuHtyBRk2iUZ%252F-MNiwvGaHXkWRB3lADOs%252FSTF-12_csl_stf_click_profiles_1.png%3Falt%3Dmedia%26token%3D69628b43-8cf3-415d-baa9-d013fa4683ab&width=768&dpr=3&quality=100&sign=e63906c4&sv=2) ➡️Click **View and Apply** to preview. If a profile contains exposure groups, it will create a new exposure group in the session. ➡️Click **Apply Profile** to load the profile into the current Session. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiwTBlSuHtyBRk2iUZ%252F-MNiwyS7vQwcEnaOzLk_%252FSTF-13_csl_stf_apply_saved_prof_1.png%3Falt%3Dmedia%26token%3De07e34c9-e2ca-4823-a0cf-63121d6dfeec&width=768&dpr=3&quality=100&sign=29aafbe5&sv=2) [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-3.-start-session-and-view-exposures) 3\. Start Session and View Exposures ------------------------------------------------------------------------------------------------------------------------------------------------------------ > As the session progresses, view diagnostic plots and metadata for each exposure. Start Manual Picking anytime, and optionally Reject or Reprocess individual exposures. Once you have configured and saved all required parameters and exposure groups, the Session can be started anytime. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiwTBlSuHtyBRk2iUZ%252F-MNix15bp3Hdm6nGKGrw%252FSTF-14_csl_stf_start_anytime_1.png%3Falt%3Dmedia%26token%3D52693cde-9631-4b1c-acc9-773e0c19a67b&width=768&dpr=3&quality=100&sign=7e799e47&sv=2) ➡️**Step 1: Start Session** ➡️Click **Start Session** (from the Configuration Tab, or from the Header). ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiwTBlSuHtyBRk2iUZ%252F-MNix44qV45j1m3tHBxO%252FSTF-15_csl_stf_start_button_2.png%3Falt%3Dmedia%26token%3D77f0613c-1167-47df-a05b-c019e047b16e&width=768&dpr=3&quality=100&sign=476b1125&sv=2) As exposures are captured, detected, and loaded into CryoSPARC Live, a visualization/thumbnail for each appears sequentially from left to right in the top bar. Near the bottom of each thumbnail, a blue progress bar indicates the status of pre-processing (motion correction, CTF estimation, picking, extraction, etc). Rejected exposures are indicated with a red “X”. Exposures are processed in they order they are found, as follows: patch motion correction (global and local) > patch CTF estimation > particle picking > particle extraction. Exposures being processed are indicated with a flashing blue outline and a blue progress indicator labelled with the current processing stage (e.g., Motion Correction). ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiwTBlSuHtyBRk2iUZ%252F-MNix73nO4PLE2ZBjAQH%252FSTF-16_csl_stf_exp_incoming_1.png%3Falt%3Dmedia%26token%3Da11d1cee-f480-4ae3-bfe0-30a50162249e&width=768&dpr=3&quality=100&sign=2ffd0635&sv=2) Once a session is started, you can see sessions statistics at a glance in the sidebar on the left. This includes the number of exposures found, processed, accepted, rejected, and the rates at which exposures are being found and processed. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiwTBlSuHtyBRk2iUZ%252F-MNixDtmu_p8FIDC4QXf%252FSTF-17_csl_stf_statistics_2.png%3Falt%3Dmedia%26token%3Da8314cf2-6990-4fd6-9d85-8aa3dffcdfd6&width=768&dpr=3&quality=100&sign=8b26edfb&sv=2) ➡️**Step 2: View Individual Exposures information** Clicking on the **Individual Exposure Tab** will bring up diagnostic plots for the exposure that is currently selected in the Exposure Feed. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiwTBlSuHtyBRk2iUZ%252F-MNixGnlDMcM6c-8jVwk%252FSTF-18_csl_stf_indiv_exp_tab_1.png%3Falt%3Dmedia%26token%3D97b8d50a-8680-41a8-a2ac-db11f989c656&width=768&dpr=3&quality=100&sign=96c14a6d&sv=2) You can traverse the Exposure Feed by clicking on a thumbnail or using the navigation buttons at the top. The arrows allow for traversing through exposures including the ability to follow the latest exposure, or navigate to a specific exposure using its ID. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiwTBlSuHtyBRk2iUZ%252F-MNixLM6i--SPbL5NXPA%252FSTF-19_csl_stf_exp_feed_nav_1.png%3Falt%3Dmedia%26token%3D8f185e5d-ec7b-44a3-b56b-ce12728bc920&width=768&dpr=3&quality=100&sign=9e3d05f8&sv=2) **Clicking on the |<-- button will follow the latest incoming exposure.** On the right side of the **Individual Exposure Tab** is the Exposure Viewer. You can adjust the Low Pass Filter (LP Filter) slider (top right of the exposure canvas) as required. Buttons are available in the bottom left for zooming, panning and resetting the view. If an exposure has failed processing for any reason, the corresponding **traceback/error message** will be displayed in the Individual Exposure Tab. ➡️**Step 3: Reject individual exposures and/or reset failed exposures** If you wish to **Reject** an individual exposure, click on the dropdown menu above the Exposure Viewer and click **'Reject'**. Alternatively, use the keyboard shortcut by pressing **"R"** on the keyboard. A red **'M'** (for manually rejected) will appear on the thumbnail of any manually rejected exposure. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiwTBlSuHtyBRk2iUZ%252F-MNixOktjbR9M65xI83x%252FSTF-20_csl_stf_man_reject_1.png%3Falt%3Dmedia%26token%3D34f50775-31ad-42a2-8651-b06866c88131&width=768&dpr=3&quality=100&sign=b3105a07&sv=2) To un-reject an individual exposure, or to reset an exposure that has failed, click **'Unreject'** or **'Reprocess exposure'** from the same dropdown menu. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiwTBlSuHtyBRk2iUZ%252F-MNixSFB8TxlLiZ3RRr2%252FSTF-21_csl_stf_unrej_1.png%3Falt%3Dmedia%26token%3D467c618c-9f31-435b-8705-218a5a61cf46&width=768&dpr=3&quality=100&sign=24583644&sv=2) **➡️Step 4: Modify Exposure Processing Priority** The priority in which exposures are processed by the preprocessing GPU worker(s) can be modified at any time. For more information, see the following section: [Live Jobs and Session-Level Functions](https://guide.cryosparc.com/live/live-jobs-and-session-level-functions) [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-4.-exclude-poor-quality-exposures-from-downstream-processing) 4\. Exclude Poor Quality Exposures from Downstream Processing -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > The **Overview Tab** contains time-series plots depicting the evolution of computed data attributes. Use these to optionally exclude groups of exposures based on various thresholds. The [**Overview Tab**](https://guide.cryosparc.com/live/ui-overview#overview-tab) contains a number of plots that are useful for assessing the quality of processed images. You can hover over any individual exposure (dot) on any graph to view a small thumbnail and other details of the exposure. Clicking on any dot will cause the selected exposure to be displayed in the exposure viewer and selected in the feed. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiwTBlSuHtyBRk2iUZ%252F-MNixWrKnCztzTg6S-XA%252FSTF-22_csl_stf_view_thumb_1.png%3Falt%3Dmedia%26token%3D3d596c3e-f1c5-4712-95fa-44508f741c7e&width=768&dpr=3&quality=100&sign=d1cf4948&sv=2) Use the threshold sliders or the input fields to optionally exclude groups of exposures that do not meet desired criteria. For example, you may wish to exclude exposures with a calculated CTF Fit (Å) value greater than 4Å from downstream processing in Live (i.e., further particle picking, inclusion of the corresponding particles in 2D classification and refinement). The exposures that are rejected in this way will still be available for later downstream processing in CryoSPARC or for export - they are not deleted. You can apply, adjust and clear thresholds **at any time** during a Live Session. Particles from accepted micrographs will be used for downstream 2D and 3D processing. If a micrograph is rejected, its particles will be removed from consideration downstream and outputs that depend on particles such as the 3D refinement density map and FSC curves will be re-calculated at the next update. ➡️There are two ways to adjust a threshold on a particular attribute of micrographs. **Graphically**: Click on **'Select Thresholds'** in the bar above the overview plots to set the plot mode to selection. Then, click and drag over a vertical selection of the plot for the attribute you wish to filter, to select a range that includes the data you wish to **keep**. When you release the mouse button after dragging, you will see your selection appear on the graph as a **green** highlight, and the threshold slider above the attribute will change to the new values you have selected. **Manually**: Use the threshold sliders or input boxes to enter the exact values you wish to use for filtering for a given attribute. You will see a highlighted region appear on the graph in **green** corresponding with the slider position. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiwTBlSuHtyBRk2iUZ%252F-MNix_ZztLAMCidejc4R%252FSTF-23_csl_stf_thresh_green_1.png%3Falt%3Dmedia%26token%3D61abb04c-4042-4473-bbcf-f317f6fca55a&width=768&dpr=3&quality=100&sign=70e73e33&sv=2) ➡️Once you see a region highlighted in **green**, you can finalize your selection by clicking **'Set Threshold'** for that attribute. Otherwise you can click 'Cancel'. Once you set the threshold, all existing micrographs outside the set range will be **threshold rejected** as will any new micrographs that are processed. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiwTBlSuHtyBRk2iUZ%252F-MNixcLuzE2a_5tqoTTl%252FSTF-24_csl_stf_thresh_applied_1.png%3Falt%3Dmedia%26token%3D3bf179a5-8de1-4a8a-8d95-28aae0165dad&width=768&dpr=3&quality=100&sign=e29ed517&sv=2) In the exposure feed, rejected exposures will have a reject icon. You can apply thresholds on multiple attributes, e.g., on both CTF Fit (Å) and on Defocus Range (Å). Exposures that are rejected for any reason will show as red dots in all graphs. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiwTBlSuHtyBRk2iUZ%252F-MNixhQH7UI_Cjs2xhZu%252FSTF_25_csl_stf_rej_for_any_reason_1.png%3Falt%3Dmedia%26token%3Dd81479fe-0a37-433f-9023-1830adbf15ce&width=768&dpr=3&quality=100&sign=d1e1e823&sv=2) ➡️**To clear thresholds**, click **'Clear'** on any attribute and this will reset the exposures that were previously rejected. It is possible to both manually reject and threshold reject an exposure. [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-5.-browse-and-download-exposure-stats) 5\. Browse and Download Exposure Stats ---------------------------------------------------------------------------------------------------------------------------------------------------------------- > The **Browse Tab** contains various exposure statistics that can be plotted, compared and downloaded. At any time during a session, you can view exposure statistics, filter values and download a `.csv` file containing all of the data. ➡️Navigate to the [**Browse Tab**](https://guide.cryosparc.com/live/ui-overview#browse-tab) to view statistics and scatter plots comparing any two attributes. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiwTBlSuHtyBRk2iUZ%252F-MNixkuWDu4stdQEG-5p%252FSTF_26_csl_stf_browse_tab_1.png%3Falt%3Dmedia%26token%3D766f19a3-346e-407d-8247-f7cb28714ddd&width=768&dpr=3&quality=100&sign=2074b1fa&sv=2) [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-6.-fine-tune-particle-picking) 6\. Fine-tune Particle Picking ------------------------------------------------------------------------------------------------------------------------------------------------ > Perform manual, blob or template-based particle picking from the **Picking Tab** as the session progresses, with the option to set new particle score thresholds and re-extract particles during preprocessing, which will be fed into downstream steps. ### [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#default-blob-picker) Default: Blob Picker The Blob Picker is the default picker enabled in CryoSPARC Live for a new session. The CryoSPARC Live GPU Workers will perform blob-based picking and extraction on all incoming movies using the `Minimum particle diameter (A)`, `Maximum particle diameter (A)` and `Extraction box size (pix)` that you provided on when configuring the Session, until and unless Blob Picker settings are adjusted or the active picker is changed. ➡️**Step 1: View Blob Picks** Blob picks **"B"** are displayed in yellow in the exposure viewer. Picking statistics are displayed on the Blob Picker button and can also be viewed by clicking on the dropdown. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiwTBlSuHtyBRk2iUZ%252F-MNixrr_sCRxufyjSLZd%252FSTF_27_csl_stf_blob_picker_1.png%3Falt%3Dmedia%26token%3D894d6273-058f-4e0a-9ee0-d26047fe2a5e&width=768&dpr=3&quality=100&sign=c12e755c&sv=2) To hide or view blob picks, click on the Blob Picker button at the top of the Exposure Viewer. By default, CryoSPARC Live displays pick locations as dots. **Shift + Click** on the Blob Picker button at the top of the Exposure Viewer to cycle through circular, square and dot pick markers. You can also adjust how picks are displayed using the expansion menu on the right of the picker buttons. **For a full list of keyboard shortcuts, click on the Main Menu in the top left corner of any Session.** ➡️**Step 2: Adjust Blob Picking Parameters** As more exposures are processed, you may wish to adjust Blob Picker parameters to obtain better picks. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiwTBlSuHtyBRk2iUZ%252F-MNixuzWjh2ZneWpYhTM%252FSTF_28_csl_stf_blob_picker_adjust_1.png%3Falt%3Dmedia%26token%3D647ed184-5db3-4031-baaa-4cc448195d97&width=768&dpr=3&quality=100&sign=7bd3820&sv=2) You can adjust the `Minimum particle diameter (A)`, `Maximum particle diameter (A)`, blob shape (circular, elliptical, blob, or a combination of them), `Lowpass filter to apply (A)` and the `Min. separation distance (diameters)` at any time during a session. ➡️Adjust the value(s) as desired and then click **Activate for All** or **Activate for Future** in order to trigger re-picking and re-extraction based on your changes. Or, use Test Adjustments, explained next. * `Activate for All`: Applies the thresholds and. any changed parameters to all exposures (i.e., will cause all exposures to be re-processed with the changes) * `Activate for Future`: Applies the thresholds and any changed parameters to all exposures that have not yet been processed. Useful if you do not want your changes to trigger reprocessing. ➡️**Step 3: Test Adjustments (Optional)** If you are not sure how your parameter changes might affect processing or if you would like to experiment, you can use **'Test Adjustments'** which will cause only the active (currently selected) exposure to be re-processed with the new picking parameter changes, and re-extracted. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiwTBlSuHtyBRk2iUZ%252F-MNixyGpjJ3ZrkReNGYt%252FSTF_29_csl_stf_test_1.png%3Falt%3Dmedia%26token%3D61b3423a-2649-4574-ad0b-0c58f42c0a8a&width=768&dpr=3&quality=100&sign=45691670&sv=2) After clicking 'Test Adjustments' you may have to wait a few seconds or minutes until one of the CryoSPARC Live GPU Workers becomes available to pick up the test micrograph. Once this process is complete, the new pick locations will appear on the active micrograph. Exposures to which a **Test** parameter has been applied are indicated with a purple "**T"** on their respective thumbnails. You can apply '**Test Adjustments'** on as many individual exposures as you like. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiwTBlSuHtyBRk2iUZ%252F-MNiy13076zb_8gjYb-Q%252FSTF_30_csl_stf_test_T_2.png%3Falt%3Dmedia%26token%3D88af0846-3048-4e42-ad85-f6a96f42f29a&width=768&dpr=3&quality=100&sign=462301da&sv=2) ➡️Once satisfied with the new picker settings, click **'Activate for All'** or '**Activate for Future'** as desired. This will trigger re-picking and re-extraction. Unless one of the **Activate** buttons is clicked following **Test**, the exposure on which **Test** was run, will simply be excluded from any further processing (i.e., from particle extraction and steps downstream). To undo **Test** mode on a particular exposure, i.e., to reset it so that it can be included in further processing, click on the dropdown above the Exposure Viewer and click Reprocess exposure. ➡️**Step 4: Filter and Extract Blob Picks** Along with picker parameters, you will need to adjust the Normalized Cross-Correlation (NCC) and Power Score thresholds, similar to `Inspect Picks` in CryoSPARC, to adjust and exclude poor quality picks. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNiyAqoxwQfmBQGy1mv%252FSTF-31_csl_stf_filter_extract_blob_1.png%3Falt%3Dmedia%26token%3De4220c7d-2688-4e6b-86f8-b709fa306880&width=768&dpr=3&quality=100&sign=a1c2c7c3&sv=2) ➡️Adjust the thresholds or input new values as desired and then click **Confirm Thresholds** or Cancel if you wish to revert to the previous setting. As you drag the sliders, you will see picks appear and disappear in real time on the exposure viewer. Typically, you will want to adjust the NCC and power sliders until only "true" particles visually remain. For the Blob Picker, the power scores are often most useful. Removing low-power picks will exclude picks in empty patches of ice, smaller contaminants, and broken particles. Removing high-power picks will exclude carbon edges, ice crystals, gold particles, overlapping particles, etc. ➡️Click **Extract for All** or **Extract for Future** to trigger re-extraction. The extraction step will attempt to extract all particles falling within the new thresholds, and will remove particles that are too close to the edges of the exposures. Note that if you make changes to 'Adjust Blob Picker' parameters and 'Filter and Extract Blob Picks' extraction thresholds, **both** 'Activate for All' and 'Extract for All' buttons will become available. Clicking **either** button will cause **both** new settings to be applied. ### [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#manual-picker) Manual Picker The Manual Picker can be engaged anytime during a Session. To pick particles, navigate to the Exposure Viewer, click the "arrow+" button in the bottom right and begin selecting particles by left-clicking. To remove manual picks, either right-click, or click the "arrow-" button and then left-click over the picks you wish to remove. Manual picks are displayed in a table on the Manual Picker tab. Clicking on a table row will navigate to the exposure indicated. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNiyENeBn2CnNdsgGdO%252FSTF_32_csl_stf_man_picks_on_exp_1.png%3Falt%3Dmedia%26token%3De61dc4cd-8c19-41f6-a2e6-abcb397046d4&width=768&dpr=3&quality=100&sign=75316526&sv=2) ➡️To extract Manual Picks, click **Start Manual Extraction** on the Manual Picker tab. These picks can be fed into the Template Picker (by generating templates, below), or simply exported. ### [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#template-picker) Template Picker The Template Picker can be engaged anytime after starting a Session. To do so, you can either choose to generate templates from available blob picks or manual picks in the session, or load in available templates from any existing CryoSPARC Project/Job. ➡️**Step 1: Create or Load Templates** You can either generate templates using **2D Classification from Latest Exposures** or **Load Existing Templates**. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNiyIOYWD0-_-vE9EEW%252FSTF_32_csl_stf_template_picker_1.png%3Falt%3Dmedia%26token%3Dbfc8da32-99c2-44a0-adab-1ac86a581d2c&width=768&dpr=3&quality=100&sign=441727f2&sv=2) ➡️To **generate templates**, specify: * Classes (number of classes for 2D classification) * Exposures (particles from this many exposures will be included) * Classify (whether to classify the existing Blob Picks, Template Picks or Manual Picks) ➡️Click Generate Templates, which will start a 2D Classification job on the Auxiliary Lane that will classify the particles. Once complete, you will be able to select the desired classes below to use as templates for the Template Picker. ➡️To **Load Templates** that you already have available, enter the CryoSPARC Project ID and Job ID corresponding to the available particles/templates. These will be loaded and then they can be selected. If you have already started Streaming 2D Classification in CryoSPARC Live (eg using blob picks) you can enter the project ID and job ID of the streaming 2D class job to use the current templates to instantiate the template picker. ➡️**Step 2: Select Templates** Select as many templates as desired by clicking on each class. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNiyMPhnzahEo6tpZUU%252FSTF_33_csl_stf_templates_selected_1.png%3Falt%3Dmedia%26token%3D32158a3d-ca7e-415f-93b8-be5207b67f8c&width=768&dpr=3&quality=100&sign=8244d541&sv=2) ➡️**Step 3: Adjust Template Picker** ➡️Enter (or adjust) the Particle Diameter (A) **(required)**. If desired, adjust the Lowpass filter to apply (A), Ang. sampling (degrees) and Min. separation dist (diameters). The particle diameter should be set equal or larger than the longest expected dimension of the particle. You can Test Adjustments if desired, and then click **Apply to All** or **Apply to Future** to trigger template-based picking using your parameters. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNiyR55v7jIuqplINfb%252FSTF_34_csl_stf_triggered_template_1.png%3Falt%3Dmedia%26token%3Dcc139dad-70e6-4f4e-807d-cb771596231c&width=768&dpr=3&quality=100&sign=309c4757&sv=2) ➡️**Step 4: Filter and Extract Templates** You will need to adjust the Normalized Cross-Correlation (NCC) and Power Score threshholds, similar to `Inspect Picks` in CryoSPARC, to adjust and exclude poor quality picks. See the [similar section for Blob picking here.](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#default-blob-picker) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNiyVElfDTJ0Z_qWX8y%252FSTF_35_csl_stf_adjust_temp_thresh_1.png%3Falt%3Dmedia%26token%3D392236f4-44da-41de-be3e-fc8eeda656f8&width=768&dpr=3&quality=100&sign=e894f99a&sv=2) ➡️Adjust the thresholds or input new values as desired and then click **Confirm Thresholds** or Cancel if you wish to revert to the previous setting. ➡️Click **Extract for All** or **Extract for Future** to trigger re-extraction. The extraction step will attempt to extract all particles falling within the new thresholds, and will remove particles that are too close to the edges of the exposures. [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-7.-start-streaming-2d-classification) 7\. Start Streaming 2D Classification -------------------------------------------------------------------------------------------------------------------------------------------------------------- > 2D classification is performed seamlessly while a session is in progress. Newly available particles can be classified into existing classes as they are extracted, or 2D classification can be re-started at any time. You can start Streaming 2D Classification at any time once you have started the session. You may wish to start 2D Classification early on into a session to see more about the visual quality of the particles extracted thus far and make choices about tuning picking parameters or about the sample itself. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNiyaeorCTveI8s6Eo-%252FSTF_36_csl_stf_start_2d_1.png%3Falt%3Dmedia%26token%3Db2e444e9-9175-4984-8e10-f15e89c2f037&width=768&dpr=3&quality=100&sign=223b97e7&sv=2) ➡️**Step 1: Start Streaming 2D Classification** ➡️On the 2D Classification Tab, click the **Gear** icon and enter the number of Classes. * Alternatively, you can **Build with Custom Parameters** by clicking on the "hammer" button. This will create a new Streaming 2D Classification job in the CryoSPARC Project where the Live Session is housed. * **Navigate to the CryoSPARC Project** and find the new job (set to Building status). Enter any custom parameters you wish to change in the Job Builder. **DO NOT launch the job from the regular CryoSPARC interface.** * **Return to the CryoSPARC Live interface > 2D Classification Tab** and click Queue to launch the job. ➡️Click Start. The job will start and take in all the particles that have been extracted so far. The most recently extracted set of particles from each micrograph will be used. Therefore if a micrograph was originally picked with the blob picker, then re-picked with the template picker, the template picked particles will be used. ➡️**Step 2: Select 2D Classes** ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNiyeGEaFhXt6A8IUIC%252FSTF_37_csl_stf_strm_2d_waiting_2.png%3Falt%3Dmedia%26token%3D85500be7-1da0-413d-a6e5-d40cee769b4e&width=768&dpr=3&quality=100&sign=19a8cb2c&sv=2) Once the first 20 iterations of classification are complete, the job will display interactive 2D classes that can be selected. These displayed classes will update as new particles arrive. ➡️Click to select the desired class averages. Use the buttons above the class averages to sort and select, or right click on any class average to display a menu of actions. All particles falling within the selected class averages will be used for Ab-Initio Reconstruction and Streaming Refinement. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNiyiP5UYs3fMiBAQlx%252FSTF_38_csl_stf_strm_2d_actions_1.png%3Falt%3Dmedia%26token%3D8272da7e-c3a3-4d97-a15c-fd8866f3d2bb&width=768&dpr=3&quality=100&sign=635f9535&sv=2) **2D Classes will continue to update with new particles every few minutes, as new particles arrive.** The rate of updating depends on how fast new particles are coming and, and how long it takes to update classes. Once 10,000 new particles are seen, Streaming 2D classification will also go back and re-classify all existing particles into the updated classes, which can take several minutes. Since the 2D templates that are resolved will change only slowly as new particles are seen, the selection of 2D classes that are made will persist over updates of streaming 2D classification. However, it can be a good idea to return to the streaming 2D class tab periodically to check if any classes should be newly selected or unselected. While waiting for new particles, the Streaming 2D Classification job will enter `Waiting` status. ➡️**Step 3: Stop, Re-Start or Attempt Resume (Optional)** ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNiymJcEq9w7v3py1Uc%252FSTF_39_csl_stf_strm_2d_gear_2.png%3Falt%3Dmedia%26token%3D5063382f-02d7-432a-a9ca-423002c7ce0e&width=768&dpr=3&quality=100&sign=be6b14d2&sv=2) **Stop** At any time during 2D Classification, you can also choose to **Stop** the job. This will kill the Streaming 2D Classification job. If you wish to start another Streaming 2D job from scratch, you will need to configure the number of classes and click **Start**. **Re-Start** After streaming 2D class has been stopped or killed, you can "force" Re-Start 2D Classification. This will start off 2D classification from scratch without reference to previous results. **Attempt Resume** Alternatively, if the number of classes has not changed, you can attempt to resume streaming 2D classification from the previous results that are currently displayed. This will work if the previous 2D class job was killed or stopped after writing out some results, and if the particle box size, pixel size, and number of classes has not changed. If resuming fails, 2D class will start from scratch. [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-8.-start-ab-initio-reconstruction) 8\. Start Ab-Initio Reconstruction -------------------------------------------------------------------------------------------------------------------------------------------------------- > On the **Ab-Initio Tab**, generate an initial model from available particles or load an initial model to use for refinement. Once you have run Streaming 2D Classification and selected some classes, the particles falling into those class averages are available for Ab-Initio Reconstruction. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNiyqWXPMTDTSi6gMgn%252FSTF_40_csl_stf_select_classes_1.png%3Falt%3Dmedia%26token%3D96f7f386-efe3-4625-8389-69140b9f2b71&width=768&dpr=3&quality=100&sign=74899dfb&sv=2) ➡️**Step 1: Configure Ab-Initio Reconstruction or Load Volume** ➡️Click on the **Gear** icon to configure Ab-Initio Reconstruction. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNiyuIvORTyJS64q9MH%252FSTF_41_csl_stf_configure_ab_initio_1.png%3Falt%3Dmedia%26token%3D57b6c7f3-e422-4d3e-84da-64e8b8b78690&width=768&dpr=3&quality=100&sign=1e8f8c2a&sv=2) ➡️Enter the `Number of Classes` if you wish to resolve more than 1 class, `Symmetry` (optional, we recommend using the default C1 as it is not necessary/recommended to enforce symmetry during ab-initio reconstruction), and the number of `Particles` you wish to use for Ab-Initio (also optional, default 100,000). Note that if you use multiple classes at this stage, you can select one as the initial model for streaming 3D refinement, **but all particles will be used for refinement**. Streaming heterogeneous refinement is not currently available in Live. * Alternatively, you can **Build with Custom Parameters** by clicking on the "hammer" button. This will create a new Ab-Initio Reconstruction job in the CryoSPARC Project where the Live Session is housed. * **Navigate to the CryoSPARC Project** and find the new job (set to Building status). Enter any custom parameters you wish to change in the Job Builder. **DO NOT launch the job from the regular CryoSPARC interface.** * **Return to the CryoSPARC Live interface > Ab-Initio Tab** and click Queue to launch the job. ➡️Click **Queue** to launch the job. Volume slices will display in the CryoSPARC Live interface and the 3D volume will be available to view in the Volume Viewer as the reconstruction progresses. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNiyz6nQfxwuEZIT_Ex%252FSTF_42_csl_stf_started_ab_initio_1.png%3Falt%3Dmedia%26token%3D617ccc3d-4830-4f6d-add2-73f81362312d&width=768&dpr=3&quality=100&sign=5ae764d5&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNiz10tvyieGrbnhyDH%252FSTF_43_csl_stf_ab_initio_progress_1.png%3Falt%3Dmedia%26token%3D1c95b1ee-58be-47e0-936e-6f34337e99f1&width=768&dpr=3&quality=100&sign=c6959bfe&sv=2) You can also view the progress of any job in Live by clicking on the job number in the sidebar or on the relevant tab, to expand the streamlog view. For example, we clicked into the Ab-Initio job from the sidebar: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNiz4-hgRRUuURhkVhb%252FSTF_44_csl_stf_ab_initio_streamlog_1.png%3Falt%3Dmedia%26token%3D27308571-d71f-4ffc-b24b-7fe3846a601d&width=768&dpr=3&quality=100&sign=4e1c7191&sv=2) **Alternatively, you can load an existing volume by entering the Project UID and Job UID corresponding to the location of the initial model in your CryoSPARC instance, and then click Load.** The volume viewer can be rotated and zoomed by dragging with the left mouse button or scrolling respectively. Holding shift and dragging with the left mouse will pan the view. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNiz9AJDbwh_X5RgwWc%252FSTF_45_csl_stf_abinit_completed_3.png%3Falt%3Dmedia%26token%3Db8721bdb-65e3-4ad2-a48c-6ee09d35d76b&width=768&dpr=3&quality=100&sign=65e8104e&sv=2) To download the volume, click 'Download map' on the bottom right hand corner of the Volume Viewer. Alternatively, you can find the volume in the CryoSPARC Project where the Live Session is housed. ➡️**Step 2: Select a Volume for Refinement** ➡️Select one volume from Ab-Initio to use for Streaming Refinement. Click on the volume slices image to select the class. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNizE8AFTh37i4bKtXq%252FSTF_46_csl_stf_select_abinit_class_2.png%3Falt%3Dmedia%26token%3D9c77bd40-2b9a-45a3-8cfb-49bdb975c62b&width=768&dpr=3&quality=100&sign=e7811789&sv=2) [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-9.-start-streaming-refinement) 9\. Start Streaming Refinement ------------------------------------------------------------------------------------------------------------------------------------------------ > Refinement in CryoSPARC Live operates in a streaming manner, taking into account new particles that become available after preprocessing. Once the Ab-Initio job has been completed (or a volume has been loaded) and one volume has been selected in the Ab-Initio Tab, you can start a Streaming Refinement job. ➡️**Step 1: Configure Streaming Refinement** ➡️Navigate to the Refinement Tab. Click on the **Gear** icon to configure refinement parameters. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNizHtiqQZ4Q0bet8iF%252FSTF_47_csl_stf_configure_strm_ref_1.png%3Falt%3Dmedia%26token%3Df6734a18-6443-4543-a7df-149928178d0f&width=768&dpr=3&quality=100&sign=44b6751b&sv=2) ➡️Specify the Symmetry for refinement if known/required. Initial volumes will be automatically aligned to the symmetry axes if symmetry is specified. * Alternatively, you can **Build with Custom Parameters** by clicking on the "hammer" button. This will create a new Streaming Refinement job in the CryoSPARC Project where the Live Session is housed. * **Navigate to the CryoSPARC Project** and find the new job (set to Building status). Enter any custom parameters you wish to change in the Job Builder. **DO NOT launch the job from the regular CryoSPARC interface.** * **Return to the CryoSPARC Live interface > Refinement Tab** and click Queue to launch the job. ➡️Click **Queue** to launch the job. Various plots will display in the CryoSPARC Live interface and the refined 3D volume will be available to view in the Volume Viewer as the reconstruction progresses. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNizM6LHs0IkdDYXlfC%252FSTF_48_csl_stf_refinement_early_iter_1.png%3Falt%3Dmedia%26token%3Dbea63fb1-4998-4189-a3a6-551dee633832&width=768&dpr=3&quality=100&sign=c72a2790&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNizOrolXSS-fFML6mo%252FSTF_49_csl_stf_strm_ref_inter_later_2.png%3Falt%3Dmedia%26token%3Df963e619-df4f-4ff4-b02d-a598cdf72939&width=768&dpr=3&quality=100&sign=42811758&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNizTKeSMyXPWDA_kTq%252FSTF_50_csl_stf_strm_iter_later_4.png%3Falt%3Dmedia%26token%3D611a526d-fcea-4200-96a1-29d691e7dace&width=768&dpr=3&quality=100&sign=9788366d&sv=2) To download the volume, click 'Download map' on the bottom right hand corner of the Volume Viewer. Alternatively, you can find the volume in the CryoSPARC Project where the Live Session is housed. **As new particles are picked, extracted and classified, they will be picked up by the Streaming Refinement job so that the refinement volume will update over time in the Volume Viewer.** Streaming 3D refinement will update the 3D map, FSC resolution estimate, and diagnostic plots repeatedly as new images are collected. When sufficient new particles are available that pass the previous filtering stages and 2D class selection, refinement will backtrack to a lower resolution map, and re-perform refinement of all particles until convergence. The rate of updating will depend on the number of particles already collected and the current resolution. While waiting for new particles, the Streaming Refinement job will enter `Waiting` status. [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-10.-end-the-session) 10\. End the Session ---------------------------------------------------------------------------------------------------------------------------- > What to do once you have finished processing in Live. ### [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#pause-session-and-auto-pause) Pause Session and Auto Pause Once you have completed processing in CryoSPARC Live for the day/dataset, you can pause the session to free up compute resources by clicking **Pause Session** in the header. Pausing will retain all Session configuration and parameters and all results. Running preprocessing workers will be allowed to finish their current exposure and then gracefully exit. Running 2D or 3D streaming jobs will be killed and marked as completed so that their latest results become available to use for further processing in CryoSPARC or resuming in CryoSPARC Live. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNizZ6mDpz9cj8k_5Ni%252FSTF_51_csl_stf_pause_session_button_1.png%3Falt%3Dmedia%26token%3D535c71d1-ea21-4e4e-b494-ae29e94c9453&width=768&dpr=3&quality=100&sign=52d29b22&sv=2) #### [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#auto-pause-v5.0) Auto Pause (v5.0+) You can also configure the session to automatically pause if no new exposures are found or processed for a specified amount of time. This will free up the compute resources in use by the session for use in other jobs or sessions automatically. Two modes of Auto Pause are available: * Standard: Pause the session immediately after the timeout has expired. This will kill any running 2D/3D streaming jobs. * Graceful: After the idle timeout has lapsed, also wait for 2D/3D streaming jobs to finish before pausing the session. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FWGOmcig0TBa5pNpwG9T1%252Fimage.png%3Falt%3Dmedia%26token%3D278eb994-b3a6-4df6-8006-dc7f09084083&width=768&dpr=3&quality=100&sign=b1a3c2b1&sv=2) In combination with the [`Delay worker startup until ready`](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#run-configuration) option (also new in v5.0+), auto pause allows you to create and start multiple Live sessions at the same time, each pointing to a different on-disk directory where microscope control software will write out new images as they are acquired. For example, in a multi-grid acquisition setting, one session can be created for each grid that will be imaged. All sessions can be started, but none will consume hardware resources until data appears in their respective directory. At that point, the session will automatically queue Live workers to process the data. Once new data stops being added to the directory, the session will automatically pause, freeing up resources for the next session to process its data. ### [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#mark-as-complete) Mark as Complete If you have finished with a Session and do not plan to return to it, you may wish to **Mark as Complete** from the header. This will set the session to completed status and separate it from running and paused sessions in the browse sessions page. A completed session can still be started again and no results are lost. For more information on Session-Level Functions, please see: [Live Jobs and Session-Level Functions](https://guide.cryosparc.com/live/live-jobs-and-session-level-functions) [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-11.-view-interact-with-outputs-and-perform-further-processing) 11\. View/interact with Outputs and Perform Further Processing ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- > Interacting with outputs and performing advanced processing. ### [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#continue-processing-directly) Continue Processing Directly For any Session, you can navigate into the CryoSPARC Project where the Live Session is housed and use outputs from Streaming 2D Classification or Streaming Refinement (e.g., particles, volumes, templates, etc), directly for further processing. Using particles from Streaming 2D classification will allow using the 2D alignments and assignments downstream (e.g., for select 2D or re-centering extraction). Using the particles from Streaming Refinement will allow using the 3D alignments downstream (e.g., for reconstruction, 3D Variability, CTF refinement, local refinement, etc). ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNizaqwM0OJtlXUJdls%252FSTF_52_csl_stf_cs_project_1.png%3Falt%3Dmedia%26token%3De881586a-0513-4636-9d04-6cd193a58ec9&width=768&dpr=3&quality=100&sign=583db317&sv=2) Note that if you create a new job in CryoSPARC and use the outputs of a Live streaming job that is still running or waiting as input, the new job will remain Queued until the streaming job enters completed status. You can force this to happen by stopping the streaming job from the Live interface. The streaming job will be killed and marked as completed. You can then start streaming processing again from Live and this will create a new streaming job. ### [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#export-exposures-and-or-particles) Export Exposures and/or Particles At any time during a Session, you can navigate to the **Details** Tab and click **Export Exposures** or **Export Particles** to cause the available exposures and/or particles to be made available in CryoSPARC. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNizeP71GlNh7XOj_Q9%252FSTF_53_csl_stf_details_tab_export_1.png%3Falt%3Dmedia%26token%3D089139fd-da25-4505-87ab-76805e5a075d&width=768&dpr=3&quality=100&sign=4340dbb4&sv=2) No data is copied in this process. Rather, you will see a new job or jobs appear in the workspace in CryoSPARC corresponding to your Live session. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNizhY4hvAcYhHnCBvL%252FSTF_54_csl_stf_export_exposures_1.png%3Falt%3Dmedia%26token%3D52f86600-3c53-404f-87d2-2316eccc0c09&width=768&dpr=3&quality=100&sign=687fc3f0&sv=2) These jobs will have outputs pointing to the data from CryoSPARC Live, and can be used as any other CryoSPARC job output for connecting to new jobs for more processing. * `CryoSPARC Live Exposure Export` Job: This job will run in the CryoSPARC Project where the Live Session is housed and will separately output accepted, rejected, and manually rejected exposures for each Exposure Group in the Session. * `CryoSPARC Live Particle Export` Job: This job will export all particles from the Live session that pass the threshold tests in picking and have been extracted. These are the same particles that would be seen by streaming 2D classification. These particles will not come with alignment or class assignment information, but do contain `location`, `pick_stats`, `blob` and `ctf` outputs. You can continue processing in CryoSPARC from the outputs of the above jobs. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNizlE5IHLDe2aMaHy1%252FSTF_55_csl_stf_export_outputs_2.png%3Falt%3Dmedia%26token%3Dd862dd5a-54d0-4768-97a5-9a6a9181b3c5&width=768&dpr=3&quality=100&sign=73e53a9d&sv=2) [](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-12.-record-notes-and-view-session-details) 12\. Record Notes and View Session Details ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ > The **Details Tab** contains session history and a handy notes features. The Details Tab contains information about the user, session directory, session-level functions and start/pause history for the session along with a notes feature that includes checklists. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-M7DGv3GkRvGGpbVPCgg%252F-MNiy7gY01UzTL3Qqqd7%252F-MNizoEFBeLcPfTabYhr%252FSTF_56_csl_stf_details_tab_notes_1.png%3Falt%3Dmedia%26token%3D45bbfd88-c162-400c-967d-35ff0f7dd7a6&width=768&dpr=3&quality=100&sign=e9f0c40d&sv=2) [PreviousUI Overview](https://guide.cryosparc.com/live/ui-overview) [NextCryoSPARC Live Tutorial Videos](https://guide.cryosparc.com/live/tutorial-videos) Last updated 5 months ago * [Start to Finish Guide](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#start-to-finish-guide) * [1\. Create New Session](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-1.-create-new-session) * [2\. Configure New Session](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-2.-configure-new-session) * [Configuration Section](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#configuration-section) * [Parameters Section](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#parameters-section) * [Exposure Group Section](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#exposure-group-section) * [Load Configuration Profile](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#load-configuration-profile) * [3\. Start Session and View Exposures](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-3.-start-session-and-view-exposures) * [4\. Exclude Poor Quality Exposures from Downstream Processing](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-4.-exclude-poor-quality-exposures-from-downstream-processing) * [5\. Browse and Download Exposure Stats](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-5.-browse-and-download-exposure-stats) * [6\. Fine-tune Particle Picking](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-6.-fine-tune-particle-picking) * [Default: Blob Picker](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#default-blob-picker) * [Manual Picker](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#manual-picker) * [Template Picker](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#template-picker) * [7\. Start Streaming 2D Classification](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-7.-start-streaming-2d-classification) * [8\. Start Ab-Initio Reconstruction](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-8.-start-ab-initio-reconstruction) * [9\. Start Streaming Refinement](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-9.-start-streaming-refinement) * [10\. End the Session](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-10.-end-the-session) * [Pause Session and Auto Pause](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#pause-session-and-auto-pause) * [Mark as Complete](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#mark-as-complete) * [11\. View/interact with Outputs and Perform Further Processing](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-11.-view-interact-with-outputs-and-perform-further-processing) * [Continue Processing Directly](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#continue-processing-directly) * [Export Exposures and/or Particles](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#export-exposures-and-or-particles) * [12\. Record Notes and View Session Details](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-12.-record-notes-and-view-session-details) Copy 32, 36, 40, 42, 48, 56, 60, 64, 70, 72, 80, 84, 90, 96, 100, 108, 112, 120, 128, 144, 160, 180, 192, 200, 216, 224, 240, 256, 270, 288, 300, 320, 324, 336, 384, 400, 432, 448, 450, 512, 576, 640, 648, 672, 720, 768, 784, 810, 864, 882, 1024, 1152, 1280, 1296, 1344, 1440, 1568, 1620, 1728, 1792, 2000, 2048 --- # Tutorial: Common CryoSPARC Plots | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots.md) . [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#particle-picking) Particle Picking ------------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#ncc-vs-power-score) NCC vs Power Score ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FPfT7LBXFYqXDzobmwww8%252Fv4-4-0-plotexpl-ncc-plots.png%3Falt%3Dmedia%26token%3D7fe9f1fe-7457-44ed-9c83-76e1164941b8&width=768&dpr=3&quality=100&sign=3b102ac2&sv=2) This 2D histogram is presented in Inspect Particle Picks as well as in CryoSPARC Live’s Picking tab. The Normalized Cross Correlation (NCC) is binned along the x-axis, and the local Power Score is binned along the y-axis. The NCC tells us how well how a particle candidate matches the template (used for picking) in terms of its shape; the value is equal to the cross correlation between the template and the patch of the micrograph at each point. It is often helpful to remove picks with low NCC scores. The power score is a measure of pixel intensity at a particular location; the value is equal to the squared amplitude of the signal, after background subtraction. Regions of the micrograph with low power score often correspond to empty patches, or false positive picks. Regions of the micrograph with high power score often correspond to aggregated proteins, nanoparticles, carbon edges, or crystalline ice. Thus, it is often helpful to remove picks with extreme (large or small) power scores relative to the dataset’s distribution. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#example-of-using-power-to-remove-ice-and-aggregation) Example of using power to remove ice and aggregation ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FyOsa0yNpzWeYZ9wqOpfP%252Fv4-4-0-plotexpl-highpowerfiltering-1410to887.gif%3Falt%3Dmedia%26token%3Db6a6dfc2-28d8-4716-af4e-81c90c50e178&width=768&dpr=3&quality=100&sign=ddba14ae&sv=2) This is an example of a micrograph’s picks, before and after removing high-powered picks (in this dataset, power score greater than 887). Note how picks from the ice region near the top left are removed, as well as picks from areas of aggregated proteins near the bottom and right side. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#example-of-using-power-to-remove-low-contrast-picks) Example of using power to remove low contrast picks ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FKbYgBa36mv9nHHUYsaLn%252Fv4-4-0-plotexpl-lowpowerfiltering-139to552.gif%3Falt%3Dmedia%26token%3De449f0b9-31cd-46c0-944f-44f0ad9d7c3c&width=768&dpr=3&quality=100&sign=b1d7cb4f&sv=2) This is an example of a different micrograph’s picks, before and after removing low-powered picks (in this dataset, picks with a power score less than 552 are removed). Note how picks from areas of particularly low contrast, containing many false positive picks, are removed. Note that if micrographs from a [Micrograph Denoiser](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/exposure-curation/job-micrograph-denoiser-beta) job are used to pick particles, the power and NCC will be different from if a raw micrograph is used. This is because Inspect Particle Picks calculates the scores against the denoised micrographs, if they are present. In either case, the particles will be extracted from the raw micrograph. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FnYAz1p9yZGId6fHnnTyJ%252Fdenoising-pick-scores.png%3Falt%3Dmedia%26token%3Db6c2074e-87d2-46bf-a5ee-2003a78f2cde&width=768&dpr=3&quality=100&sign=29d7c468&sv=2) The same particle pick locations are plotted in each heatmap. On the left, the raw micrographs were provided to the Inspect Particle picks job. On the right, the denoised micrographs were provided. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#basic-2d-plots) Basic 2D plots --------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#class-ess-effective-sample-size-histogram) Class ESS (Effective Sample Size) Histogram ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FJLBb8FSIjRWF3iASVQGB%252Fv4-4-0-plotexpl-2dclass-ess-hist.png%3Falt%3Dmedia%26token%3D6e7becf8-11fb-41e6-86f4-d40a91679fb6&width=768&dpr=3&quality=100&sign=717f055e&sv=2) This histogram shows the distribution of the effective sample size (ESS) of the class posterior distribution across particles. ESS is a measure of the ‘peakedness’ of a probability distribution. A particle with an ESS of 111 confidently belongs to only one class. A particle with an ESS equal to KKK, where KKK is the total number of classes, has a uniform probability of 1/K1/K1/K of belonging to all classes. When many particles have ESS much greater than 1 (as shown in the figure above), the classification routine is uncertain due to duplicate/overlapping classes, overall poor class quality, or incomplete classification. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#probability-of-best-class-histogram) Probability of Best Class Histogram ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F8FiBFfP5ZjyDY4yzpXWh%252Fv4-4-0-plotexpl-2dclass-best-class-hist.png%3Falt%3Dmedia%26token%3D40b242f2-3660-4941-b314-ec4d24bb60c7&width=768&dpr=3&quality=100&sign=33a81821&sv=2) This histogram shows the distribution of the maximum probability across classes for each particle. A particle with low probability in its best class has significant probability distributed across other classes (i.e., it has high class ESS), meaning that the particle’s classification is uncertain. When most particles have a probability of best class near 1.0, the particle set is confidently classified and classification has converged. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#class-averages) Class Averages ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fx1hAUqAgFzOJR09suf1U%252Fv4-4-0-plotexpl-2dclass-images.png%3Falt%3Dmedia%26token%3D0643f215-de34-4a0d-a7ee-db545b2803c5&width=768&dpr=3&quality=100&sign=b4090fcd&sv=2) This figure displays a grid of class averages with overlaid metrics. The metrics are: (1) the number of particles assigned to the class, (2) the FRC (Fourier Ring Correlation) resolution of the class, and (3) the median class ESS of particles assigned to that class. The resolution reported (metric 2) is the value at which the FRC crosses a threshold of 0.5 for each class. Classes with poor resolution contain many junk particles. Classes with high median particle ESS contain many uncertain particles, indicating that the class may be too similar to other classes, or may contain particles that should belong to several different classes. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#basic-3d-plots) Basic 3D plots --------------------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#real-space-slices) Real-Space Slices ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Ffam3NpPNX8BS9OghnnQ3%252Fv4-4-0-plotexpl-real-slices.png%3Falt%3Dmedia%26token%3Dcf8e8d6b-b84e-4f1a-b210-5f716fbd78fa&width=768&dpr=3&quality=100&sign=1baa08b5&sv=2) Three real-space slices of a 3D density. These are produced by many refinement jobs within CryoSPARC. Each subplot shows a real-space density slice along one of the coordinate planes: z-y, z-x, and y-x, respectively. The pixel colour is proportional to the scalar density value at each voxel. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#real-space-projections) Real-Space Projections ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FpJcvFGaOi0KL9fC0HuhG%252Fv4-4-0-plotexpl-projection-plots.png%3Falt%3Dmedia%26token%3Dc44930e0-a6bb-4c3b-bd95-f52febb2806d&width=768&dpr=3&quality=100&sign=2922b1e6&sv=2) Three real-space projections of a 3D density. These appear primarily in Ab-initio Reconstruction’s structure plots. Instead of slicing the density along a plane, the density is summed (i.e. integrated) along the normal to that plane, and the resulting sum is displayed, for the z-y, z-x, and y-x planes respectively. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#fourier-space-slices) Fourier-Space Slices ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FZiv7UqDS1imivd5hfwsu%252Fv4-4-0-plotexpl-fourier-slices.png%3Falt%3Dmedia%26token%3De12098dc-a540-4320-a443-dae18f1a6124&width=768&dpr=3&quality=100&sign=8587031d&sv=2) These three subplots display coordinate-plane slices of the Fourier volume. The Fourier volume is the 3D grid of complex numbers that result from applying a 3D discrete Fourier transform to the real-space density. Colours correspond to log amplitude (also called the ‘magnitude’ or ‘modulus’) of each Fourier coefficient. Note that although each Fourier component is complex-valued, only the amplitude (and not the phase) is displayed in this plot. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#guinier-plot) Guinier Plot ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FY7a1eb2U8ok3VkxCkxv8%252Fv4-4-0-plotexpl-guinier.png%3Falt%3Dmedia%26token%3D7d7e096d-8bd2-4e72-8663-2ab4200cf229&width=768&dpr=3&quality=100&sign=969e5118&sv=2) The Guinier plot displays the following: * In green: the logarithm of the ‘structure factor’ F (i.e., the logarithm of the shell-averaged squared norm of the Fourier coefficients) * In blue: the straight-line envelope function computed from the ‘B factor’. This envelope function is calculated by fitting a line to the log-structure factor between 10 Angstroms and the 0.143 FSC resolution, and the fitted B-factor is proportional to the slope of this envelope function. Some nuances about how this differs from bfactor estimation in other SPA software can be found on [our discussion forum](https://discuss.cryosparc.com/t/estimated-bfactor-differ-by-a-factor-of-two-compared-to-relion/10971/14?u=mmclean) . The envelope function models the cumulative effect of all resolution-limiting factors present in the imaging conditions. Estimating the envelope function is useful as it can be used to restore the expected power spectrum through a process called [Sharpening](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-sharpening-tools) . The envelope function itself is given parametrically by a squared-exponential falloff over frequency, with scaling factor BBB: E(d)\=exp⁡(−B4ωd2)E(d) = \\exp{\\left(-\\frac{B}{4} \\omega\_d^2\\right)}E(d)\=exp(−4B​ωd2​) as described in section 4.7 of (Glaeser et al., 2021), and (Rosenthal & Henderson, 2003). ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#orientation-plots) Orientation Plots The next two plots contain information regarding the distribution of orientations in the dataset. For a more thorough discussion of orientation-related diagnostics, including metrics to diagnose preferred orientation, refer to the [Orientation Diagnostics job](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-orientation-diagnostics) and [tutorial](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-orientation-diagnostics) . #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#viewing-direction-distribution) Viewing Direction Distribution ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fd8PqHr8UlOGCcFFzxjVy%252Fv4-4-0-plotexpl-viewdir-hist.png%3Falt%3Dmedia%26token%3D747ef5fb-baff-47c7-8d1e-f68b46403fc8&width=768&dpr=3&quality=100&sign=b3fc7c0b&sv=2) The viewing direction distribution plot is one of two plots illustrating the diversity of orientations in the dataset. Every particle has an associated viewing direction, which is understood as the direction vector of the integral projection that the 2D particle was generated from, relative to the global orientation of the 3D volume. The set of possible viewing direction vectors can be interpreted as the surface of a unit sphere, or a “globe”. Thus in the viewing direction plot, the x-axis corresponds to azimuth (analogous to longitude) and the y-axis corresponds to elevation (analogous to latitude). The viewing direction plot is a 2D-histogram that shows the number of particles with a viewing direction at a particular elevation/azimuth bin. The viewing direction distribution plot is useful for understanding the diversity of orientations present in the dataset. However, it generally cannot be directly used to infer if the dataset has preferred orientation issues, because the viewing direction distribution doesn’t directly visualize the directions along which the volume is well-sampled. The [Orientation Diagnostics job](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-orientation-diagnostics) provides a more thorough set of tools for diagnosing orientation issues. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#posterior-precision-directional-distribution) Posterior Precision Directional Distribution ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FL3a7IxBYc6l8RlshVNMR%252Fv4-4-0-plotexpl-posterior-prec-hist.png%3Falt%3Dmedia%26token%3D71c174a2-0d35-49a4-bfdd-b2b55090516c&width=768&dpr=3&quality=100&sign=e0789152&sv=2) The posterior precision directional distribution plot is another plot illustrating the diversity of orientations in the dataset. If the volume is located at the center of a large circumscribed sphere, the elevation and azimuth angles define the direction of a radial line segment pointing out from the center of the structure. The posterior precision directional distribution plot displays roughly _how many images contributed to the voxels that lie along this radial line segment._ Note that this is different from the viewing direction distribution, which shows the axis along which the particle was viewed, i.e. the axis along which the volume was projected to generate the particle. The two plots are related as follows: if the viewing direction plot shows non-zero density at a viewing direction of v, then the posterior precision plot will show nonzero density at the set of all vectors orthogonal to v, i.e. the plane with normal vector v. For a greater understanding of the geometric relation between these two plots, it is useful to gain an understanding of the [Fourier-slice theorem](https://en.wikipedia.org/wiki/Projection-slice_theorem) . A related plot of the “Fourier Sampling” displayed in the [orientation diagnostics job](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-orientation-diagnostics) is very similar to the posterior precision directional distribution plot. The difference between the two is that the posterior precision plot accounts for the loss of information induced by the CTF of the particles, whereas the Fourier Sampling plot displays purely geometric information related to the particles’ alignments. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#fourier-shell-correlation-plots) Fourier Shell Correlation Plots CryoSPARC’s automatic FSC generation algorithm changed in v5.0. The plots produced by CryoSPARC v5.0+ will not match all of the images shown here, but the advice given regarding the shape of the FSC curves and what that may mean about the data still holds. Fourier Shell Correlation (FSC) plots display the correlation coefficient between spherical shells in Fourier space. FSCs are typically calculated after applying a mask to each half-map, which excludes solvent noise and other unwanted signal that exists outside the region covered by the target molecule. CryoSPARC’s calculates the FSC using several different masks and plots a curve for each in the FSC figure. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#prior-to-cryosparc-v5) Prior to CryoSPARC v5 Prior to CryoSPARC v5, refinements plotted five curves at each iteration: * No Mask: no mask is applied to the half maps. * Spherical: A soft spherical mask starting 85% of the way to the box edge and ending at the box edge is used. * Loose and Tight: Automatically generated masks which follow the contours of the map, each with fixed dilation and soft padding amounts in Angstroms. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FmBEap0NOScUBi5hAIWxC%252Fpre-v5-iter.png%3Falt%3Dmedia%26token%3Dd82ece90-d7f2-45c9-953a-f213a8aa856f&width=768&dpr=3&quality=100&sign=ba793776&sv=2) Once a refinement converged, the spherical mask FSC was no longer plotted. Instead, an FSC corrected by [noise-substitution](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#high-resolution-phase-randomization) was added. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FjBRNScQumyEl4wcE4h2I%252Fpre-v5-final.png%3Falt%3Dmedia%26token%3Df871ed7b-7608-435e-af3a-2376d94c3859&width=768&dpr=3&quality=100&sign=16c97aaa&sv=2) #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#cryosparc-v5) CryoSPARC v5+ Starting with CryoSPARC v5, the loose and tight masks are replaced with a single resolution mask. The lines also have different line styles to aid differentiation between them. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FLn6DfnACo7b9QHn4gPji%252Fpost-v5-iter.png%3Falt%3Dmedia%26token%3Dfc644adb-e099-4862-a819-2cf2036a397b&width=768&dpr=3&quality=100&sign=7b69ce1c&sv=2) The final plot shows only the resolution mask, the auto-tightened resolution mask, and the auto-tightened resolution mask corrected by [noise substitution](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#high-resolution-phase-randomization) . ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fk4AIApbEEsUpXuqdUVlw%252Fpost-v5-final.png%3Falt%3Dmedia%26token%3D5b1f92d3-b1c2-4249-bc7c-64bd1a0c125f&width=768&dpr=3&quality=100&sign=e3e21569&sv=2) Some jobs (such as [Homogeneous Reconstruction Only](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-homogeneous-reconstruction-only) and [Local Refinement](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-new-local-refinement-beta) ) use the input mask directly for computing FSCs rather than generating a new one. The FSC plots produced by these jobs use a different color and name to highlight this fact. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FcDkHV7HPwfuvsc7R4xn0%252FJ798_fsc_iteration.png%3Falt%3Dmedia%26token%3Dcc52abfb-f68e-4d4f-826c-454d5de649ce&width=768&dpr=3&quality=100&sign=9f125970&sv=2) #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#high-resolution-phase-randomization) **High resolution phase randomization** The “corrected” FSC curve is the FSC curve obtained by following a similar procedure to that outlined by Chen et al. in their 2013 publication, [**High-resolution noise substitution to measure overfitting and validate resolution in 3D structure determination by single particle electron cryomicroscopy**](https://www.sciencedirect.com/science/article/pii/S0304399113001472) . This procedure is applied to any standard refinement algorithm, such as homogeneous refinement in CryoSPARC. As published by Chen at al, it consists of creating a second set of particles that are identical to the original dataset except with random phases beyond a certain resolution in the particle dataset; this dataset is identical to the first at low and medium resolutions, but does not contain phase information of the signal at high resolutions. Then, this dataset is to be separately refined against a reference, simultaneously with the original particle dataset. This procedure was developed as a way to measure systematic contamination (or “overfitted noise”) that has been induced by application of a mask during FSC calculation; the FSC curves from the “phase-randomized” dataset can be compared quantitatively with the FSC curves from the original refinement. While this procedure robustly detects overfitted noise that builds up over the course of a refinement, it is twice as computationally costly as a standard refinement procedure. Thus, a cheaper approximate version of the procedure has been adopted in CryoSPARC and other SPA softwares. The implemented version of high-resolution phase randomization instead only happens on the _raw half-maps_ in a refinement, for the purpose of calculating a corrected FSC curve as specified in equation (4) of Chen et al. Specifically, the way in which this corrected FSC curve is computed involves randomizing the phase of the half-maps’s Fourier coefficients beyond a certain frequency, set to be 75% of the frequency at which the tight masked curve crosses the 0.143 threshold. In CryoSPARC, the plotted “corrected” curve is coincident with the standard “Tight” masked FSC curve below this resolution. Above this resolution, the “corrected” curve is given by equation 4 of [Chen et al. (2013)](https://www.sciencedirect.com/science/article/pii/S0304399113001472) , referred to by the authors as the FSCtrue\\text{FSC}\_{\\text{true}}FSCtrue​ curve. At the phase randomization resolution, the curve often has a sharp dip which arises due to the discontinuity in Fourier structure phases. These dips are a common occurrence and are generally regarded as a positive indicator that phase randomization was correctly applied. It is important to note that this modified phase-randomization procedure means that the corrected FSC curve _does not_ reliably indicate whether overfit noise has built up during the refinement. **The corrected FSC curve can only indicate whether the mask used to compute the FSC (this is the “Tight” mask in any FSC plot) is “too tight” to reliably report resolution**. Devising improved resolution metrics is an important problem facing the overall field of cryo-EM, and a foolproof metric of resolution does not currently exist. In the figure below, three examples of FSC curves along with associated mask tightnesses are shown. The leftmost side shows an example of a corrected FSC curve that indicates a mask with good tightness has been used, with minimal shared overfitting between the half-maps. The middle plot shows an example of the mask being slightly too tight — note how the “corrected” curve drops around 3.8 Å but eventually returns to being coincident with the tight curve. Finally, the rightmost plot shows an example where the tight mask is significantly too tight, made clear by the corrected curve substantially deviating from the uncorrected (tight) curve, and remaining this way indefinitely. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FwlyJe2V6H9ysOFK22aoT%252Fmask-creation_mask-tightness.png%3Falt%3Dmedia%26token%3D47fb2796-bb8a-4450-be5d-0703633146b0&width=768&dpr=3&quality=100&sign=989ff91c&sv=2) This figure displays three examples of FSC curves along with associated mask tightnesses. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#fsc-curves-not-dropping-to-zero) FSC Curves not dropping to zero Ideally, the FSC curve should drop to 0.0 before the Nyquist limit. When this occurs, the reconstruction resolution is limited by particle image quality and not pixel or image size. If the FSC remains positive all the way to the Nyquist limit, that means the two half maps are positively correlated at the highest frequency represented in the images. There are two reasons this typically happens: particle images which have been downsampled to too small a box size, and duplicate particles. It is common practice to significantly downsample particles early in the processing pipeline. This speeds early steps during which reconstructions are not expected to achieve high resolutions. Eventually, the particle stack becomes clean enough that the resulting reconstruction achieves Nyquist at this downsampled box size. In this situation, the FSC stays very high across the entire frequency range available in the images. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fo6sroRSFJBc5X7MZY6WJ%252Fv1-fsc-downsampled-images.png%3Falt%3Dmedia%26token%3D604a4803-b6a9-440e-ae68-7684db518499&width=768&dpr=3&quality=100&sign=fc90db0f&sv=2) This FSC remains high all the way to Nyquist. This means there is likely still good information which has been cut off by downsampling. In these cases, it is highly likely that re-extracting these particles with a larger box size (i.e., with less downsampling) will improve the resolution of the reconstruction. This is because downsampling the particle images removes high frequency information. However, the high FSC value at Nyquist indicates that this higher-frequency information would likely correlate between the two maps. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fl8Yv8VYbw5cqcs1804RQ%252Fv1-fsc-fullsize-images.png%3Falt%3Dmedia%26token%3D75b490ab-bb46-415b-99b7-4486c5bc3bcf&width=768&dpr=3&quality=100&sign=bbef2711&sv=2) The same particles as the previous image, in the same poses, re-extracted to a full box size. Note that the resolution significantly improves without any further alignment, and the FSC reaches zero well before Nyquist. On the other hand, FSC curves for maps with duplicate particles remain positive all the way up to Nyquist, but have a long rightward “tail” as shown in the image below. This can occur when particle picks are too close to each other in the dataset, which may happen when combining particle picks from multiple picking strategies. Particles that are too close may become coincident after being aligned to the reference during a refinement, and if these particles are present in two different half-sets, they will break the independence assumption between half-sets and thus invalidate the reconstructions. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FmlNU7ZwJK1fLQFleRW2r%252Fv4-4-0-plotexpl-fsc-duplicate-particles.png%3Falt%3Dmedia%26token%3D5cf90421-0cf7-413b-8e80-4cfb57289eb8&width=768&dpr=3&quality=100&sign=da010e6a&sv=2) A particularly dramatic example of artifactual FSC curves arising from duplicate particles being present in both half-sets. The [Remove Duplicate Particles](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-remove-duplicate-particles) job may be used to discard particles that are too close to each other, if particle pick locations are available. Note that [Helical Processing tools](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta) in CryoSPARC address this problem differently, by explicitly placing particles with overlapping signal into the same half-set, to preserve the independence between half-sets. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#sharp-bumps-in-fsc-ctf-issues) Sharp bumps in FSC (CTF issues) Systematically incorrect CTF parameters can often manifest as oscillations in a refinement’s FSC. These are characterized as multiple oscillations in the FSC that appear like the curves in the image below. If these are observed in final refinements, it is likely that one or more of the microscope optical parameters are incorrectly specified: important parameters to check are the pixel size, accelerating voltage, and spherical aberration. This phenomenon is discussed in more detail in [RELION’s documentation](https://relion.readthedocs.io/en/release-4.0/Reference/PixelSizeIssues.html#cs-and-the-error-in-the-pixel-size) . ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fqqt2qffJvRgqXHHQ5plJ%252Fv4-4-0-plotexpl-wrong-ctf-fsc.png%3Falt%3Dmedia%26token%3D5df15b77-9e18-4e04-8c5d-5989f2497241&width=768&dpr=3&quality=100&sign=1117d0a3&sv=2) An example of artifactual FSC oscillations owing to an incorrect spherical aberration specified at movie import time. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#dip-in-fsc-due-to-disordered-regions) Dip in FSC due to disordered regions For membrane proteins with disordered regions (e.g. micelles or nanodiscs), it is common for there to be a region in the frequency band (approximately between ~9 Å and ~5 Å) where the FSC value dips lower than surrounding values. This is due to the stronger presence of disorder in those frequency bands from the lipids forming the micelle or nanodisc, which have no fixed position relative to the protein structure. Generally, this dip is an expected artefact when refining membrane proteins. An example is shown below. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FvxTXFRCgxb77thuSW9aB%252Fv4-4-0-plotexpl-fsc-8to5-dip.png%3Falt%3Dmedia%26token%3Dc701070d-fc24-4da2-8b6d-98a362090988&width=768&dpr=3&quality=100&sign=f94f4d22&sv=2) Example of a healthy FSC curve of a membrane protein, with a dip in the frequency band (approximately between ~9 Å and ~5 Å) owing to disorder. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#noise-model) Noise model The noise model used in a CryoSPARC job is a parameter of the statistical model that governs image formation. Observed images are modelled to be a tomographic projection of the underlying density at some pose, convolved with the point spread function (PSF), and subjected to additive gaussian noise. Physically, this gaussian noise is used to represent the “shot noise” induced during the imaging process in the microscope’s detector. The images, projections, and noise are all represented as two dimensional quantities, and the underlying density is represented as three dimensional. When the image formation model is expressed in Fourier space, gaussian noise is parameterized as having a diagonal covariance, and subject to the further constraint that all noise variance values are constant across pixels belonging to the same frequency band. This is equivalent to the assumption that noise is isotropic over direction, and therefore all noise models are functions of Fourier shell numbers only. This can be best visualized in the advanced noise model plot below, which shows (on the right) a 2D colour-map of the noise model plot in Fourier space; note the values being constant over a given ring. In each frequency ring, the noise variance is estimated via computing the Fourier-space “residual” in each image – this is the squared difference between the noisy raw data, and the CTF-corrupted projection of the signal. The squared residual is averaged across all images, and is further averaged across frequency-band, to produce the noise estimate. A common trend in the noise variance is an increase at high frequencies. An example of this is illustrated in the basic noise model plot, below. This is due to the effect of dose-weighting. While we expect that the noise variance in each individual movie frame is approximately _**white**_ (that is, approximately constant over different frequencies), the motion-corrected micrograph itself is comprised of a sum of movie frames, and we do not use a uniform weight over frequency or over frame when summing frames to produce a micrograph. This means that the noise variance of the **micrograph** is not expected to remain white, even if the noise variance of each movie frame is white. In all cases, CryoSPARC uses near uniform weights over frame index when averaging at low-frequencies. At high-frequencies, the weights are large for early frame indices, and small for later frame indices, which has the effect of increasing the noise variance at high-frequencies relative to low-frequencies. More information about dose-weighting schemes can be found in our [Reference-based Motion Correction documentation](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/motion-correction/job-reference-based-motion-correction-beta) . ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FQIhGJ2WjzBzimqSmMewL%252Fv4-4-0-plotexpl-refinement-noise-model.png%3Falt%3Dmedia%26token%3Dad3dcc7c-7489-4057-a326-6a053f2df45d&width=768&dpr=3&quality=100&sign=55fc9399&sv=2) Basic noise model plot (produced by many refinement jobs) — This plot shows the current estimated noise variance, σ2\\sigma^2σ2, as a function of wavelength, shown in units of Angstroms (based on the pixel size). ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F9pw2JmSPVBj8MsHIPjtq%252Fv4-4-0-plotexpl-abinit-noise-model.png%3Falt%3Dmedia%26token%3Df4520320-dca0-49ae-9dcb-77696ab1729c&width=768&dpr=3&quality=100&sign=5b1de808&sv=2) Advanced noise model plot (produced by ab-initio) — this plot is similar to the basic noise plot, but explicitly shows the difference the total noise (sigma) and the empirical error (error), either averaged per shell (left) or as a 2D projection (right). The difference between the two is a result of noise priors and regularizers (cf., Punjani (2016)). Specifically, the error plot is the result of averaging the squared residual across all processed images in the dataset, and the sigma plot is the result of further averaging across frequency-band (this can be thought of averaging across concentric circular bands centered at the plot’s origin). [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#references) References ------------------------------------------------------------------------------------------------------------------------------------- * R. M. Glaeser, E. Nogales, and W. Chiu, “4.7 B factors and map sharpening,” in Single-particle cryo-em of biological macromolecules, Bristol, UK: IOP Publishing, 2021, pp. 4-59-4–67 * P. B. Rosenthal and R. Henderson, “Optimal determination of particle orientation, absolute hand, and contrast loss in single-particle electron cryomicroscopy,” Journal of Molecular Biology, vol. 333, no. 4, pp. 721–745, 2003. doi:10.1016/j.jmb.2003.07.013 * S. Chen _et al._, “High-resolution noise substitution to measure overfitting and validate resolution in 3D structure determination by single particle electron cryomicroscopy,” _Ultramicroscopy_, vol. 135, pp. 24–35, 2013. doi:10.1016/j.ultramic.2013.06.004 [PreviousTutorial: Tips for Membrane Protein Structures](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures) [NextTutorial: Negative Stain Data](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/negative-stain-data) Last updated 18 days ago * [Particle Picking](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#particle-picking) * [NCC vs Power Score](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#ncc-vs-power-score) * [Basic 2D plots](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#basic-2d-plots) * [Class ESS (Effective Sample Size) Histogram](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#class-ess-effective-sample-size-histogram) * [Probability of Best Class Histogram](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#probability-of-best-class-histogram) * [Class Averages](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#class-averages) * [Basic 3D plots](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#basic-3d-plots) * [Real-Space Slices](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#real-space-slices) * [Real-Space Projections](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#real-space-projections) * [Fourier-Space Slices](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#fourier-space-slices) * [Guinier Plot](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#guinier-plot) * [Orientation Plots](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#orientation-plots) * [Fourier Shell Correlation Plots](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#fourier-shell-correlation-plots) * [Noise model](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#noise-model) * [References](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#references) --- # Tutorial: 3D Classification | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification.md) . Note that the 3D Classification job was substantially improved in CryoSPARC v4.0. This tutorial describes the changes and new behaviour in detail below. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#introduction) Introduction ------------------------------------------------------------------------------------------------------------------------------------ [3D Classification (BETA)](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-3d-classification-beta) is a tool in CryoSPARC v3.3+ that provides a way to distinguish discrete heterogeneous states, or classes, from single particle cryo-EM data. Namely, this job currently implements 3D classification _without alignment (i.e., realignment of particle orientations or shifts)_ through a hybrid online and batch expectation maximization algorithm. By avoiding the computationally burdensome job of realigning particles and relying on higher-order interpolation rather than zero-padding of volumes, 3D Classification can efficiently separate particles into a large number of classes for further downstream analysis at high speed and without very large GPU memory requirements. Furthermore, unlike Heterogeneous Refinement, this job does not require any 3D maps for initialization. Instead, we provide two different initialization modes that can 'bootstrap' reasonable initializations via back-projection. Finally, we also allow for a (soft) mask input to 'focus' the classification on a specific region of heterogeneity, ignoring variation that may be present elsewhere. **In CryoSPARC v4.0, 3D Classification was updated with several significant modifications to the underlying computational algorithm, accepted inputs, initial parameters, and diagnostic plots. Accordingly, this tutorial has also been modified with new salient considerations, and with analysis of two new representative datasets that demonstrate both the power and the limitations of the updated job. Please see the** [**job guide page**](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-3d-classification-beta) **for a detailed list of new 3D class features included in v4+.** ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F1PVgs7bRWLQYzPw7sSPC%252Fv4-0-0-class3D-tut-E10261-navchan-resized.gif%3Falt%3Dmedia%26token%3D7dc4d579-0007-4013-b846-b8f6d1987b57&width=768&dpr=3&quality=100&sign=9226422a&sv=2) 10 states recovered from (focussed) 3D Classification of the voltage-gated sodium channel (Xu et al., 2019). Density shown at two different thresholds. Data from EMPIAR-10261. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#usage) Usage ---------------------------------------------------------------------------------------------------------------------- The 3D Classification job has one primary input requirement: particles with orientations and shifts in the `alignments3D` key. A typical pipeline may look as follows: 1. _Particle extraction, 2D classification, motion correction, CTF computation, etc_ 1. Make sure to remove as many 'junk' particles as possible, though it may also be feasible to use the 3D Classification job itself to identify junk classes and remove associated particles 2. _(Single- or multi-class)_ `_Ab-initio reconstruction_`_;_ 3. _(Optional)_ `_Homogeneous refinement_` _**or\*\*\*\* \*\*\*\***_`_**Non-uniform Refinement**_` _to find improved alignments for a final set of particles;_ 4. (_Optional,_ _**updated in v4.0**_) _Mask generation:_ 1. Focus mask ([see our guide on mask generation in Chimera](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/mask-selection-and-generation-in-ucsf-chimera) ); 2. Solvent mask from `Homogeneous refinement` or `_**Non-uniform Refinement**_` 5. `**3D Classification (BETA)**` with particles from step 2 or 3, and a focus/solvent mask from step 4. 6. Further refinement of (a subset of) class volumes [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#salient-parameters) Salient parameters ------------------------------------------------------------------------------------------------------------------------------------------------ ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#general) **General** `**Number of classes**` 2-100+. Unlike Heterogeneous Refinement, this job can feasibly classify large datasets into a large number of classes. **For example, as of CryoSPARC v4.0, we can classify a ~1.2 million particle dataset (EMPIAR-10077) into 100 classes in approximately 8.5 hours (exclusive of final output checks) on the** [**CryoSPARC testing hardware**](https://guide.cryosparc.com/live/performance-metrics#hardware-configurations-used-for-benchmarking) **.** `**Target resolution**`2-10Å. This will define the 3D map box size and consequent Nyquist cut-off resolution. The resolution should represent a reasonable size scale at which the heterogeneity is expected, while being as low (i.e., numerically large) as possible to exclude noise. Computation time will also increase as resolution is increased. **Note that in v4.0, each class may also be low-pass filtered below this cut-off, in accordance with its computed FSC curve.** `**Use FSC to filter each class**` (default: on). Starting with v4.0, 3D classification now has the (recommended) option to filter each class using its intra-class Fourier Shell Correlation. During online EM, we use a sliding window approach to FSC filtering to avoid computing FSCs with small batch sizes. Accordingly, we apply FSC regularization at every O-EM iteration, but update the per-class FSC curves every 10th iteration (including iteration 0) using a decaying sum of sufficient statistics from the past. During F-EM iterations, FSC curves are re-computed every iteration. _**Updated in v4.1:**_ `**Per-particle scale**`(default: optimal). Starting with v4.1, 3D classification now has the option to optimize per-particle scales with respect to the fixed consensus reconstruction prior to starting classification. By default this is turned on, though upstream scales ('input') or a constant scale of 1.0 ('none') can also be selected. Refer to the [considerations below](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#effects-of-particle-scale-factors-and-anisotropic-magnification) for more information about these settings. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#online-expectation-maximization) **Online Expectation Maximization** `**Number of O-EM epochs**`**,**`**O-EM batch size (per class)**` **,** `**O-EM learning rate init**` **:** These three parameters will have coupled effects on the variety and quality of the classes at the end of Online Expectation Maximization (O-EM). For a fixed number of epochs, reducing the batch size will increase the amount of O-EM iterations, the effect of which will also depend on the learning rate. In general, if you observe unexpected class ‘collapse’ during O-EM, we suggest reducing the learning rate in 0.1 increments, and/or reducing the batch size. If the average Class ESS is not near 1 at the of O-EM, we suggest increasing the number of epochs. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#full-batch-expectation-maximization) **Full-Batch Expectation Maximization:** `**Convergence criterion (%)**`**:** this parameter determines when to stop F-EM based on the amount of particles that switch classes from the previous iteration to the current one. We find that leaving this at 2% works well in many cases. For more difficult datasets, this may need to be increased to account for some particles that switch persistently because they cannot be classified with high certainty. Alternatively, you can turn on the secondary convergence criterion discussed below. `**RMS density change convergence check**`**:** monitor the root mean square (RMS) of the difference between class volume densities across iterations. For more difficult classification tasks, this number can be quite low despite a relatively high number of class switches (e.g., 5% +). In effect, particles may shuffle around classes, but have no significant effect on the class volumes. To ensure that classes with very few particles don’t have disproportional effect on this number, we computed a weighted average across classes, with weights set based on the relative size of the class. _**Updated in v4.0:**_ Other parameters such as `**Initialization mode**` and `**Class similarity**` may also affect classification, but we have not found their impact to be significant in our testing. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#important-considerations) Important considerations ------------------------------------------------------------------------------------------------------------------------------------------------------------ #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#source-of-alignments3d) **Source of** `alignments3D` This job can classify any input particles with the `alignments3D` key set (e.g., from an ab-initio job, from homogeneous refinement, from an imported EMPIAR particle dataset, etc.), however the quality of the alignments may affect the resulting classes. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#solvent-and-focus-masks) **Solvent and Focus Masks** During the expectation step of both full and online EM, we evaluate the likelihood of each particle under the following class volume Vk←S∗(F∗Vk+(1−F)∗Vˉ),V\_k \\leftarrow S \* (F \* V\_k + (1-F)\*\\bar{V}),Vk​←S∗(F∗Vk​+(1−F)∗Vˉ), where SSS is the solvent mask, FFF is the focus mask, Vˉ\\bar{V}Vˉ is the consensus volume (computed at the outset and fixed), and VkV\_kVk​ is the volume associated with class kkk. In words: At each iteration, each 3D class volume is masked by the focus mask, and the voxels outside the focus mask are replaced with the consensus reconstruction density voxels (rather than zero). This result is then masked by the solvent mask, and the voxels outside the solvent mask are replaced with zero. If the focus mask is not supplied, we set F\=1F=1F\=1and use Vk←S∗Vk.V\_k \\leftarrow S \* V\_k.Vk​←S∗Vk​. If VkV\_kVk​ contains a micelle, and no focus mask is supplied, the heterogeneity present within the micelle itself will usually dominate the classes and prevent 3D classification from identifying biologically relevant heterogeneity. Whenever possible, we recommend supplying a focus mask. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#effects-of-particle-scale-factors-and-anisotropic-magnification) **Effects of Particle Scale Factors and Anisotropic Magnification** The class volumes produced by 3D classification may also be affected two other important elements: 1. **Anisotropic magnification** Anisotropic magnification will cause class volumes to look stretched along orthogonal axes (often resulting in a ‘wobbling’ effect when classes are animated sequentially, as below). Although 3D classification does not currently estimate anisotropic magnification within the job, it can use upstream estimates encoded in the particle stack to compensate for this effect. If you suspect anisotropic magnification is affecting your results, you can run a `Global CTF refinement` job and reconnect its output to the 3D classification job, ensuring that `**Correct anisotropic magnification**` is turned on. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FyPxYWCbjy6hB6q9eeske%252Fv4-0-0-class3D-tut-E10077-wobble2-resized.gif%3Falt%3Dmedia%26token%3D6c7a2230-83b2-4881-8e7f-db7979b44e14&width=768&dpr=3&quality=100&sign=e0cca2bc&sv=2) Anisotropic magnification causes classes to 'wobble'. 2\. **Particle Scale Factors** 🆕 **As of CryoSPARC 4.1, 3D Classification includes built-in per-particle scale optimization via the parameter** `Per-particle scale` **. By default, this parameter is set to** `optimal`. **In this mode, optimal per-particle scales are computed with respect to the consensus volume and fixed particle alignments prior to classification. This procedure, combined with some algorithmic changes in v4.1, should largely avoid the convergence behaviour described below.** **Note that in some cases it may still be beneficial to use** `input` **scales obtained via a refinement job prior to 3D classification, as these will be optimized simultaneously with alignments and may therefore differ from those computed herein (which use fixed alignments).** **Finally, in some further cases (e.g., in data with significant compositional/discrete heterogeneity), it may also be useful to set this parameter to** `none` **. In these cases, per-particle scale optimization may make it more difficult for 3D classification to separate classes with missing/additional density.** Even if two particles contain the same signal, their relative scale (i.e., mean intensity) may differ due to ice thickness and other factors. This can have a significant effect on 3D classification. Similar to anisotropic magnification, we don’t compute these scales within the 3D classification job itself, but the job can use optimal scales from an upstream job to obviate their effect. A common manifestation of unequal particle scales is dramatic ‘reshuffling’ during F-EM iterations. If you observe this, (re-)run a `Homogeneous refinement` job with `Minimize over per-particle scale` turned on. If per-particle scale factors are indeed an issue, you may observe a multi-modal scale factor histogram (as seen below). The new particle stack from the refinement job can then be input to 3D classification which will account for these scale factors. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F2oRZdUlb1fWKMR9JeIn1%252Fv4-0-0-class3D-tut-scale-classflow-shuffle.png%3Falt%3Dmedia%26token%3Ddd1c0c84-131d-4905-aa23-d69b919cf417&width=768&dpr=3&quality=100&sign=85079db3&sv=2) An example of a class flow diagram from a 3D classification job where particle scales are not equal. We often find that unequal scales cause the F-EM iterations to dramatically ‘shuffle’ the classes. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FYEtU9gCDyRTDwkrpBdEa%252Fv4-0-0-class3D-tut-scale-hist.png%3Falt%3Dmedia%26token%3D11103799-d501-46d3-a83a-7c50c2bd58ba&width=768&dpr=3&quality=100&sign=4bbd9e77&sv=2) A bimodal distribution of per-particle optimal scales computed with a homogeneous refinement job. Computing these per-particle scales prior to connecting particles to a 3D classification job can significantly affect the resulting 3D classes. **Effective sample size (ESS) and soft class assignments** The ESS is a simple measure of the extent to which a discrete probability distribution is 'dispersed.' In 3D Classification, the _class_ ESS is evaluated over the posterior of class assignments for each particle. Numerically, the per-particle class ESS is equal to the inverse of the sum of squared class probabilities and ranges from 1 to the number of classes. A value near the number of classes indicates that the class posterior is near a uniform distribution, while a value near 1 represents a \`hard' selection of a single class. _**In v4.0, we now display a histogram of class ESS values as part of the standard suite of diagnostic plots.**_ ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F1QLy1fPbKgdvyDoF862Z%252Fv4-0-0-class3D-tut-ess-hist.png%3Falt%3Dmedia%26token%3D48efa668-10fc-4ce4-b946-a9c3c211d9cf&width=768&dpr=3&quality=100&sign=b0bc8ce4&sv=2) Per-particle Class ESS Histogram displayed in 3D Classification (≥v4.0). **Weighted back-projection and further refinement** In 3D Classification, the final output volumes are constructed using a weighted back-projection with weights on each particle defined based on the class posterior. This means that although the output class particle sets are disjoint, each particle may contribute to multiple (or all) volumes. When the dataset-wide mean class ESS is near 1, this effect is minimized. Nevertheless, the volumes themselves are primarily useful as a visualization of each class, and further refinement should be done on the relevant particle sets for a final reconstruction. **In v4.0, 3D Classification includes the option to disable this ‘soft back-projection’ by turning on the parameter named** `**Force hard classification**`**.** [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#example-results-and-analysis) Example Results **and Analysis** ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ We present 3D classification results from several publicly available datasets. All volumes are visualized and animated using [ChimeraX](https://www.cgl.ucsf.edu/chimerax/) . ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#empiar-10077) [EMPIAR-10077](https://www.ebi.ac.uk/empiar/EMPIAR-10077/) _Ribosome with selenocysteine delivery in E. coli, Fischer et al. (2016)_ This data captures a ribosome complex binding with a ligand. In the original publication, 6 distinct states (see Figure below) were found. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F6yKaz6BkMf15gHNONdEW%252Fv4-0-0-class3D-tut-E10077-paper-fig.png%3Falt%3Dmedia%26token%3Db762b4ce-9641-491c-8727-a1e5a3add242&width=768&dpr=3&quality=100&sign=2375a27c&sv=2) Figure 1a, Fischer et al. (2016). #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#inputs) Inputs * _Particles_ * Part 1: 1.19 million particles from `Homogeneous Refinement` * Part 2: 1.19 million particles from `Global CTF Refinement` * _Solvent mask_ * Mask from `Homogeneous Refinement` #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#non-default-parameters) **Non-default parameters** * None #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#part-1-classification-without-anisotropic-magnification-correction) **Part 1:** _**Classification without anisotropic magnification correction**_ Without correcting for anisotropic magnification, the classes include biologically-salient conformations but also display a characteristic ‘wobble’ (discussed in important considerations above): ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FKp1vrGeOIWGfoRCzZUqa%252Fv4-0-0-class3D-tut-E10077-wobble-resized.gif%3Falt%3Dmedia%26token%3De81077ce-acc0-46e4-8f84-749b5c6d0a34&width=768&dpr=3&quality=100&sign=ab5e57e9&sv=2) Results of a 10 class 3D Classification job without correcting for anisotropic magnification Here, in another 3D classification run, this wobbling is even more pronounced and most noticeable when there is little other conformational change: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FyPxYWCbjy6hB6q9eeske%252Fv4-0-0-class3D-tut-E10077-wobble2-resized.gif%3Falt%3Dmedia%26token%3D6c7a2230-83b2-4881-8e7f-db7979b44e14&width=768&dpr=3&quality=100&sign=e0cca2bc&sv=2) Results of a (different) 10 class 3D Classification job without correcting for anisotropic magnification — here the ‘wobble’ is very evident. **Part 2:** _**Classification without anisotropic magnification correction**_ To correct for this, we ran a `Global CTF refinement` job (with `Tilt` , `Trefoil`, and `Anisotropic Mag.` fits turned on) on the particle stack: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FDcOcCUf9UrLHMYc5VUkN%252Fv4-0-0-class3D-tut-E10077-ctf-resized.png%3Falt%3Dmedia%26token%3D41b5e9f4-8aeb-4b4e-b664-37723dc0b185&width=768&dpr=3&quality=100&sign=310d5e4a&sv=2) Then, using the new particle output group, we re-ran 3D classification which produced 10 classes that were no longer stretched in the same way: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F4jjsNbJxDcRlzjYBijSb%252Fv4-0-0-class3D-tut-E10077-nowobble-resized.gif%3Falt%3Dmedia%26token%3Dc64ff4ad-abd9-4dbf-9564-7de04cb8c5f2&width=768&dpr=3&quality=100&sign=8401ea4d&sv=2) Results of a 10 class 3D Classification job correcting for anisotropic magnification Note that we still see one class (class 2) with significantly more density: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FjwhYOrYq8j9JyUjFsql7%252Fv4-0-0-class3D-tut-E10077-diff.png%3Falt%3Dmedia%26token%3D0aedc4dc-951b-48d1-a271-e33cc08b5937&width=768&dpr=3&quality=100&sign=858cf641&sv=2) This may be indicative of some particle scaling issues. We discuss one way to account for these in the next dataset. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#empiar-10697) [EMPIAR-10697](https://www.ebi.ac.uk/empiar/EMPIAR-10697/) _Human RNA polymerase III, Girbig et al. (2021)_ ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FBMYT9FHReA79OUzR1oZ5%252Fv4-0-0-class3D-tut-E10697-paperfig-resized.png%3Falt%3Dmedia%26token%3D2479a3d4-0f14-431c-83bd-462e62bf7adf&width=768&dpr=3&quality=100&sign=880db316&sv=2) Two different conformations of the clamping mechanism of the RNA polymerase, from Fig. 2c, Girbig et al. (2021). #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#inputs-1) Inputs * _Particles_ * Part 1: 166K (polished) particles from [EMPIAR 10697](https://www.ebi.ac.uk/empiar/EMPIAR-10697/) * Part 2: 166K particles from `Homogeneous Refinement` (Minimize over per-particle scale on) * _Solvent mask_ * Part 1: mask from `Homogeneous Reconstruction Only` job * Part 2: mask from `Homogeneous Refinement` #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#non-default-parameters-1) **Non-default parameters** * None #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#part-1-classification-with-equal-particle-scales) **Part 1:** _**Classification with equal particle scales**_ **As of CryoSPARC 4.1, 3D Classification includes built-in per-particle scale optimization via the parameter** `Per-particle scale` **. Please see the note above regarding updated convergence behaviour in v4.1.** We ran 3D classification with the default parameters on the imported dataset from EMPIAR 10697 consisting of approximately 166K particles. With these inputs, the job required over 30 F-EM iterations to converge below the standard threshold of 2% class switches. We often observed significant class ‘shuffling’, as you can see below. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F2oRZdUlb1fWKMR9JeIn1%252Fv4-0-0-class3D-tut-scale-classflow-shuffle.png%3Falt%3Dmedia%26token%3Ddd1c0c84-131d-4905-aa23-d69b919cf417&width=768&dpr=3&quality=100&sign=85079db3&sv=2) After 67 total iterations, the job did converge, with the following class distributions: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F9k1DOV9WINjb2iQGGLuw%252Fv4-0-0-class3D-tut-E10697-class-hist.png%3Falt%3Dmedia%26token%3Daafca344-7ecc-400f-a5d0-3ae071504a01&width=768&dpr=3&quality=100&sign=3cc230a&sv=2) At first it may seem that we’ve found a number of ‘low population’ classes. However, upon further inspection we see that many of these states are similar: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FYmgzHboi8r8khen6lp1t%252Fv4-0-0-class3D-tut-E10697-10class-no-scales-resized.gif%3Falt%3Dmedia%26token%3D60af58f0-3900-4276-aefa-49cc226c7cb7&width=768&dpr=3&quality=100&sign=9c7684b1&sv=2) Ten (of ten) classes from a 3D Classification job run on data from EMPIAR 10697, with no modification to particle scales. **Part 2:** _**Classification with optimized per-particle scales**_ Investigating this further, we see that if we run `Homogeneous refinement` on the particles with `**Minimize over per-particle scale**` turned on, we see the following scale distribution: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FYEtU9gCDyRTDwkrpBdEa%252Fv4-0-0-class3D-tut-scale-hist.png%3Falt%3Dmedia%26token%3D11103799-d501-46d3-a83a-7c50c2bd58ba&width=768&dpr=3&quality=100&sign=4bbd9e77&sv=2) This type of distribution is indicative of multiple scale ‘modes’ which will affect 3D Classification. Indeed, if we re-run the job with these scale-optimized particles, the 3D classification job converges within 2 F-EM iterations to the following class distribution: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FLpnOhXm2ebfo62c2CmCM%252Fv4-0-0-class3D-tut-E10697-new-class-hist.png%3Falt%3Dmedia%26token%3D57919301-6699-401a-a81b-9e34268cdc44&width=768&dpr=3&quality=100&sign=7ccab708&sv=2) This results in four distinct classes, animated below: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FWvjMSjtayWULzQC1Emmz%252Fv4-0-0-class3D-tut-E10697-4class-with-scales-resized.gif%3Falt%3Dmedia%26token%3Dfadd68b5-66df-4d60-9f9a-644c9bf1bbac&width=768&dpr=3&quality=100&sign=467a5b3f&sv=2) Four states recovered using 3D Classification of human RNA Polymerase III (EMPIAR 10697), after per-particle scale optimization. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#empiar-10425) [EMPIAR 10425](https://www.ebi.ac.uk/empiar/EMPIAR-10425/) _A.baumannii MlaBDEF complex bound to AppNHp, Mann et al. (2021)_ ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fc5lmbRjfz65Z80kVCEDy%252Fv4-0-0-class3D-tut-E10425-paperfig1.png%3Falt%3Dmedia%26token%3D156f7775-94c5-4119-83f1-158518d26f2f&width=768&dpr=3&quality=100&sign=80b5ffc5&sv=2) The MlaBDEF complex, from Fig 1c, Mann et al. (2021). ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FmbOBvxNMq8eOsMfr9bYr%252Fv4-0-0-class3D-tut-E10425-paperfig2-resized.png%3Falt%3Dmedia%26token%3Dd6ae653e-b5f1-42eb-91aa-04e0e4c5d4a4&width=768&dpr=3&quality=100&sign=f73ac30c&sv=2) Different MlaB binding configurations, from Supp. Fig. 2, Mann et al. (2021). #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#inputs-2) **Inputs** * _Particles_ * 80K particles from `Non-uniform refinement` (after particle picking, 2D class, ab-initio) * _Solvent mask_ * Mask from `Non-uniform refinement` * _Focus mask_ * Mask created using ChimeraX (following the CryoSPARC mask generation tutorial) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FCbtHliWCkZxvb4oPebZ3%252Fv4-0-0-class3D-tut-E10425-mask.png%3Falt%3Dmedia%26token%3Df144984b-f927-471f-8982-7bf46657d258&width=768&dpr=3&quality=100&sign=ac334ee3&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fr2Vjo1wEtSLDYMxRqBHv%252Fv4-0-0-class3D-tut-E10425-3Dmask-resized.png%3Falt%3Dmedia%26token%3D3f554549-40de-4c14-bdff-ea84a7cc5975&width=768&dpr=3&quality=100&sign=e389eafa&sv=2) Focus mask isolating the binding sites of MlaB. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#non-default-parameters-2) **Non-default parameters:** * Part 1: * Classes: 5 * Part 2: * Classes: 5 * Force hard classification: True #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#part-1-classification-with-a-focus-mask-on-mlab-binding-sites) **Part 1:** _**Classification with a focus mask on MlaB binding sites**_ Applying 5-class focussed classification on this dataset results in the following classes (after 6-FEM iterations): ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FKhVNCrE94wbIlRCUsMZL%252Fv4-0-0-class3D-tut-E10425-class-hist.png%3Falt%3Dmedia%26token%3D7d53a908-a6eb-464d-80df-87f4321e2aa6&width=768&dpr=3&quality=100&sign=6c63ea3c&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FIP54dcOc7jhhUuBjyNdt%252Fv4-0-0-class3D-tut-E10425-ess-hist.png%3Falt%3Dmedia%26token%3D936194f2-adfc-4d61-bd76-5d3ff924676d&width=768&dpr=3&quality=100&sign=18af2793&sv=2) Note the small hump around 2 in the ESS histogram. This indicates that several thousand particles still have significant probability of belonging to two classes — these particles are ‘spread’ about two volumes. When we take a look at the volumes themselves: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FOqorgyZsKXdWTxWPwqKt%252Fv4-0-0-class3D-tut-E10425-real-slices.png%3Falt%3Dmedia%26token%3Dd56b4770-3691-4a5b-8027-01e27756a1ea&width=768&dpr=3&quality=100&sign=8c791334&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FlI7oSfNfjhNjnyGUgkty%252Fv4-0-0-class3D-tut-E10425-diff-slices.png%3Falt%3Dmedia%26token%3D09021511-23f7-4ee7-b2b4-9ff8c42fed66&width=768&dpr=3&quality=100&sign=2cd3314e&sv=2) We see that there is no class that contains no MlaB units. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FanE0O4q0ojdhrlSD5fgY%252Fv4-0-0-class3D-tut-E10425-softclass-resized.gif%3Falt%3Dmedia%26token%3D452335fd-cb58-49e3-a509-8e1bfa5f654b&width=768&dpr=3&quality=100&sign=70bf692e&sv=2) Five classes from 3D classification performed on EMPIAR 10425 with a focus on mask on the MlaB binding sites. In this type of case, it might be useful to observe what happens when we turn off weighted backprojection, and instead classify particles using ‘hard classification’. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#part-2-hard-classification-with-a-focus-mask-on-mlab-binding-sites) **Part 2:** _**(Hard) Classification with a focus mask on MlaB binding sites**_ With hard classification turned on, we see a very different class distribution: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FG9lssuIJhfuzy7xV7Dz8%252Fv4-0-0-class3D-tut-E10425-class-hist2.png%3Falt%3Dmedia%26token%3De681d8b7-3465-482a-8964-be5eda6735b0&width=768&dpr=3&quality=100&sign=fb82a5fd&sv=2) A plurality of the particles are now in class 4, which is a class that may have no MlaB units (though this requires further investigation). ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F1KqyeQqmVqNHGD8scvxE%252Fv4-0-0-class3D-tut-E10425-diff-slices2.png%3Falt%3Dmedia%26token%3D44eee4ef-85f4-4a6c-80fe-3718f9c15106&width=768&dpr=3&quality=100&sign=760ad9a&sv=2) ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FNbo6sw5G2sICio9Iz4LN%252Fv4-0-0-class3D-tut-E10425-hardclass-resized.gif%3Falt%3Dmedia%26token%3D909c73b7-4c70-4930-80b9-53e84586d05a&width=768&dpr=3&quality=100&sign=340f6a08&sv=2) Five classes from 3D classification performed on EMPIAR 10425 with a focus on mask on the MlaB binding sites. In this case, hard classfication is turned on, and we see the potential presence of a ‘no binding’ class. Thus, for data where a significant potion of particles cannot be classified into a single class with certainty (i.e., their class ESS is ≥ 2), turning on hard classification may help uncover classes that would otherwise be ‘smeared’ out by this uncertainty. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#citations) Citations ------------------------------------------------------------------------------------------------------------------------------ Fischer, Niels, et al. "The pathway to GTPase activation of elongation factor SelB on the ribosome." _Nature_ 540.7631 (2016): 80-85. Girbig, Mathias, et al. "Cryo-EM structures of human RNA polymerase III in its unbound and transcribing states." _Nature structural & molecular biology_ 28.2 (2021): 210-219. Mann, Daniel, et al. "Structure and lipid dynamics in the maintenance of lipid asymmetry inner membrane complex of A. baumannii." _Communications biology_ 4.1 (2021): 1-9. Xu, Hui, et al. "Structural basis of Nav1. 7 inhibition by a gating-modifier spider toxin." _Cell_ 176.4 (2019): 702-715. [PreviousCase Study: Yeast U4/U6.U5 tri-snRNP](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp) [NextTutorial: 3D Variability Analysis (Part One)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one) Last updated 1 month ago * [Introduction](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#introduction) * [Usage](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#usage) * [Salient parameters](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#salient-parameters) * [General](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#general) * [Online Expectation Maximization](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#online-expectation-maximization) * [Full-Batch Expectation Maximization:](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#full-batch-expectation-maximization) * [Important considerations](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#important-considerations) * [Example Results and Analysis](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#example-results-and-analysis) * [EMPIAR-10077](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#empiar-10077) * [EMPIAR-10697](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#empiar-10697) * [EMPIAR 10425](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#empiar-10425) * [Citations](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification#citations) --- # Case Study: Yeast U4/U6.U5 tri-snRNP | CryoSPARC Guide For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp.md) . Overview In this tutorial we will work step-by-step through an ideal use of Local Refinement. Although we will explain the motivation behind our choice of jobs and parameter settings, the main [Local Refinement guide page](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-new-local-refinement-beta) is an excellent resource for explanations of the theoretical and practical meanings of the parameters. The tri-snRNP complex is a core component of the spliceosome. It comprises four main domains: the body, head, arm, and foot. These domains are arranged in a triskelion-like shape, with the head, foot, and body radiating from the center and the arm extending past the distal end of the body. For this tutorial we will use the clean particle set from EMPIAR-10073. This dataset was originally collected and processed by [Nguyen et al](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#references) . Note that at each step your results may not look exactly like those in the guide due to randomness inherent in the alignment algorithms. As long as your map looks similar overall, and you see similar increases in quality in the focused regions, you are on the right track. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#setting-up) Setting Up ----------------------------------------------------------------------------------------------------------------------------------------- Before beginning this tutorial, you should create a new project and a workspace within that project. Download the particle stack to a location of your choosing. Our data is downloaded to a directory called rawdata in the project directory using the command: Copy wget ftp://ftp.ebi.ac.uk/empiar/world_availability/10073/data/\*.mrcs Next, download the [STAR file](https://s3.wasabisys.com/cryosparc-test-data-dist/shiny_correctpaths_cleanedcorruptstacks.star.gz) . This file has starting poses for the particle images, skipping the initial volume generation steps. Finally, import the data using an [Import Particles](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/import/job-import-particle-stack) job. The `Particle meta path` should match the location of the STAR file, and the `Particle data path` should be the directory containing the downloaded `.mrcs` files. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#global-refinement) Global Refinement ------------------------------------------------------------------------------------------------------------------------------------------------------- To ensure that your particles were loaded correctly, plug the `Imported particles` output into the `Particle stacks` input of a [Homogeneous Reconstruction Only](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-homogeneous-reconstruction-only) job. This will use the poses from the STAR file and the imported particle images to build a 3D map, without performing any alignment. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FIW0OpxHNKdKVxJ66omNq%252Ftri-snRNP_domains.png%3Falt%3Dmedia%26token%3D6e7b9216-35bd-4fb9-9865-8d5e7dc3035f&width=768&dpr=3&quality=100&sign=b203bf3e&sv=2) A reconstruction of the input particle poses. Domains are highlighted with consistent colors throughout this case study. At a low contour, all four domains are visible. However, at a higher contour the arm and head disappear entirely, and the quality of the foot also degrades. What is a contour?[](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#what-is-a-contour) Visualization of 3D maps typically relies on “contouring”, where all the points in the map which equal a value are displayed as a surface. The selection of this value in ChimeraX is achieved by dragging the bar in the Volumes pane to the right (increase) or left (decrease). A specific value can also be entered in the map’s text box. Increasing the contour makes that surface adhere to where electron potential is higher — this typically displays better-aligned regions in higher resolution. Decreasing the contour expands the surface to regions where electron potential is lower — this typically shows blurry, poorly aligned regions. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FEulITP9K7MHIM1DV7a0L%252Ftri-snRNP_domains_high-contour.png%3Falt%3Dmedia%26token%3Dc08d9c15-d770-4cef-9555-8e676a7da379&width=768&dpr=3&quality=100&sign=de8ed9c3&sv=2) Increasing the contour causes the head and arm to fade into the background. In cases like this, where different regions of the target have dramatically different resolutions, [Non-Uniform Refinement](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-non-uniform-refinement-new) often performs exceptionally well compared to traditional Homogeneous Refinements. The poses in the STAR file were generated with homogeneous refinement, so the map may improve simply by performing a Non-Uniform Refinement instead. Plug the Homogeneous Reconstruction Only job’s particles and volume into a Non-Uniform Refinement job as inputs. Leave the mask blank to generate a dynamic mask. Leave all settings as default and launch the job. Non-Uniform Refinement outperforms Homogeneous Refinement in cases like this because, in each iteration, the map is filtered based on its local quality rather than the global quality. For more information on this algorithm and how it is implemented in CryoSPARC, see the [Non-Uniform Refinement page](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-non-uniform-refinement-new) . ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Ff0zWcormARNFDWd4jT4F%252Ftri-snRNP_non-uniform-refinement.png%3Falt%3Dmedia%26token%3Dbe797896-76c9-4acb-8af8-9231140d0f51&width=768&dpr=3&quality=100&sign=c98fa842&sv=2) Non-uniform refinement significanly improves map quality in the body and foot, but does not recover information from the head or arm. The Non-Uniform Refinement significantly improved the map, both as assessed by GSFSC resolution (4.17 Å → 3.55 Å) and by visual inspection, especially in the foot and body regions. However, the head and arm are still not visible at medium or high contours. To summarize: * We can successfully align the particle to a consensus reconstruction of with a nominal resolution of 3.5 Å * We can resolve the body and foot domains at a very low resolution, so we know they are present in the particles * When we increase the contour, the head and arm disappear, indicating they are poorly aligned The reason the head is blurred is that the particles can only align to either the head or the body or the foot — there is no pose which will perfectly align every domain of the particle. Since the body is the largest domain, that region of the particle is preferentially aligned. Relative to the body, the foot moves the least, the arm the most, and the head somewhere in between. This is why the three domains are blurred, and why we can see more of the foot than the head or arm. Local refinement solves this problem by creating a mask around a sub-volume of choice (for instance, the head). Using this mask eliminates the rest of the volume. When the search volume only contains the head, an image’s assigned pose will only improve when the head is well aligned. In other words, aligning the larger body/foot region will result in a poorer score. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F9y22Bc4WzhZ0L0zqKjf3%252Ftri-snRNP_subvolume-alignment.png%3Falt%3Dmedia%26token%3D591e730b-7067-4dfc-8c87-9bbf929fc9fa&width=768&dpr=3&quality=100&sign=526b676d&sv=2) In a global refinement, the entire volume is compared with the full particle image. Local refinement only uses a masked sub-volume. In a global alignment, it’s possible that the head would be too small to align on its own, or the masked head-only volume might incorrectly align to the foot at low resolutions. Local refinement solves this problem by incorporating pre-existing knowledge about these particles. We know the approximate pose of the head in all of our images. We use Local Refinement to fine-tune it, while not allowing the head to move so far that it aligns to the wrong domain or to background noise. Let’s proceed to a local refinement of the head domain. The first step in doing so is creating the mask we will use to select only that sub-volume. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#mask-generation) Mask Generation --------------------------------------------------------------------------------------------------------------------------------------------------- Mask generation is a complicated skill that is essential for cryoEM image processing. For more information and guidance about making and using masks, see the [Mask Creation guide page](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera) . ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#create-the-mask-base) Create the mask base To generate our mask around the head, first load the Non-Uniform Refinement result volume into ChimeraX. To smooth the map and attenuate high-frequency noise in the head, apply a Guassian filter to the map (replace #1 with the number for your input map): What is the difference between a Gaussian filter and a lowpass filter?[](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#what-is-the-difference-between-a-gaussian-filter-and-a-lowpass-filter) This command applies a Guassian filter, or blur, to map #1. In essence, every point of the map is spread out into a Guassian peak with a standard deviation of (in this case) 2. A Gaussian blur is like a low-pass filter in that it attenuates high-resolution information while preserving low-resolution information, but its shape is different from the Butterworth filter used in most low-pass applications. As such, there is no way to filter to a specific resolution (e.g., 6 Å) using a Guassian filter. However, to make a mask, we only care that high-resolution features and high-frequency noise are removed, not the exact resolution of the final map. For that reason, a Guassian filter is sufficient here. Why should masks be smooth?[](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#why-should-masks-be-smooth) Beyond the practical benefit of making it easier to generate them, it is actually important for the reliability of your results that your masks are smooth. If your mask contains high-resolution information (that is, if your mask was not blurred), it may induce spurious correlations between half-sets based solely on the mask rather than the volume themselves. This results in an inflated GSFSC resolution estimate. We therefore recommend that you always filter your mask to a resolution well below the expected GSFSC resolution of your alignment before generating them. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fybau1rBgSFYkwkRfK71J%252Ftri-snRNP_blur-comparison.png%3Falt%3Dmedia%26token%3D744b72a8-3dde-4b82-8459-0a63555eda7e&width=768&dpr=3&quality=100&sign=80e8cddf&sv=2) A comparison between the unblurred (left) and Gaussian filtered (right) maps. Removing the high-frequency noise makes it much easier to see each domain and aids in building masks that are nice and smooth. Next, we segment this map using [Segger](https://www.cgl.ucsf.edu/chimerax/docs/user/tools/segment.html) , which segments a volume using watershed segmentation. A GUI for the tool is opened with Tools > Volume Data > Segment Map. Click the “Shortcuts Options” dropdown to display buttons which run convenient commands. Select your Guassian-filtered map in the “Segment map” dropdown and click the “Segment” button, leaving all other settings as default. The results should look something like this: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2Flh7-us.googleusercontent.com%2FcjZmuaz5Jq3NObksVhSNqvzUwAbN8hysDaRUMWxTxO-ILoPmsuVwK_tz0gEwLVou23ZiRj7o4A7NV6QL1h9TG0pXwWcgn-5dkjQlg7pjJAM0vjvGBRbV1U45YS6phhJl-jZF6oJNn5XKjtd2iq3aDHk&width=768&dpr=3&quality=100&sign=9c3b79eb&sv=2) The map has been segmented into several regions, each with a distinct color. Segger has split the map into (in this case) 61 regions. You can build your mask by selectively hiding these regions, leaving only the part of the map that is to be included in the mask. * Control-clicking a region selects it * Control-shift-clicking a region adds it to the current selection * Clicking “Hide” hides a region without deleting or un-selecting it * Clicking “Show” shows a region * Clicking “Delete” deletes a region * Clicking “Ungroup” splits a region into smaller subregions * Clicking “Group” combines two or more regions into one larger region There is no undo feature in Segger! We highly recommend that you click “Hide” before deleting a region. If more is hidden than you expect, you can click “Show” and ungroup the region before trying again. If you go straight to “Delete”, you’ll have to start over! To make a mask around the head, hide all of the regions corresponding to the foot, body, and arm. The final Segger model looks like this, with the map in displayed grey: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fq7CZT47vA7dm4KGFhSIf%252Ftri-snRNP_segger-model.png%3Falt%3Dmedia%26token%3Decb0e9d5-9b10-4519-b5ff-61af1c37afd1&width=768&dpr=3&quality=100&sign=3a2f7492&sv=2) Only the Segger regions corresponding to the head remain Next, the Segger model must be converted into an `.mrc` file which CryoSPARC can read. To do this, first select the remaining regions by control-click and dragging over them. Then click File > Save selected regions to .mrc file… in the Segger panel. You can name this file anything you like. This saves an `.mrc` map file with only the selected regions included. However, it is the wrong box size! In this guide we call the volume we just generated a “mask base” because we will use it to create a mask, but it has not yet been dilated and padded, and so should not be used as a mask in any refinements. To make sure the mask base is on the same box as the input map, it must be resampled. Luckily, ChimeraX has a function to do this. In the example command below, #4 should be your mask base .mrc volume and #1 should be your original, unblurred volume. Change the numbers as necessary to match your work. Now your mask base and volume are in the same box size! To upload the mask base, it must first be saved to the local computer. In the command below, we save the mask (#5, adjust as necessary) to the desktop with a filename indicating * On what job is this mask based? * Project 300 (P300), job 3 (J3) * Which regions of the map are included in this mask base? * Head Note that this naming convention is entirely optional, you can choose any name you like. Regardless of the name, the mask base can now be uploaded to the compute system which runs CryoSPARC. In our case, mask bases are stored in separate directories per-target, but you can use any organizational scheme that helps you! ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#dilate-and-pad-the-base-to-create-a-mask) Dilate and pad the base to create a mask Back in the CryoSPARC UI, run an [Import Volumes](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/import/job-import-3d-volumes) job to import the mask base you just uploaded. You can leave everything else as default, including that we are importing a map. Since the mask base has not yet been binarized and padded, we don’t want to accidentally use it where we need a mask! ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FytqoFqqhF22kJMur2gDW%252Ftri-snRNP_J7_imported_volume.png%3Falt%3Dmedia%26token%3D35b56d33-80cc-4905-9894-08372c77af06&width=768&dpr=3&quality=100&sign=5e91a95b&sv=2) The mask base imported into CryoSPARC Binarization is the process of converting a map (which has smoothly varying values ranging from, typically, 0.0 to 1.0) to a mask that has only 0.0 or 1.0. Padding is the process of adding a soft edge to the binary mask to reduce ringing artifacts. The next step is dilating and padding the mask base to produce our final mask. Create a Volume Tools job and connect the imported volume as the Input Volume and change the following settings Parameter Value Type of output volume mask Threshold 0.05 Dilation radius 5 Soft padding width 17 These settings will binarize our mask so that everything we included during segmentation (which all has a value greater than 0.05) is set to 1.0 and everything else is 0.0. Then, the mask will be expanded with 1.0 outward by 5 pixels (7 Å). This setting is the Dilation radius. We pad the mask to make sure that all of the information in the volume is covered by 1.0 once the alignment improves and the amount of the head we can see increases. Finally, the mask is padded with a soft edge that is 17 pixels (23.8 Å) wide. This is the Soft padding width parameter. This is a bit wider than the minimum we recommend (in this case, 13 pixels), but it is generally better to start with too large of a soft edge and decrease it if alignments don’t improve. It is absolutely critical that any mask which is used to cut through map density has a soft edge. See the [Mask Creation page](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#why-do-masks-need-a-soft-edge) for more discussion of artifacts caused by masks with a hard edge. Launch the Volume Tools job. It should run relatively quickly. Once it is complete, download the result and open it in the same ChimeraX window as your map. It should cover the head but not the rest of the tri-snRNP. As you contour the mask down, it should slowly expand away from your selection. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fde2fy6qT6psq19vMfrs8%252Ftri-snRNP_contour.gif%3Falt%3Dmedia%26token%3Dde94475f-cdb3-4fac-a707-d270fc0955b2&width=768&dpr=3&quality=100&sign=94102f3f&sv=2) The mask (blue mesh) slowly expands as the contour is decreased from 1.0 to 0.0. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#important-parameters) Important Parameters ------------------------------------------------------------------------------------------------------------------------------------------------------------- Before creating our first Local Refinement job, we will cover a few commonly-changed parameters. A full discussion of all the settings is available in the [main job page](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-new-local-refinement-beta) . ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#rotation-and-shift-search-extents) Rotation and Shift Search Extents One of the major differences between Local Refinement and other types of refinement is that Local Refinement uses our existing knowledge about the particle poses, rather than starting from scratch. During each iteration, the refinement algorithm checks what the particle’s pose currently is. Then it checks the poses within a certain distance from that starting pose to see which one matches the volume best. The distance the algorithm checks is the Search Extent. For example, we know the head is not currently well aligned, but it is also not totally out of alignment. In other words, we expect that there is a moderate amount of rotation and movement, so the algorithm should check poses that are a moderate distance away from the current pose. If, however, we were aligning the body (which is already quite well-aligned), we could reduce the search extents significantly. Counterintuitively, if you are working with a small flexible domain (such as the arm), you may want to reduce the search extents even if you believe the domain is quite flexible. When the algorithm is aligning a small domain it doesn’t have much information to work with. Only letting it move a small distance from the initial alignment prevents it moving these small domains far away from the main bulk of the protein due to nearby noise. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FLxiBZgiBq99V8CzIBZHX%252Ftri-snRNP_search-extents.png%3Falt%3Dmedia%26token%3Dcfea90bb-98cd-4b2a-8133-36541260e49a&width=768&dpr=3&quality=100&sign=e95e7b09&sv=2) Poorly-aligned particles will require larger search extents than particles which are already well-aligned. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#fulcrum-position) Fulcrum Position #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#why-set-a-fulcrum) Why set a fulcrum? Consider the alignment algorithm again: 1. Mask out the subvolume 2. Search local translations and rotations for a better pose 3. Generate a new volume and repeat It’s step 2 that’s important here: when the particle is rotated, what is it rotating around? ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F0IdXvrF8gDt3VtrDorRy%252Ftri-snRNP_fulcrum-cartoon.png%3Falt%3Dmedia%26token%3Dd5647c43-58a2-4856-a747-1b1cf811e6c5&width=768&dpr=3&quality=100&sign=32cd04d&sv=2) The result of a 20° rotation depends on the fulcrum about which the object rotates. There’s no obvious best answer, so by default we rotate around the center of the mask. This works well when **the mask covers a large proportion of the total volume**. The head domain does not cover a large proportion of the volume. In this case, it might be better to rotate the particles around the center of the box, since that’s closer to the real hinging motion we expect to see. Rotation around the box center tends to work better when **the mask covers a moderately-sized proportion which rotates relative to the main volume**. But there’s no reason we have to pick one of these two points. Our intuition tells us that **the head ought to rotate around some point on the head/body interface**. In the next step you will pick a point on this interface and set that as the fulcrum. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#finding-the-fulcrum) Finding the fulcrum Look back at your mask base in ChimeraX. To set the fulcrum, CryoSPARC needs the coordinates (in pixels, counting from the corner of the box) of the point we want to rotate about. So we need to determine the coordinates of a position on the surface of the interface, which is the edge of our mask base. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FWQ0siq96RwciYamwEpDo%252Ftri-snRNP_just-head.png%3Falt%3Dmedia%26token%3Dad8bdf5f-aa45-49a9-9b83-ddc49fe3ada6&width=768&dpr=3&quality=100&sign=e9645e33&sv=2) The fulcrum should be somewhere on the interface between the head and the rest of the tri-snRNP molecule. To get these coordinates in ChimeraX: 1. Navigate to the “Markers” ribbon menu. 2. Click “Surface” in the “Place markers” group. 3. Orient the mask so that you can see the surface you want to place the fulcrum on. 4. Right-click the surface to place a marker. 5. Read the coordinates in Å from the log. 6. Divide the coordinates by the pixel size (which is available with the command `info #1`, replacing `#1` with the correct number for your map) to get the pixel coordinates. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FLiLxJdNDOvqmbTa0b3gv%252Ftri-snRNP_marker-placement.png%3Falt%3Dmedia%26token%3Db4c733de-c889-4d25-a16b-288c2a62a751&width=768&dpr=3&quality=100&sign=7368e694&sv=2) ChimeraX reports the position of markers in Å, which must be converted to pixels using the map's pixel size. In this example, the marker was placed at `(249.5, 250.1, 253.3) Å`, so the fulcrum position will be `(178.2, 178.6, 180.9) px`. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#build-the-job) Build the Job ----------------------------------------------------------------------------------------------------------------------------------------------- It is finally time to build the first Local Refinement job! Create a Local Refinement and connect the particles and volume from the Non-Uniform Refinement to the correct slots. Then connect your mask to the static mask slot. Finally, set the fulcrum you calculated in the last step. CryoSPARC expects the fulcrum in the form x,y,z, with no parentheses or spaces. For example, `178.2,178.6,180.9` Leave all the other parameters set to their default and launch the job! [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#diagnostic-plots) Diagnostic Plots ----------------------------------------------------------------------------------------------------------------------------------------------------- Local Refinement takes about as long as other refinement jobs, depending on the search extents and the quality of the initial alignment. Once the first iteration finishes, you will have access to several diagnostic plots. These plots are useful in assessing job progress and ensuring that parameters were set as expected. The first three plots that Local Refinement shows you are slices through the real space of your map, the Fourier space of your map, and the real space of your mask. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FWqgwqy1HjEEaaqGsFFSN%252Ftri-snRNP_J9_real_space_slices_iteration_000.png%3Falt%3Dmedia%26token%3Dd3955143-e738-45f4-80bb-30f45317622e&width=768&dpr=3&quality=100&sign=eba31e50&sv=2) Slices through the map in real space. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FiDxdPUUWQr1SsW0ZCgD2%252Ftri-snRNP_J9_fourier_space_slices_iteration_000.png%3Falt%3Dmedia%26token%3D89e5976d-64f8-4f24-9d91-005d478499e1&width=768&dpr=3&quality=100&sign=43f6ae0a&sv=2) Slices through the map in Fourier space. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FTB28cqrvA6WN6YSqzV9t%252Ftri-snRNP_J9_real_space_mask_slices_iteration_000.png%3Falt%3Dmedia%26token%3Dfbc85b90-05ea-4fcd-aca2-2e06c711a0b8&width=768&dpr=3&quality=100&sign=4aea80d4&sv=2) Slices through the mask in real space. These plots largely exist to give you a sense of how the refinement is progressing without having to download the map at each stage of the refinement. One annotation to note is the pair of white dotted lines in the map and mask slices, one vertical and one horizontal. The intersection of these lines shows you the fulcrum point. Note that the fulcrum looks like it’s positioned at the head/body interface as expected! ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fx1KnVKTSfqtLw8zrs0cY%252Ftri-snRNP_J9_fsc_iteration_000.png%3Falt%3Dmedia%26token%3D152acdeb-cf32-40d0-8db4-cdcd643ad2be&width=768&dpr=3&quality=100&sign=4b3a726f&sv=2) The Gold Standard Fourier Shell Correlation (GSFSC) plot. Plots of the Gold Standard Fourier Shell Correlation (GSFSC) demonstrate the correlation between the two independent half maps and determine the resolution to which we can trust our maps. It’s not uncommon for the unmasked GSFSC curve to be relatively poor during a Local Refinement. The alignment ignores everything outside the mask, which means the score of a given pose is not affected by even significant mismatches outside the mask. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F7JBA24Us3AwHtmnL2aIk%252Ftri-snRNP_J9_guinier_plot_iteration_000.png%3Falt%3Dmedia%26token%3Da54728cb-e69d-40ec-9f33-a5fb87612847&width=768&dpr=3&quality=100&sign=3d20cb80&sv=2) The Guinier plot. A Guinier plot visualizes the contribution of a given resolution shell to the final map. As discussed in [Rosenthal and Henderson](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#references) , this plot is used to determine the optimal sharpening factor (the B-factor). The “sharp” map output has this B-factor applied to it, but you can always generate maps with other sharpening factors using the Sharpening Tools job. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FaEInNWdcfZR0xpnCoomd%252Ftri-snRNP_J9_noise_model_iteration_000.png%3Falt%3Dmedia%26token%3D0759c9de-d736-4c89-814b-a1a64c7159aa&width=768&dpr=3&quality=100&sign=9d972814&sv=2) The noise model. The noise model is an important component of any cryoEM image processing algorithm. Briefly, the noise model is used to modulate the penalties associated with poor correlation in a frequency shell by the expected quality of signal in that frequency shell. Put another way, if a particular frequency is very noisy, it should not surprise us when the 3D model does not agree well with the images in that frequency. Noise models for cryoEM generally have a high peak at the low resolutions (left), rapidly drop in the moderate resolutions (middle), and steadily rise as the resolution increases. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FYPNf6osNUel3ZB7BloQ4%252Ftri-snRNP_J9_viewing_direction_distribution_iteration_000.png%3Falt%3Dmedia%26token%3D7858a43f-8064-4c00-81d6-5c2350a05347&width=768&dpr=3&quality=100&sign=90eb7b41&sv=2) The distribution of particle viewing directions. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FVP3BQmCPZFRUUTvycHvv%252Ftri-snRNP_J9_posterior_precision_directional_distribution_iteration_000.png%3Falt%3Dmedia%26token%3Dcdd11d2c-cfa2-4a1c-a8cd-654f11bebd8d&width=768&dpr=3&quality=100&sign=ead68491&sv=2) The posterior precision plot. The viewing direction and posterior precision distributions are used to determine whether a particle stack suffers from orientation bias. The direction distribution directly plots the number of particles with a given pose. The posterior precision distribution is a measure of how confident we are in the volume’s quality when viewed from each direction. As long as your lowest and highest values in this plot are within an order of magnitude, your dataset likely samples all orientations enough to avoid significant anisotropy. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FghrFq6IKBJcv17dAmII2%252Ftri-snRNP_J9_magnitudes_of_alignment_changes_iteration_000.png%3Falt%3Dmedia%26token%3D69f13057-6d5f-4393-b87e-bffba8843c3e&width=768&dpr=3&quality=100&sign=31f28de0&sv=2) The distribution of angle and shift changes. At this early iteration, shifts are high and running up against the maximum shift extent. These histograms display how much each particle moved during this iteration. In this first iteration, particles are moving a lot and (especially in the shifts), bumping up against our search extent. However, right now the map of this region is not very good (remember that the input map was lowpass filtered, so these particles are aligning to a 12 Å map). If we see large peaks at the edge of our search extents in late iterations, we will have to consider re-running the job with larger search extents. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FsKhjUwVXE5PM8q5bwYqa%252Ftri-snRNP_J9_per_particle_scale_factors_000.png%3Falt%3Dmedia%26token%3D31560fd5-a7b6-4b13-9db6-1dacfccbdd4c&width=768&dpr=3&quality=100&sign=99a255fa&sv=2) Per-particle scale distribution. In this example, per-particle scale has not been optimized, so all particles have a scale of 1.0. Finally, per-particle scale is a way of accounting for the fact that different images will have different absolute contrast due to different ice thickness, defocus, etc. Generally one should only refine per-particle scale when looking at the entire volume, so we have not refined these values. Thus, all particles are still at 1.0. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#results-local-refinement) Results - Local Refinement ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FQKl7vfcjCAdt3e9WletT%252Ftri-snRNP_J9_magnitudes_of_alignment_changes_iteration_008.png%3Falt%3Dmedia%26token%3De6462cf1-1295-4544-bf53-c7b12481ec30&width=768&dpr=3&quality=100&sign=a93d07fe&sv=2) The final distribution of angle and shift changes. Note that the early peak at the maximum shift change has disappeared. These look good: they are both smooth, and there are no large spikes at the edge of our search parameters. There are some particles shifting all the way out to 10 pixels so future similar jobs may benefit from increasing the shift search extent, but this looks fine for now! The subvolume is indeed quite flexible, with thousands of particles rotating 20° or more! Next, take a look at the GSFSC curve: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FcJK7k5ECfxyMTmPihxLo%252Ftri-snRNP_J9_fsc_iteration_008_after_fsc_mask_auto_tightening.png%3Falt%3Dmedia%26token%3D202e8b66-1e34-490a-b448-2ac995ebe2b9&width=768&dpr=3&quality=100&sign=68d89a03&sv=2) The GSFSC curves for the final iteration It is not surprising that the unmasked FSC curve is poor, since we’re only aligning a small subvolume. It’s good that all three curves are smooth and decrease all the way to zero. The mask in this case may have been a little tight — the Corrected curve does not closely track the Tight curve until higher resolutions. However, it does “catch up” eventually, so these results are likely still trustworthy. When performing Local Refinements, pay attention to the Corrected GSFSC curve. If it does not closely track the Tight curve, your mask may be too tight. See the [Mask Creation page](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera) for more information. Finally, download the sharp map. This map has the sharpening factor from the Guinier plot automatically applied and will give you a good sense of the quality of the alignment. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FKQWil5USLrBwnUOy5GaB%252Ftri-snRNP_no-sigsub_head.png%3Falt%3Dmedia%26token%3Dbb24c08b-281a-4a5d-8909-b929eecc0e10&width=768&dpr=3&quality=100&sign=d8168f5a&sv=2) The result of performing a Local Refinement with a mask around the head. This is a dramatic improvement in how much of the head is visible. Note, though, that the maps of the body and foot are much worse — when we align the head, the head/body flexibility causes these domains to blur out instead! The GSFSC resolution (4 Å) is already slightly improved over that of the published result (4.2 Å) for the head. However, there is another step which may further improve the results. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#particle-subtraction) Particle Subtraction ------------------------------------------------------------------------------------------------------------------------------------------------------------- Recall that Local Refinement aligns a masked volume to the full particle images. In this case, the masked volume is just the head, while the images contain all four domains. Many of the particle images have the body, foot, or arm directly above or below the head. When the electron beam passes through, these other domains “cast a shadow” on the image of the head domain. This can hurt the alignment, since the masked volume does not have information from these domains but the images do. To fix this problem, we can first project the volume of the foot, body and arm (but not head) for each image, and then subtract this projection from each image. This leaves images containing only the information from the head domain, the same as our masked volume. The efficacy of this technique depends on the quality of the subtracted domains’ alignment. Subtracting a blurry or flexible domain from images will leave shadows and other artifacts, which wouldn’t improve the results. Ultimately, whether Particle Subtraction helps or hurts with a particular dataset is empirical: you won’t know until you try! ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#mask-creation) Mask Creation For this job, we will make a mask using the same process as for the head, except including everything other than the head. Be sure you build this mask using the Non-Uniform Refinement, since we need the body and foot to be well-aligned! ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FiCSB3VaJSomP8pquDnp1%252Ftri-snRNP_not-head-mask.png%3Falt%3Dmedia%26token%3D23ff98d1-8da0-40b1-b811-1c484328ffc9&width=768&dpr=3&quality=100&sign=e80e9a3&sv=2) The mask for particle subtraction, which excludes the head domain. If in the future you find that you are often performing both Particle Subtraction and Local Refinement, going through the process of map segmentation twice can be irritating. To avoid this, after saving your first mask (of the region you want to keep), you can delete the regions used to create the mask. Finally, show all regions to bring back the hidden regions. You can then save these regions to make your mask for Particle Subtraction without having to re-select anything. More tips on mask generation can be found in [our guide page](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera) . For this mask you should not dilate the base at all, since you do not want to subtract surrounding noise from the particle images. A soft edge is still necessary. For this example, we used the recommended minimum soft padding of 5×resolutionapix5 \\times{} \\frac{\\mathrm{resolution}}{\\mathrm{apix}}5×apixresolution​ which works out to be 12 pixels. Plug the particles and volume from the Non-Uniform Refinement into a Particle Subtraction job along with the mask you just made. All default parameters are fine, so go ahead and launch the job! Once the job completes, we recommend performing a Homogeneous Reconstruction Only job to ensure that the results are as expected. This step is optional, as you won’t use the resulting map for anything, but it only takes a few minutes to run and can save you a lot of time if you catch a failed subtraction before running a whole refinement! The reconstructed map from the subtracted particles in this example looks like this: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FtO1Ixc3iXa4wwitl3byg%252Ftri-snRNP_labeled_subtracted-recon.png%3Falt%3Dmedia%26token%3D999ec7d0-9775-4582-a80b-77cc98cc6191&width=768&dpr=3&quality=100&sign=6eea8f83&sv=2) The results of subtracting the arm, body, and foot domains. Clearly, the body and foot have been subtracted successfully. At lower contours there is still some remaining signal from the arm. This isn’t entirely surprising, since the arm’s alignment was bad to begin with. In any case, the arm is small, so we have successfully subtracted most of the volume that lies outside our mask. Use these subtracted particles in a Local Refinement to see if you can improve the resulting map. Clone your previous local refinement and replace the particles with the new, subtracted particle stack. You should also slightly increase the shift search extent to 14 A, since some particles were against the edge of the extent in the first refinement. Leave all other settings as you had them previously and launch the job. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#results-local-refinement-after-particle-subtraction) Results - Local Refinement after Particle Subtraction ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- With our masks, particle subtraction improved the GSFSC resolution by an additional 0.1 Å, which is within the realm of how much any two reconstructions might differ by chance. More importantly, signal subtraction kept the GSFSC curve higher in the middle resolutions, which has a significant impact on the overall quality of the map despite the similarity of the GSFSC resolution. When directly comparing maps with and without particle subtraction, it appears that some regions benefit greatly from subtracting away the body and foot: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F9rqBfZY4RxnRd4wlTsCH%252Fpart-sub_compare-better_smaller.gif%3Falt%3Dmedia%26token%3D48add29c-22e7-4ac5-b60d-72d71ce397b4&width=768&dpr=3&quality=100&sign=b72b4e2c&sv=2) A comparison of the local refinement with (purple) and without (cyan) particle subtraction shows significant improvement in this region, with side chains visible and more connected density in the particle subtracted map. while other regions only benefit modestly: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FpHB6IAjaqAn7ZhClJNeQ%252Fpart-sub_compare-worse_smaller.gif%3Falt%3Dmedia%26token%3Db8b216ee-e2a0-4ad7-a706-dd7ed5467052&width=768&dpr=3&quality=100&sign=c45924af&sv=2) A comparison of the local refinement with (purple) and without (cyan) particle subtraction shows only modest improvement in this region. In the end, like many other steps of a cryoEM workflow, the optimal combination of subtraction, masking, and parameters must be determined empirically for each dataset. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#conclusion) Conclusion ----------------------------------------------------------------------------------------------------------------------------------------- Local Refinement is an essential tool in the analysis of targets with rigid domains separated by a hinge. In each step of a Local Refinement, the optimal pose for a masked subvolume is found for a given set of particle images, which may or may not have signal from other regions of the particle subtracted. This leaves three major domains of optimization for the user: 1. The mask 2. Search extent and other refinement parameters 3. Particle subtraction These domains can be optimized together or independently, and often several iterations are necessary to achieve the best result. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#exercises) Exercises --------------------------------------------------------------------------------------------------------------------------------------- ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#docs-internal-guid-da9bd864-7fff-1258-2029-53da86db3392) The foot domain Perform a local refinement of the foot domain. The published map for the foot domain alone is at 3.7 Å. We were able to improve the map of the foot all the way to 3.4 Å using the same techniques as for the head domain! ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FDsbwZrY0FEIzo12FOsYD%252Ffoot-improvement.gif%3Falt%3Dmedia%26token%3D24e3af0c-925d-4188-bd5b-215aca9d68d6&width=768&dpr=3&quality=100&sign=96e96f3f&sv=2) Significant improvement of the foot domain before (purple) and after (cyan) Local Refinement. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#investigate-mask-parameters) Investigate mask parameters Make a series of masks using the same mask base but varying the dilation radius and soft padding width. Run Local Refinements with all settings the same, except using a different mask in each (you may want to use a subset of particles to speed things up). Compare the results. Do you notice any trends? Which mask do you consider optimal for this refinement? Are the same settings best for other domains? ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#masking-small-domains) Masking small domains What is the smallest domain you can mask before the results become unreliable? Why do you think it’s harder to align smaller domains? Can you think of any settings you could change to improve the results? [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#references) References ----------------------------------------------------------------------------------------------------------------------------------------- 1. Thi Hoang Duong Nguyen et al., “Cryo-EM Structure of the Yeast U4/U6.U5 Tri-snRNP at 3.7 Å Resolution,” _Nature_ 530, no. 7590 (February 1, 2016): 298–302, [https://doi.org/10.1038/nature16940](https://doi.org/10.1038/nature16940) . 2. Grigore Pintilie and Wah Chiu, “Comparison of Segger and Other Methods for Segmentation and Rigid-Body Docking of Molecular Components in Cryo-EM Density Maps.,” _Biopolymers_ 97, no. 9 (September 2012): 742–60, [https://doi.org/10.1002/bip.22074](https://doi.org/10.1002/bip.22074) . 3. Peter B. Rosenthal and Richard Henderson, “Optimal Determination of Particle Orientation, Absolute Hand, and Contrast Loss in Single-Particle Electron Cryomicroscopy,” _Journal of Molecular Biology_ 333, no. 4 (October 31, 2003): 721–45, [https://doi.org/10.1016/j.jmb.2003.07.013](https://doi.org/10.1016/j.jmb.2003.07.013) . [PreviousTutorial: Dynamic Masking in Refinements (v5.0+)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-dynamic-masking-in-refinements-v5.0) [NextTutorial: 3D Classification](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification) Last updated 2 years ago * [Setting Up](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#setting-up) * [Global Refinement](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#global-refinement) * [Mask Generation](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#mask-generation) * [Create the mask base](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#create-the-mask-base) * [Dilate and pad the base to create a mask](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#dilate-and-pad-the-base-to-create-a-mask) * [Important Parameters](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#important-parameters) * [Rotation and Shift Search Extents](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#rotation-and-shift-search-extents) * [Fulcrum Position](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#fulcrum-position) * [Build the Job](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#build-the-job) * [Diagnostic Plots](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#diagnostic-plots) * [Results - Local Refinement](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#results-local-refinement) * [Particle Subtraction](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#particle-subtraction) * [Mask Creation](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#mask-creation) * [Results - Local Refinement after Particle Subtraction](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#results-local-refinement-after-particle-subtraction) * [Conclusion](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#conclusion) * [Exercises](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#exercises) * [The foot domain](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#docs-internal-guid-da9bd864-7fff-1258-2029-53da86db3392) * [Investigate mask parameters](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#investigate-mask-parameters) * [Masking small domains](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#masking-small-domains) * [References](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp#references) Copy volume gaussian #1 sDev 2 Copy volume resample #4 ongrid #1 Copy save ~/Desktop/P300-J3_head.mrc #5 Copy scp ~/Desktop/P300-J3_head.mrc {username}@{host}:~/masks/tri-snRNP/ --- # Case Study: End-to-end processing of encapsulated ferritin (EMPIAR-10716) | CryoSPARC Guide ![Page cover](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fh4Ai64DL2O0M3lCPJoOy%252Fv4-5-encftn-encapsulin_hierarchy_png.png%3Falt%3Dmedia%26token%3Dd17c37ef-63cb-4307-8a56-4cc24a612f8e&width=1248&dpr=3&quality=100&sign=8022ebe3&sv=2) For the complete documentation index, see [llms.txt](https://guide.cryosparc.com/llms.txt) . This page is also available as [Markdown](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716.md) . [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#introduction) Introduction -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The aim of this case study is to demonstrate some advanced tools and processes within CryoSPARC that enable processing of structures with unconventional symmetry present. The dataset this case study will cover is an encapsulin nanocompartment originally collected by [Jennifer Ross, et al. (2022)](https://www.science.org/doi/10.1126/sciadv.abj4461) , containing four encapsulated ferritin (EncFtn) decamers within. The raw data is publicly available for download as [EMPIAR-10716](https://www.ebi.ac.uk/empiar/EMPIAR-10716/) . The main topics of focus covered in this case study include: high-symmetry ab-initio reconstruction, local symmetry, non-point-group symmetry, symmetry expansion, custom geometry operations. The jobs focused on include: [local refinement](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement) , [3D classification](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-3d-classification-beta) , [volume alignment tools](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools) , [align 3D maps](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-align-3d-maps) , [particle subtraction](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta) , [regroup 3D](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-regroup-3d-classes) , [symmetry expansion](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-symmetry-expansion) . Encapsulins are a type of protein that contain internal cargo, and this encapsulin is an icosahedrally-symmetric molecule from the Haliangium ochraceum bacteria. The geometry of the nanocompartment is best illustrated in Figure 1: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fh4Ai64DL2O0M3lCPJoOy%252Fv4-5-encftn-encapsulin_hierarchy_png.png%3Falt%3Dmedia%26token%3Dd17c37ef-63cb-4307-8a56-4cc24a612f8e&width=768&dpr=3&quality=100&sign=8022ebe3&sv=2) Figure 1. Schematic diagram of the geometry of encapsulin and encapsulated ferritin in the Haliangium ochraceum bacteria. Due to the complicated geometry of this nanocompartment, processing it can be challenging. The four EncFtn decamers are arranged in an approximate tetrahedral shape, which complicates solving this structure due to the mismatch in symmetry of the cargo and the shell. Furthermore, each EncFtn molecule itself has 5-fold dihedral symmetry, meaning there is an additional symmetry mismatch between the tetrahedral arrangement and the internal D5 symmetry of each EncFtn molecule. While refinement of the encapsulin is fairly straightforward, recovering high resolution in the internal EncFtn is difficult without custom steps that take care to respect the geometry of the cargo. This case study walks through the steps we took to handle this geometry in CryoSPARC, achieving a final high-resolution structure of both encapsulin and encapsulated ferritin. The previously published map of encapsulated ferritin from this dataset reached resolutions of 5-6 Å; using the techniques of this case study, we were able to resolve the encapsulated ferritin to a sub-3 Å structure. This case study is divided into two sections, each with subsections covering the major processing tasks: * Section A: Encapsulin Processing * A1: Preprocessing and Particle Picking in CryoSPARC Live * A2: 2D Classification * A3: Encapsulin 3D Reconstruction * Section B: Encapsulated Ferritin Processing * B1: Group Re-alignment on Tetrahedron * B2: Custom Symmetry Expansion * B3: Group Re-alignment on Encapsulated Ferritin * B4: Local Refinement All processing was done in CryoSPARC and CryoSPARC Live v4.5. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#a-encapsulin-processing) A: Encapsulin Processing ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- This case study begins with processing the dataset, with the goal of reconstructing encapsulin. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#a1-preprocessing-and-particle-picking) A1: Preprocessing and Particle Picking Preprocessing of exposures consists of import, motion correction, and CTF estimation. These steps can either be completed separately using individual jobs in CryoSPARC, or simultaneously using CryoSPARC Live. The latter can be quicker as it allows processing exposures in a streaming fashion, where one exposure can be imported, motion corrected, and CTF estimated all in sequence (i.e. without waiting for all other exposures to finish each step). We will use CryoSPARC Live to perform import, motion correction, CTF estimation, particle picking and extraction. If you haven’t used CryoSPARC Live before, you can review this [Start to Finish Guide](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide) . #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#download-raw-data-subset) Download Raw Data (Subset) This dataset comprises 8,109 movies. Due to the large size of the dataset, we chose to only process a subset of the 8,109 movies. As we will see, that was sufficient for high resolution reconstruction due to the high symmetry present, but further improvements in map quality are possible if the entire dataset is used. Thus in order to make processing quicker, we will only download the movies in two subdirectories, `GridSquare_16285984` and `GridSquare_16286188` which amounts to 2,815 movies in total. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#set-up-live-session) Set up Live Session * Create a new CryoSPARC Project, and within this project, create a new Live Session. Under the configuration tab, enter the following configuration information and parameters: Parameter Value Raw pixel size (A) 0.326 Accelerating voltage (kV) 300 Spherical Aberration 2.7 Total exposure dose (e/A^2) 40.509 Save Results in 16-bit floating point Yes Output F-crop factor 0.5 Minimum particle diameter 160 Maximum particle diameter 230 Use circular blob Yes Use ring blob Yes Extraction box size 800 Fourier crop to box size 512 * In the Configuration Tab, create two exposure groups, with the following fields set: Exposure Group 1 Exposure Group 2 Directory to watch `.../10716/data/micrographs/GridSquare_16285984/Data` `…/10716/data/micrographs/GridSquare_16286188/Data` File name wildcard filter `*fractions.tiff` `*fractions.tiff` Enable continuous import True True * Use at least one _Preprocessing GPU worker(s)._ Set the number of _Reconstruction GPU workers_ to 1 (note reconstruction tasks i.e. ab-initio and refinement will not be used in Live for this case study). * Click “Start session” to begin processing. CryoSPARC Live will automatically begin motion correction, CTF estimation, particle picking, and particle extraction. * In the [Overview Tab](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-4.-exclude-poor-quality-exposures-from-downstream-processing) , modify the upper CTF fit resolution threshold to 6 Å. * In the [Picking Tab](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-6.-fine-tune-particle-picking) , adjust the Normalized Cross Correlation (NCC) and Power Score sliders to remove false positive picks. CryoSPARC Live will work through the exposures and process them until extraction is complete for each exposure. You can tell when the exposures have finished processing when the number of processed exposures equals the number of total exposures. We are now done with CryoSPARC Live for this case study. For the remainder of the processing, we will use the standard CryoSPARC interface. * Navigate to the top dropdown menu, and click “Go to session workspace”; in this workspace we will carry out the rest of the jobs. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FZ19lk7Kk11QNwIrp7bVB%252Fv4-5-encftn-cslive.png%3Falt%3Dmedia%26token%3D56f5dfa7-6510-499f-a663-ef30e1b78151&width=768&dpr=3&quality=100&sign=c43317b1&sv=2) ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#a2-2d-classification) A2: 2D Classification Once we have an exported stack of particles in CryoSPARC, we will use 2D Classification to curate our particle stack and remove false positive particle picks. * In the session workspace, locate the most recent “Live Particle Export” job. Add these particles to a new [2D Classification](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/job-2d-classification) job with the following parameters: **Parameter** **Value** Number of classes 80 Minimum separation distance (A) 60 Number of GPUs to parallelize 3 * Once the 2D Classification job is complete, use [quick actions](https://guide.cryosparc.com/application-guide-v4.0+/creating-and-running-jobs#job-quick-actions) to queue a Select 2D Classes job. Select classes that show high resolution detail in the encapsulin. Note that since the encapsulin is the dominant signal in the images at this stage, the interior cargo will likely remain blurry and ill-defined for most of the classes, even the classes for which encapsulin is well-defined. Reject all classes that have multiple overlapping encapsulin particles, are empty, include ice or carbon edges, or otherwise have junk in them. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FdBJBBL3s3DYXCfXpphTu%252Fv4-5-encftn-2dclasses.png%3Falt%3Dmedia%26token%3Df98ff19a-d211-4dd7-a450-d4dbb140c2c6&width=768&dpr=3&quality=100&sign=d393d5f1&sv=2) Figure 2. A screenshot of the Select 2D Classes job used to select a subset of particles to continue to 3D reconstruction. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#a3-encapsulin-3d-reconstruction) A3: Encapsulin 3D Reconstruction The figure below illustrates the workflow for subsection A3 of this case study. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FC4iKYuytan1ERlwCBQGi%252Fv4-5-encftn-july2024_flowcharts_png_jul16.png%3Falt%3Dmedia%26token%3D0e7763f3-d4e1-4e67-9d30-e2ee97725055&width=768&dpr=3&quality=100&sign=13fdad3c&sv=2) Figure 3. Flow chart of particle processing for 3D reconstruction of encapsulin #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#initial-model-generation) Initial Model Generation Now that we have a set of curated particles, we will move onto 3D initial model generation. High-symmetry structures often require special treatment during initial model generation, as the particle images for these types of structures typically look very similar to each other at low and medium resolutions, regardless of the particle pose. This lack of information in the data makes it difficult for algorithms like Ab-Initio Reconstruction to reconstruct the correct structure when using default parameters. Typically, running Ab-Initio Reconstruction in this setting will yield “flattened” density, with all particles assigned to the same viewing direction. There are two options to work around the lack of information in the images: 1. **Enforce symmetry during Ab-Initio Reconstruction**: This will guarantee that a symmetric structure is found. 2. Alternatively, **Disable “Enforce non-negativity**”: This parameter has empirically been found to help discourage ab-initio from producing flattened models. We can see the difference between options 1 and 2 in the final structures found by Ab-initio in each case: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FRXs8ZDMrwXiAeN08i1vX%252Fv4-5-encftn-abinit-symenforced.png%3Falt%3Dmedia%26token%3Dc8a0ea5e-239e-4c40-86e2-738ed2785bc7&width=768&dpr=3&quality=100&sign=db3ad422&sv=2) Figure 4. Symmetry-enforced ab-initio reconstruction. Internal details are lost ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FVztx6hciKPEFnC0vbuYe%252Fv4-5-encftn-abinit-nosymenforced.png%3Falt%3Dmedia%26token%3De0e8fa07-37aa-4f32-96b4-4bee0dd4ee7e&width=768&dpr=3&quality=100&sign=eec7fc9d&sv=2) Figure 5. Asymmetric ab-initio reconstruction (with non-negativity off). Internal tetrahedral arrangement of EncFtn is visible. The first option is undesirable because we would like to preserve the internal asymmetric structure within the encapsulin as best as possible. The internal structure _doesn’t_ follow an icosahedral symmetry like the outer shell does, so enforcing symmetry will prevent any details from being resolved inside of the encapsulin. * Taking the particles from the Select 2D Classes job, we will build an [Ab-initio Reconstruction](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-reconstruction/job-ab-initio-reconstruction) job with the following parameters: Maximum Resolution (Angstroms) 8 Initial Resolution (Angstroms) 25 Center Structures in Real space Off Enforce non-negativity Off #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#volume-alignment-tools-symmetry-alignment) Volume Alignment Tools (Symmetry Alignment) After obtaining an initial model of the encapsulin, we would like to refine it to high-resolution. Since we gave Ab-Initio no symmetry information, the structure is oriented arbitrarily. This is fine for intrinsically asymmetric structures, but for symmetric structures, we must ensure they are aligned to the symmetry axes if we later want to [enforce symmetry](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-homogeneous-refinement#common-parameters) or enable [symmetry relaxation](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation) . Alignment of the initial volume to the symmetry axes is also done automatically by the subsequent Homogeneous Refinement job, but we include it explicitly here to get familiar with the Volume Alignment Tools job’s parameters, inputs and outputs. * To do this, build a [Volume Alignment Tools](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools) job, activate symmetry alignment, and input “I” as the symmetry string. Connect both the volume and particles from Ab-Initio Reconstruction to this job. The output volume should be aligned to the icosahedral symmetry axes. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#refinement-of-encapsulin) Refinement of encapsulin Now that we have an aligned model, we can refine this to high resolution using a [Homogeneous Refinement](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-homogeneous-refinement) job. For this job, we will use _symmetry relaxation_ to give the refinement the best chance of preserving the asymmetry of the encapsulin contents. * To do this, build a [Homogeneous Refinement](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-homogeneous-refinement) , and set the following parameters: Symmetry I Symmetry Relaxation Method maximization Dynamic mask start resolution (A) 1 Setting the symmetry relaxation to “maximization” enables symmetry relaxation. It can also be set to “marginalization”, which uses a [slightly different method](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation#comparing-refinement-methods) for finding the optimal pose. Setting the dynamic mask start resolution to 1 Å causes the job to use no mask, which is important as dynamic masking can remove lower-contrast asymmetric details that we’d like to preserve, such as the internal contents of the encapsulin. From here, with ~250k particles, we obtained a C1-refined structure of encapsulin at around 3.0 Å. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#homogeneous-reconstruction-only) Homogeneous Reconstruction Only Now that we have a high-resolution reference structure, there are many avenues to further improve resolutions using reference-based algorithms for latent variable estimation. In this tutorial, we’ll use Global CTF Refinement to correct for high-order aberrations. We’ll also create a symmetrized version of the encapsulin portion of the reference, and then use this for Particle Subtraction \*\*to generate particle images with only signal from the internal contents present. This will prepare us for section B of this case study: processing the internal encapsulated ferritin structure. To get a icosahedral (I) symmetric reference for subtracting the encapsulin away, we don’t have to repeat a full refinement. * Instead, connect the particles from the previous C1 refinement to a [Homogeneous Reconstruction Only](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-homogeneous-reconstruction-only) job, and run the job with a symmetry of I specified. This will work as we have ensured our initial reference to the homogeneous refinement was symmetry aligned. Note that despite enforcing symmetry in this Homogeneous Reconstruction \*\*Only, the particles will retain their C1 alignments — thus the particles will remain suitable for downstream processing of the tetrahedral arrangement of EncFtn, and we won’t lose the effort put into preserving the symmetry-break. This would not be true if we re-ran a refinement with symmetry enforced, as alignments would be re-calculated against the symmetric reference. Application of symmetry will result in a significant increase in resolution owing to the greater number of asymmetric units contributing to the structure. In our case, the resolution improved from 3.0 Å to 2.5 Å over the C1-refined structure. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#mask-generation-using-volume-tools) Mask generation using Volume Tools [The Particle Subtraction job](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta) takes in a set of previously-aligned particle images, the corresponding reference volume to which they’ve been aligned, and a [mask](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera) covering the region of the volume that we would like _removed_ from the particle images. We’d like to subtract the encapsulin away, leaving particle images with just the encapsulated ferritin inside. To do this, we need to use Particle Subtraction, and provide it with a mask covering just the encapsulin. * To obtain this mask, download the volume from the upstream Homogeneous Reconstruction job. Open this volume in UCSF ChimeraX, and select a threshold value that preserves most of the encapsulin while removing all of the internal density. * Note down this threshold value. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FJtCyuz9gve2tGziXb6hh%252Fv4-5-encftn-thresh-comparison.png%3Falt%3Dmedia%26token%3Dcaf55f22-8b4e-4fa6-b082-5484c1452d1f&width=768&dpr=3&quality=100&sign=d2ba9dbc&sv=2) Figure 6. The symmetric reconstruction shown at two threshold levels. * Create a Volume Tools job and connect the volume from the reconstruction only job, with the following parameters: Type of input volume map Type of output volume mask Threshold _chosen value_ Dilation radius (pix) 4 Soft padding (pix) 18 #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#global-ctf-refinement) Global CTF Refinement * Connect the particles and volume from the upstream symmetry-enforced Reconstruction Only to a [Global CTF Refinement](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-refinement/job-global-ctf-refinement) job. Connect the mask from the previous Volume Tools job. Activate “Fit anisotropic mag”, to allow for estimation of anisotropic magnification; this will also set the number of iterations to 2. Global CTF Refinement works best when operating on the reference with the largest mass and highest resolution available, and on particles with refined alignments. Global CTF parameters are a function of the microscope (i.e., they have little dependence on which region of the protein is being refined), and so can be optimized with the 2.5 Å symmetry-enforced encapsulin map, which is large and high-quality. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#homogeneous-reconstruction-only-1) Homogeneous Reconstruction Only To get a reference generated from CTF-refined particles, we’ll repeat reconstruction using the CTF-refined particles. * Create a Homogeneous Reconstruction Only job. Connect the particles from the previous Global CTF Refinement along with the mask from the Volume Tools job, set the symmetry to “I”, and launch the job. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#particle-subtraction) Particle Subtraction * Connect the particles, volume, and mask from the previous Homogeneous Reconstruction Only job to a Particle Subtraction job. Since this mask only covers the encapsulin, the particles will have encapsulin subtracted away and internal contents preserved. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FokXZPHOMnCLiVpWTmJ9Q%252Fv4-5-encftn-subtracted_vs_unsubtracted.gif%3Falt%3Dmedia%26token%3D12d44211-5afb-4711-8bfe-ade33afc707c&width=768&dpr=3&quality=100&sign=b27a1bd6&sv=2) Figure 7. Original vs. subtracted images (lowpass filtered) [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#b-encapsulated-ferritin-processing) B: Encapsulated Ferritin Processing ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- We now have subtracted particles containing just the four copies of encapsulated ferritin. However, we have a tricky case of symmetry to handle: * Within each EncFtn, there is D5 (5-fold and 2-fold) symmetry * The four encapsulated ferritin are arranged in a tetrahedron configuration. However this is not equivalent to a _tetrahedral point symmetry group,_ because tetrahedral point symmetry has 3-fold symmetry at each vertex, but EncFer is 5-fold symmetric Thus, EncFtn has two different types of symmetry. First, each individual copy of EncFtn has D5 point group symmetry, which may be referred to as a _local symmetry_. Second, the overall tetrahedral arrangement imparts a _non-point-group_, four-fold symmetry, which must be treated with custom operations to superimpose each of the four EncFtn. Within each of the encapsulin nanocompartments, there are 4∗5∗2\=404\*5\*2=404∗5∗2\=40 asymmetric units available for symmetry-averaging. The remainder of this tutorial focuses on how we can use CryoSPARC to align these asymmetric units, remove broken particles/further curate the particles, and refine to high resolution. The figure below demonstrates the processing chain we will use for refining the encapsulated ferritin. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FQIjjK0DLsEoie0xVqVQ0%252Fv4-5-encftn-july2024_flowcharts_coloured.png%3Falt%3Dmedia%26token%3De4e20adf-1c52-45e6-8cef-fffe3469023d&width=768&dpr=3&quality=100&sign=d17e36b&sv=2) Figure 8. Flow chart of particle processing for 3D reconstruction of encapsulated ferritin. Encapsulated ferritin processing is divided into four sections: Group Re-alignment on Tetrahedron (B1); Custom Symmetry Expansion (B2); Group Re-alignment on Encapsulated Ferritin (B3); Local Refinement (B4) ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#homogeneous-reconstruction-and-local-refinement-demonstration) Homogeneous Reconstruction & Local Refinement (demonstration) Now that we have subtracted particles, we might first attempt to refine the interior. * First, we launch a Homogeneous Reconstruction job to generate an initial reference from the subtracted particles. * Next, using the subtracted particles and reference, launch a Homogeneous Refinement. To prevent the job from using masking altogether, set the “Dynamic mask start resolution” to 1 Å. With our stack of 250k subtracted particles, the Homogeneous Refinement reached a resolution of 10 Å. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FhdB1twoW2odULZBkbgxp%252Fv4-5-encftn-recon-from-subtracted-particles.png%3Falt%3Dmedia%26token%3D24d9bc6e-4186-4cce-b066-0234db1b5766&width=768&dpr=3&quality=100&sign=11892c1c&sv=2) Figure 9. 10 Å structure refined from subtracted particles. Next, we attempted to locally refine one of the encapsulated ferritin proteins. By using the [map eraser](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#method-two-volume-eraser) in UCSF ChimeraX, a mask was generated around one protein, and the structure was locally refined. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FQoRgJibnrLfAeTJ8XeLH%252Fv4-5-encftn-locref-before-3dclass-lowthresh.png%3Falt%3Dmedia%26token%3D12f9c9b7-0882-4d7f-90f1-99e204be8b6a&width=768&dpr=3&quality=100&sign=83904a11&sv=2) Figure 10. Locally refined EncFtn viewed at a low threshold ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FlJujrySDcsjc3wyw23sX%252Fv4-5-encftn-locref-before-3dclass-highthresh.png%3Falt%3Dmedia%26token%3Da58de7d1-e521-4758-997f-81ba88383076&width=768&dpr=3&quality=100&sign=e42976fb&sv=2) Figure 11. Viewed at a high threshold — note the artefacts are the strongest density in the map This Local Refinement stalled at a claimed resolution of ~ 6.9 Å. The non-protein streaking artefacts visible in the map above are a characteristic sign of overfitting. There are two potential reasons why this Local Refinement was unsuccessful: 1. There are many junk particles in the dataset. This is supported by the fact that one of the 3D classes (from the next step) solved does not show a clear tetrahedral configuration of the four EncFtn instances, rather showing a disordered “ring” of density where three distinct EncFtn would be expected: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FHHzkOaWxDu2lkYB6ysVb%252Fv4-5-encftn-tetr-badclass.png%3Falt%3Dmedia%26token%3D141d2593-e3e2-4d22-8b14-d8255b70ae6f&width=768&dpr=3&quality=100&sign=ab26be91&sv=2) Figure 12. One of twenty classes found in section B2. Note the disorganized density along the bottom half of the image. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FrGVmGqr6Ty034B7K0cN0%252Fv4-5-encftn-tetr-goodclass.png%3Falt%3Dmedia%26token%3D5ff08067-1d76-4ecb-a393-78aa87e975ba&width=768&dpr=3&quality=100&sign=d2cfeee4&sv=2) Figure 13. One of twenty classes found in section B2. Note that four separate densities are present. 1. Initial particle alignments are not close enough to their optimal values. Local Refinement can only move particle alignments by so much, and the larger the search space we give it to make, the greater potential there is for overfitting. Local Refinement works best when it does not have to search a large range of poses, and when tight gaussian priors are used to limit the drift of alignments. Both factors above make it more difficult for local refinement to solve for a high-resolution structure with minimal artefacts. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#b1-group-re-alignment-on-tetrahedron) B1: Group Re-alignment on Tetrahedron Instead of refinement, we’ll attempt to perform 3D Classification on the tetrahedral arrangement of EncFtn. Since 3D Classification doesn’t update alignments, we’ll be able to see if there is heterogeneity in the orientations of the “tetrahedra”. To bias the classification as little as possible, we’ll provide a spherical mask covering the inside of the encapsulin. If we used a mask generated from the consensus subtracted particles, we may bias the classification to only find classes of the tetrahedron oriented in the same orientation as the consensus structure. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#obtain-spherical-mask-ucsf-chimerax) Obtain spherical mask (UCSF ChimeraX) Before starting 3D Classification, we’d like to use a mask that excludes the corners of the box, along with any other residual density from the encapsulin. At this stage the best mask to use would be a soft spherical mask centered at the box center, as this biases the classification the least (as opposed to a mask contoured to the consensus density). **Generate a mask base in UCSF ChimeraX** Download the `map` from the most recent homogeneous reconstruction job. Open the map in a new ChimeraX session. Navigate to the “Tools” tab on the menu bar, and head to `Tools > Volume Data > Map Eraser`. The map eraser should open by default in the center of the volume; if not, adjust the pink sphere’s position to the center of the volume via clicking and dragging. Adjust the size of the map eraser to approximately surround the internal disordered density of the encapsulin; refer to the image below for an example: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FrK9w9rZEQzYzftnKrDC3%252Fv4-5-encftn-chimerax-1.png%3Falt%3Dmedia%26token%3Da482c120-5270-4d67-a2ec-fe676c0d37a9&width=768&dpr=3&quality=100&sign=cd50d350&sv=2) Lower the density threshold all the way to the lowest value in the volume — you should see a large cube of density. This is so that we can erase all density outside of a central sphere, leaving us with a spherical mask base in the center. Use the threshold operation to binarize the cube: Now click “Erase outside sphere”. You should now have a solid sphere of density: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FeCeXrol0K8srnDyLHuJT%252Fv4-5-encftn-chimerax-2.png%3Falt%3Dmedia%26token%3D52a3f2fa-eb7f-44ae-aaa1-3137afc469ab&width=768&dpr=3&quality=100&sign=d688bf3&sv=2) Finally, save this map. Click `File > Save...` and change the “Map” dropdown to the thresholded volume. Give the file a name, and click save: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FL5Q2yezVMy1BPszWeF8m%252Fv4-5-encftn-chimerax-3.png%3Falt%3Dmedia%26token%3D4e5a9e08-50af-412c-b76e-644ee7c39b06&width=768&dpr=3&quality=100&sign=24775194&sv=2) **Import Mask into CryoSPARC** * In your CryoSPARC Workspace, create an Import 3D Volumes job. Provide the path to the mask base. Change the “Type of volume being imported” to `mask`, and run the job. **Volume Tools** (padding) * Connect the imported mask to a new Volume Tools job. Set the following parameters, to pad the mask with a width of ~20 pixels: **Parameter** **Value** Type of input volume mask Type of output volume mask Threshold 0.5 Soft padding width 20 This job will produce a softly-padded mask as its output. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#group-re-alignment) Group Re-alignment * Create a 3D Classification job. Take the particles from the Particle Subtraction job, along with the spherical mask, and connect the mask to the “Solvent mask” input. Use the following parameters for the 3D Classification job: Number of classes 20 Filter resolution (Å) 8 O-EM batch size per class 2000 O-EM learning rate init 0.9 O-EM learning rate half-life (%) 0 Force hard classification On Looking at the volume series from 3D Classification, we can see that the volumes are not in total alignment; 3D Classification spent most of its capacity in finding similar volumes oriented differently relative to each other. This is not surprising, since we have not updated alignments since the initial C1 refinement of the entire encapsulin/encapsulated ferritin complex. We can place volumes back into register, as well as update particle alignments for each class, by using Align 3D Maps. * Create an [Align 3D Maps](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-align-3d-maps) job. Activate the “Update particle alignments” parameter. Connect the All volumes output from the 3D Classification job to the “Maps to align (volumes group)” input. Connect the All particles output to the “Particles (all)” input. Connect the spherical mask to the Reference mask input. Finally, pick one of the 3D classes to serve as the reference map — this can be the best resolved class — and connect it to the Reference map input. The results are shown in the video of the orange (left) volume series below, compared to the un-aligned series from 3D classification in gray (right). ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FRsRp8Z6mTBJWvdDiXmts%252Fv4-5-encftn-alignvsnoalign10716comparison.gif%3Falt%3Dmedia%26token%3D6fe99024-e161-47cf-8043-27aa856304bd&width=768&dpr=3&quality=100&sign=6b3c32d0&sv=2) Figure 14. GIF demonstrating the results of aligning the twenty classes found by 3D Classification. The volume series in gray (right) shows the twenty classes directly from 3D Classification. The series in orange (left) shows the 20 classes after running Align 3D classes. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FRRLgGJR7GbIeVIW8q9IT%252Fv4-5-encftn-align_vs_noalign_comparison_10716.png%3Falt%3Dmedia%26token%3Dbe7e74d9-b3ca-4fbd-b7e5-6b281d6c1c85&width=768&dpr=3&quality=100&sign=2eafc17e&sv=2) Figure 15. Static version of Figure 14. All twenty classes are displayed as meshes and are overlaid. Note how the classes directly from 3D classification are significantly out-of-alignment. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#homogeneous-reconstruction-only-2) Homogeneous Reconstruction Only * Using the spherical mask from the previous 3D Classification, launch a Homogeneous Reconstruction Only job. This will produce a consensus structure after the previous Align 3D Maps job. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FMWTylfHrCKPxX8MYdRsF%252Fv4-5-encftn-consensus_after_grouprealign.png%3Falt%3Dmedia%26token%3Ddcb57907-7f1a-4cfa-b07b-40c2f8565544&width=768&dpr=3&quality=100&sign=cca64606&sv=2) Figure 16. This is the consensus structure after the twenty 3D classes have been aligned, particles have been combined across the classes, and the volume has been reconstructed via the Homogeneous Reconstruction Only job. The next step we must do is “effect” the non-point-group symmetry operators through the custom symmetry expansion step. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#b2-custom-symmetry-expansion) B2: Custom Symmetry Expansion The next portion of the case study describes how CryoSPARC can be used to handle non-point-group symmetry. In this case, we would like to use CryoSPARC to superimpose the four copies of EncFtn, such that they can be combined into one structure for a final refinement, as _a priori_ it is expected that each of these four units are indistinguishable and are the same structure. Symmetry-averaging that involves point group symmetries can normally be handled via symmetry expansion or refinement with symmetry enforced. Since this symmetry does not follow a point group, particular steps are required to obtain properly symmetry-averaged structures. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#mask-generation) Mask generation Using ChimeraX, [create four mask bases](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#mask-base-creation) , each surrounding one of the four EncFtn units. This can be done most quickly using ChimeraX’s `Segment Map` tool. Our [Mask Creation Tutorial](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#method-one-volume-segmentation) covers this step in much more detail. Using the consensus map, lower the threshold value until exactly 4 disconnected discs are present in the density, one corresponding to each EncFtn, (but not too low that any of the EncFtn split into multiple disconnected blobs). The segmentation option `Group by connectivity` works well for this dataset, visible under the `Segmenting Options` drop-down menu: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FEPDapawukoJI1MD8Bkv6%252Fv4-5-encftn-chimerax-segparams-4.png%3Falt%3Dmedia%26token%3D3d92b6da-0bd8-4fc7-bc03-746dfae9d442&width=768&dpr=3&quality=100&sign=e58b307b&sv=2) This then produced four segments, each covering one of the EncFtn units. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F6bAWBBgOdLWvtod0G7Uz%252Fv4-5-encftn-segments_by_connectivity_from_J1054.png%3Falt%3Dmedia%26token%3D8a04dde1-e5fc-49d0-99db-0af585db4ecd&width=768&dpr=3&quality=100&sign=b9a8dfda&sv=2) Figure 17. The result of a segmentation on the volume displayed in Figure 16. Following the remainder of the mask generation tutorial, we are able to generate four mask bases that will subsequently be imported into CryoSPARC. Save each of these mask bases with a format such as `encftn_maskbase1.mrc`, `encftn_maskbase2.mrc`, etc., in a directory. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#id-1x-import-3d-volumes) 1x Import 3D Volumes * Build an Import 3D Volumes job. Set the path to a wildcard pointing to all four masks (for example, `/path/to/directory/encftn_maskbase*.mrc`). Change the type of imported volume to `mask`, and hit run. The job will import all four mask bases and present them as outputs. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#id-4x-volume-tools) 4x Volume Tools The next step is to generate dilated and softly-padded masks from these four mask bases. * To do this, we will need to run four Volume Tools jobs, one with each of the four masks connected as an input: Type of input volume `mask` Type of output volume `mask` Threshold _value selected in the mask generation section_ Dilation radius anywhere between 2-8, depending on how large masks are desired; here chose 2. Soft padding width At least 16; here chose 16. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FNZGWsVy4hAGfY4EYz9h8%252Fv4-5-encftn-mask-creation-tree.png%3Falt%3Dmedia%26token%3D14d06516-d16d-4097-af0d-ec386c6839a1&width=768&dpr=3&quality=100&sign=835b887b&sv=2) Tree view of the four volume tools jobs used to generate masks from the mask bases created in ChimeraX. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#id-4x-volume-alignment-tools) 4x Volume Alignment Tools The next task is to effect the non-point-group symmetry expansion step. In summary: 1. We use Volume Alignment Tools to shift each of the four masks to the center of the box. 2. Volume Alignment Tools simultaneously adjusts the positions of the volume and particles accordingly, to match each of the four shifted masks. Volume Alignment Tools also _re-generates the unique identifiers (UIDs)_ of each particle, so that CryoSPARC knows to treat each image containing four copies of EncFtn as _four separate observations_ of our protein of interest. 3. Align 3D Maps is then used to rotationally align (i.e. superimpose) all four volumes. Create four Volume Alignment Tools jobs. Connect each of the masks from the previous four Volume Tools jobs as the mask inputs. Connect the volume and particles from the homogeneous reconstruction only as the volume and particle inputs to each of the Volume Alignment Tools jobs. Finally, set the following parameters and run the jobs: Re-center to mask center of mass On Reassign UIDs On #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#align-3d-maps) Align 3D Maps Next, we use Align 3D Maps to correct the rotational mis-alignment of the four volumes from the previous Volume Alignment Tools jobs. This step completes the custom symmetry expansion. * Create an Align 3D Maps Job. * Pick one of the four EncFtn volumes from one of the previous Volume Alignment Tools jobs; this volume will serve as the reference, and the other three will be aligned to it. This will establish the orientation of the consensus of the subsequent 3D classification. (This orientation is not the final one that will be used for the highest resolution refinement, as when we later incorporate D5 symmetry, we will have to align the consensus to the D5 symmetry axes. For now, refinement is proceeding in C1, and we are ignoring the D5 symmetry until we have a cleaner subset of particles.) * Connect this reference volume and the accompanying mask to the “Reference Map” and “Reference Mask” inputs, respectively. Leave the “Maps to align (volumes group)” input empty. * Enable “Update particle alignments” parameter. Connect the four EncFtn volumes from each of the Volume Alignment Tools jobs as connections under the “Maps to align (individual volumes)” group. Finally, connect each of the particle sets corresponding the four EncFtn volumes as individual connections under “Particles (map to align, connection X).” Phew! Now run the job. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fy6oz5CRkfmAtfkLFCsdT%252Fv4-5-encftn-J1106_volumes_aligned_to_reference.png%3Falt%3Dmedia%26token%3D8b008617-94c2-4df0-a9ff-a64637dedb7c&width=768&dpr=3&quality=100&sign=5df57070&sv=2) Figure 18. Before Align 3D Maps — volumes are centered but not superimposed ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fn0BhvlgzBeQrimURFNzV%252Fv4-5-encftn-J1065_volumes_aligned_to_reference.png%3Falt%3Dmedia%26token%3D27f7aff4-7668-491c-9a9c-592dfdf13a63&width=768&dpr=3&quality=100&sign=384e475d&sv=2) Figure 19. After Align 3D Maps — volumes are superimposed and have the same rotational alignment ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#b3-group-re-alignment-on-encapsulated-ferritin) B3: Group Re-alignment on Encapsulated Ferritin At this stage of processing, we now have all of the EncFtn superimposed. However, the particle stack is quite dirty — there are many junk particles, as evidenced by the previous 3D Classification results. The first step we’ll do is repeat 3D Classification, this time using the expanded particle dataset and a mask covering only one encapsulated ferritin. 3D Classification is preferred over local refinement at this stage for a few reasons. 1. 3D Classification is less sensitive than local refinement to junk. When dirty particle stacks are given to local refinement, a common outcome is artefacts and overfitting. When dirty particle stacks are given to 3D Classification, it is often the case that poor particles can be separated from good particles reasonably well via their class assignments 2. The particle stack is (a) contaminated by lots of junk particles and (b) particles’ alignments are far away from coherently superimposing particles onto one rigid structure, as we will see 3. When alignments are far away from their optimal values, Local Refinement will struggle to align the particles 1. Heterogeneous Refinement and Local Refinement both attempt to estimate alignments on a per-particle basis, which can be problematic when there is still a lot of junk, and when alignments are far off from their optimal values. 2. 3D Classification freezes alignments, thus is not able to use alignments as a free variable to overfit. 3. To overcome the fixed alignments of particles, 3D Classification can be combined with Align 3D Maps to allow for re-alignment of classes on a _volume_ basis. #### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#group-re-alignment-repeat-x2) Group Re-alignment (Repeat x2) The next step comprises of 3D Classification followed by Volume Alignment Tools and Align 3D Maps; this step will be repeated **twice** in order to iteratively improve the quality of our classification. **3D Classification** * Build a [3D Classification](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-3d-classification-beta) job. Connect the `Particles for map {0,1,2,3}` inputs all as connections to the input particle group. Leave the initial volumes and focus mask inputs empty. Connect the mask from the reference volume chosen in the previous step to the Solvent mask input. Use the following parameters: Number of classes 20 Filter resolution (Å) 6 O-EM batch size per class 2000 O-EM learning rate init 1 O-EM learning rate half-life (%) 0 Force hard classification On Despite aligning each of the encapsulated ferritin to a common reference, the volumes have _significant_ diversity both in their position and contents! ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252Fm0SRR0KV62VV7GMhLVFw%252Fv4-5-encftn-j1067_series.gif%3Falt%3Dmedia%26token%3Da893460a-f0c0-49e4-a49a-9ad6f7fd5cec&width=768&dpr=3&quality=100&sign=abf0e0ce&sv=2) Figure 20: Movie of the twenty Encapsulated Ferritin classes directly from the 3D Classification. Note the diversity in position of the twenty classes. Repeating Align 3D Maps (described in the next sub-section) will help position these volumes back into register, as best as possible, and set us up best for a final Local Refinement. **Volume Alignment Tools (D5 symmetry alignment)** Before running Align 3D Maps, we will use Volume Alignment Tools to align the structure to the D5 symmetry axes. This is in preparation for the final local refinement we’ll do to high-resolution. * Create a Volume Alignment Tools job. * Pick the class from the previous 3D Classification, that shows the strongest 5-fold symmetry. Connect this class to the volume input, and connect its corresponding particles to the particles input. Connect the solvent mask from 3D Classification to the mask input. * Activate the “Do symmetry alignment” parameter. Set the symmetry string to “D5”. Run the job. **Align 3D Maps** * Create an Align 3D Classes job. Activate the “Update particle alignments” parameter. Connect the `All volumes` group from the 3D classification to the `Maps to Align (volumes group)` input group. Connect the `All Particles` group from 3D classification to the `Particles (all)` input group. Use the volume and mask from the previous Volume Alignment Tools job as the reference map and reference mask. Run the job, and observe that the volumes are much closer to alignment than previously: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FFPnOShFYUD0mLOqSzHaS%252Fv4-5-encftn-j1068_series.gif%3Falt%3Dmedia%26token%3D3d56b0fc-74cd-4b9d-aa36-7e60c2e27d0d&width=768&dpr=3&quality=100&sign=88a3b84f&sv=2) Figure 21. Movie of the 20 Encapsulated Ferritin classes after running Align 3D. Note that the twenty classes are in-register. After two iterations of Group Re-alignment, much more detail is beginning to form in the classes. Classes can be visualized by downloading the volume `series` from the Align 3D Maps job: ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FuHRSnGKvMe6N9ytcLMK4%252Fv4-5-encftn-download-series.png%3Falt%3Dmedia%26token%3D42c88aa2-fd63-4d26-b7d1-a4b92e8284ea&width=768&dpr=3&quality=100&sign=f062b519&sv=2) Inspect the classes, and note down which classes are of the intact EncFtn structure, show clear 5-fold symmetry, and are not at low resolution. Below is an example of the classification for 20 classes, with classes selected for further refinement highlighted in blue. ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252F96HKG1Ni6O3IoajAXEKn%252Fv4-5-encftn-select3D_after_j1069.png%3Falt%3Dmedia%26token%3D9626343e-276a-44af-bfc3-f98bcc48cb3a&width=768&dpr=3&quality=100&sign=7c750b68&sv=2) Figure 22. Selected classes from the most recent 3D Classification. The particles assigned to the blue highlighted classes are carried forward into Local Refinement. ### [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#b4-local-refinement) B4: Local Refinement * Create a [Local Refinement](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement) job. Connect each particles group corresponding to each of the selected classes from the previous Align 3D Maps job. Connect the volume and mask from the latest upstream Volume Alignment Tools job to the Initial Volume and Static Mask inputs, respectively. Use the following parameters: Use pose/shift gaussian prior during alignment On Standard deviation (deg) of prior over rotation 6 Standard deviation (A) of prior over shifts 3 Re-center rotations each iteration? On Re-center shifts each iteration? On Symmetry D5 Number of extra final passes 0 With selecting a good subset of 3D classes, we retained ~445k of ~1,009k particles for this Local Refinement. This subset refined to a resolution of ~2.8 Å, compared to previous Local Refinement of the encapsulated ferritin reaching only in the 6-9 Å. To help understand these results, it’s helpful to examine what changed from the initial Homogeneous Refinement job on all four EncFtn molecules. Why did it work better now? 1. The particles now had accurate-enough starting orientations. Earlier in the workflow, orientations were very poor, and likely too far away from their optimal values. * Resolving the internal orientation diversity via group re-alignment was crucial to accomplish this. Group re-alignment allowed us to put particles in register, before the reference was high-resolution enough to allow for per-particle-pose estimation 2. Local Refinement also worked better because there was minimal heterogeneity — the broken/misaligned ferritin classes were removed in this final 3D Classification. 3. Finally, Local Refinement had access to a greater number of particles, due to the custom symmetry expansion and D5 symmetry enforcement. [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#encapsulated-ferritin-density) Encapsulated Ferritin Density ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ Comparing our density map from this case study to the previously published density map, we can see that accounting for the challenging symmetry of this sample paid off! ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FtEtww1ugWYmAjFdgOFV3%252Fv4-5-encftn-final-density-comparison.png%3Falt%3Dmedia%26token%3Db35c8425-8711-47e8-adce-e82e1abdbf71&width=768&dpr=3&quality=100&sign=c624cca3&sv=2) Figure 23. Comparison between the map of encapsulated ferritin uploaded to EMDB from this dataset (EMDB-13608, left) and the density reconstructed in this case study (right). To see if this density map was plausible, we re-refined the protein sequence from the atomic model [PDB 5N5F](https://www.rcsb.org/structure/5n5f) . The 5N5F atomic model was obtained by [Didi He, et al. (2019)](https://pubmed.ncbi.nlm.nih.gov/30837306/) via Phenix refinement into a 2.1 Å map from x-ray diffraction. Though this structure _is_ _encapsulated ferritin_ from the same species of bacteria as the cryo-EM map, It wasn’t known to us whether we could expect the conformation of 5N5F to be identical to that of our density map. Possibly because the encapsulated ferritin in the cryo-EM sample was imaged inside of encapsulin (instead of being crystallized). ![](https://guide.cryosparc.com/~gitbook/image?url=https%3A%2F%2F1916621962-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-M7DGv3GkRvGGpbVPCgg%252Fuploads%252FMU3N5puLBWUjyl8UOWdN%252Fv4-5-encftn-5n5f_vs_rerefined_model.png%3Falt%3Dmedia%26token%3De6116874-a73f-47f2-9eea-f68442eab3db&width=768&dpr=3&quality=100&sign=834b7880&sv=2) Figure 24. Comparison between the atomic [model entry 5N5F](https://www.rcsb.org/structure/5n5f) from the PDB (left) with a re-refined model (right) into the density map obtained in this case study. The density from this case study is shown in blue and is overlaid on both models. The above figure shows the density map obtained from this case study, sharpened to a B-factor of -60. This is overlaid on the 5N5F atomic model from the PDB (left column), and the re-refined atomic model (right side). The additional density present near the N-terminus of the AA sequence enabled modelling an extra three residues (GLU5, SER4, and SER3) that were not present in the atomic model from XRD. And that’s a wrap! This case study highlighted how to use CryoSPARC to handle the unique geometry and symmetry of the encapsulated ferritin dataset. Further standard processing workflows that weren’t explored in this case study could further improve results, for example: * Repeated 3D Classification to remove more junk * Reference Based Motion Correction * Local CTF (defocus) refinement * Process _all_ movies in the dataset * * * [PreviousCase Study: End-to-end processing of an inactive GPCR (EMPIAR-10668)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-an-inactive-gpcr-empiar-10668) [NextCase Study: Discrete and Continuous Heterogeneity in FaNaC1 (EMPIAR-11631 and -11632)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-discrete-and-continuous-heterogeneity-in-fanac1-empiar-11631-and-11632) Last updated 1 month ago * [Introduction](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#introduction) * [A: Encapsulin Processing](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#a-encapsulin-processing) * [A1: Preprocessing and Particle Picking](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#a1-preprocessing-and-particle-picking) * [A2: 2D Classification](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#a2-2d-classification) * [A3: Encapsulin 3D Reconstruction](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#a3-encapsulin-3d-reconstruction) * [B: Encapsulated Ferritin Processing](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#b-encapsulated-ferritin-processing) * [Homogeneous Reconstruction & Local Refinement (demonstration)](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#homogeneous-reconstruction-and-local-refinement-demonstration) * [B1: Group Re-alignment on Tetrahedron](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#b1-group-re-alignment-on-tetrahedron) * [B2: Custom Symmetry Expansion](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#b2-custom-symmetry-expansion) * [B3: Group Re-alignment on Encapsulated Ferritin](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#b3-group-re-alignment-on-encapsulated-ferritin) * [B4: Local Refinement](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#b4-local-refinement) * [Encapsulated Ferritin Density](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716#encapsulated-ferritin-density) Copy cd /path/to/rawdata # navigate to a container directory to hold the raw data wget https://www.ebi.ac.uk/empiar/world_availability/10716/data/micrographs/GridSquare_16285984/ . && wget https://www.ebi.ac.uk/empiar/world_availability/10716/data/micrographs/GridSquare_16286188/ . Copy vop threshold #1 maximum setMaximum 1 --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/live/about-cryosparc-live.md). # About CryoSPARC Live ## From microscope to structure, in minutes CryoSPARC Live is a software platform that enables: \* Real-time cryo-EM data quality assessment \* Decision making based on 2D and 3D results during live data collection \* An expedited, streamlined workflow for processing previously collected data \* Direct seamless interoperation with CryoSPARC for advanced processing {% hint style="info" %} \*\*CryoSPARC Live is built to enable experimentation with parameters on the fly, while the software efficiently manages the reprocessing that is necessary in order to test or effect parameter changes, and while maintaining the overall progress of the Live session.\*\* {% endhint %} {% embed url="" %} Processing EMPIAR-10288 in cryoSPARC Live. {% endembed %} ## Who is CryoSPARC Live built for? CryoSPARC Live is built for: \* \*\*Data collection facilities, cryo-EM cores and and microscope operators\*\*, who want to make the most of microscope time with real-time data quality assessment and collection; \* \*\*Facilities and service providers\*\*, who want to provide their users with expedited information about sample quality and/or delivery of 3D maps; and \* \*\*Individual users\*\*, who wish to gain insights about data quality by performing 3D reconstructions in real-time with data collection, or on previously collected data in a seamless and "first cut" manner. ## Use cases for CryoSPARC Live ### Live processing at microscope, during data collection CryoSPARC Live can be used by microscope operators during a collection session. We recommend the exposures are written to fast disks as I/O can become a bottleneck. ### Live processing offsite, during data collection CryoSPARC Live can be engaged to read files that are being uploaded to the user, e.g., via AWS or other similar service. Live will read in new exposures as they are found. ### Seamless first cut processing of previously-collected data Finally, we recommend using the Live workflow for expedited preprocessing (see below) and a first-cut look at data quality in 2D and 3D, for all datasets including those already collected. ## \*\*What does CryoSPARC Live enable?\*\* ### Continuous import and expedited preprocessing !\[\](/files/-MNiohSEbV2aYK0W8eC3) CryoSPARC Live watches specified directories for new files and processes them as they become available. CryoSPARC Live preprocessing includes four steps: motion correction, CTF estimation, particle picking and extraction. CryoSPARC Live can sustain a throughput of 450 or more exposures per hour, per GPU, for K3 data. On a 4-GPU machine, that can scale to 1800+ exposures per hour! For K2 or Falcon data, performance can be even higher, upwards of 650 exposures per hour per GPU. Particles from preprocessing are seamlessly transitioned into 2D classification and 3D reconstruction, and preprocessed exposures can also be exported for further processing in CryoSPARC or other software. For extensive benchmarks and throughput statistics, please see: {% content-ref url="/pages/-MNiprFxrS7wFGz3IzpU" %} \[Performance Metrics\](/live/performance-metrics.md) {% endcontent-ref %} ### Adjustable parameters and saveable configurations \* \*\*Microscope/camera and job parameters:\*\* Any of these can be adjusted over the course of a session, and combinations of parameters can be saved as "\[Configuration Profiles\](/live/new-live-session-start-to-finish-guide.md#load-configuration-profile)" for use in future sessions. For example, microscope parameters such as the \`pixel size\`, \`spherical aberration\` and \`accelerating voltage\` are likely to be consistent for a given instrument and can be applied quickly using Configuration Profiles at the start of a new Session. If working on multiple samples of the same protein or complex, it may be useful to save picking and extraction parameters to save time. \* \*\*Exposure groups:\*\* It is possible to add, remove, and ignore exposures from one or more Exposure Groups (collections of exposures with the same optical parameters). This makes it possible to process subsets of a larger dataset and make comparisons. !\[\](/files/-MNiojmcQ4QAF3-q5wE\_) Learn more about Configuration Profiles in the UI Overview: {% content-ref url="/pages/-MNipf2Fvo\\\_LpKLhdEk5" %} \[UI Overview\](/live/ui-overview.md) {% endcontent-ref %} ### Streamlined exposure curation (threshold-based and manual) !\[\](/files/-MNiomLuMnzMYfaKKWVS) Users can review several useful statistics (e.g., CTF Fit, Defocus, Total Motion, etc) across a dataset and apply thresholds to automatically reject incoming and already-processed exposures that do not fall within the desired range for one or more parameters. Exposures can also be rejected manually. Rejected exposures are excluded from further processing unless thresholds are changed, which can be done at any time during a Live session. On export, exposures are split into different output groups (e.g., rejected and accepted) for further advanced processing outside of the Live workflow. ### Ability to test and refine picking strategies while collection is ongoing It is possible to have finalized a picking strategy within the first few hundred exposures, which can then be left to run on the remaining exposures as they come in. Blob-based picking is active by default at the start of any Live session and manual picking can be engaged at any time. Blob/manual picks can be curated and used to generate templates for template-based picking, or available templates can be loaded directly into the Session. Pick scores (NCC and Power Threshold) can be used to include or exclude picks. These and particle extraction parameters can be tested on a single or few exposures before being applied to the entire dataset, and can be updated as many times as necessary. Updated particle locations will be fed automatically into later stages of the pipeline (e.g., Streaming 2D Classification). !\[\](/files/-MNiopWZ2UpPRV34aDP6) ### \*\*Make go/no go decisions about a sample using Streaming 2D classification\*\* \*\*Real time streaming 2D classification enables assessing sample quality, preferred orientation issues and presence/absence of the expected target and/or ligands, as well as large conformational variability.\*\* 2D Classification in CryoSPARC Live picks up newly extracted particles from incoming exposures and automatically updates 2D classes every few minutes using a new streaming method. This means that after starting a Streaming 2D Classification job, class averages will update automatically as new particles become available from upstream. Class averages can be selected as soon as the initial classification is complete, and these selections will be retained, enabling Streaming 3D Refinement to take in new particles from the selected classes and update the reconstruction in real time. Early feedback in 2D (and 3D, see below) can confirm whether a collection should continue, or whether upstream steps such as sample preparation may require improvement. Within a few hours of starting a Live session, it is possible to make a "go/no go" decision about the sample and assess issues that may result in a poor 3D reconstruction. Thus, it is possible to save on microscope time, or, to at least inform the user ahead of time about the quality of result they may be able to expect. !\[\](/files/-MNiosa-jqUFi5\_bFS\_9) Learn more about CryoSPARC Live Jobs and Session-Level Functions: {% content-ref url="/pages/-MNipixNnwl\\\_RERAifqp" %} \[Live Jobs and Session-Level Functions\](/live/live-jobs-and-session-level-functions.md) {% endcontent-ref %} ### \*\*3D reconstruction and Streaming 3D refinement during data collection\*\* The ability to generate a refinement during data collection is important as a diagnostic during collection, as a first-cut structure from which to continue further processing, and in many cases is comparable to the highest resolution structure(s) that can be generated after extensive advanced processing in CryoSPARC. Streaming 3D Refinement in CryoSPARC Live also will pick up newly available particles from Streaming 2D Classification so that over the course of a collection, the 3D structure updates every few minutes. !\[\](/files/-MNiovl-x6x9e-Tq7QT7) For a detailed walkthrough of setting up your own Live session, please see: {% content-ref url="/pages/-MNiplu20pJBGs35cP4A" %} \[New Live Session: Start to Finish Guide\](/live/new-live-session-start-to-finish-guide.md) {% endcontent-ref %} ### Add or free up compute resources during a Session Over the course of a CryoSPARC Live Session, it is possible to adjust the compute resources dedicated to processing. The \`Number of Preprocessing GPU Workers\` can be increased or decreased during a session in order to free up compute resources or add more parallelization capacity for preprocessing. Additionally, the compute lanes being used can be adjusted. !\[\](/files/-MNj4RymPpaIMCsz0NoA) Learn more about GPU requirements for Live: {% content-ref url="/pages/-MNipXzsAHXuvIsuchDY" %} \[Prerequisites and Compute Resources Setup\](/live/prerequisites-and-compute-resources-setup.md) {% endcontent-ref %} ### Export of results and integration with cryoSPARC CryoSPARC Live is tightly integrated with CryoSPARC. Each CryoSPARC Live Session is housed within a CryoSPARC Project, so the results of live processing can always be used seamlessly for further advanced processing in CryoSPARC as well as for export. !\[\](/files/-MNip-oN7ap0HrHA4VjC) It is common to take the final map(s), exposures and particle stack(s) from CryoSPARC Live and hand them off directly to e.g., users of a microscope facility or lab, or to use the motion-corrected, CTF-estimated exposures directly for advanced processing without redoing preprocessing steps. !\[\](/files/-MNip1wVBWkl3G63CA4h) ### Programmatic control of CryoSPARC Live CryoSPARC Live Sessions and jobs can also be controlled via the command line: {% content-ref url="/pages/LqmcOUa40A7TRmskBFBA" %} \[Managing a CryoSPARC Live Session from the CLI (≤v4.7)\](/live/managing-a-cryosparc-live-session-from-the-cli.md) {% endcontent-ref %} ## What users say "CryoSPARC Live enables the Pacific Northwest Cryo-EM Center to keep up with data coming from all 5 of our microscopes, a data volume that has at times exceeded 15TB (30,000+ images) per day. We can monitor not only standard image pre-processing metrics, but also directly observe improvements to 3D maps concurrent with data collection! In many cases we’ve stopped data collection with a 2.2-3.5Å map already in hand. We can also quickly identify pathologies such as poor orientational distributions, or tune imaging conditions to improve alignment of difficult particles." \*\*- Craig Yoshioka, Center Co-Director, Pacific Northwest Center for Cryo-EM (PNCC)\*\* In our facility, CryoSPARC Live has completely revolutionised the way we collect and process cryoEM data. Initially, we commissioned CryoSPARC Live with the primary goal of feedback on micrograph quality through on-the-fly motion correction and CTF estimation. But it has turned out to be so much more than we hoped for. We now use it for all initial processing of data coming through our facility. Our latest GPU workstation was specifically specced out for rapid CryoSPARC processing as it has become our primary in-house processing tool. Some of the aspects we love about CryoSPARC Live include: the GUI is intuitive, easy to use, and easy to learn for new users; processing is extremely fast; parameter changes are propagated throughout the workflow in a streaming fashion; easy linkages with ‘regular’ CryoSPARC make continued processing simple; picking tools are easy to use and work for almost all targets, and micrograph curation is powerful and easy to apply. CryoSPARC has become such an essential part of our data collection and analysis workflow; we could not imagine working without it. Highly valuable microscope time is now used much more efficiently and our microscope users are very satisfied. \*\*- Simon HJ Brown, PhD, Customer Solutions Expert, Cryo Electron Microscopy - Molecular Horizons, University of Wollongong\*\* "CryoSPARC live may literally make your jaw drop. The speed of processing combined with the capability to make adjustments in real time are remarkable. It allows you to make go/no go decisions for each experiment using 2D and 3D results generated almost as fast as the images can be collected. Perhaps most impressive is the ability to go back and change almost any parameter on the fly, and it will immediately reprocess the images from that step, and key metrics such as motion and CTF fit can be examined in interactive plots where ranges can be set and immediately applied for image curation. Together these features take a large step toward the future in reducing both the time and resources that are needed to achieve a successful outcome for each data collection." \*\*- Jeff Speir, Director of Operations, NanoImaging Services Inc. (NIS)\*\* "My group and I have been using CryoSPARC Live beta for a little over a year with great successes. We currently only run CryoSPARC Live for our in house projects and it has been of great help. It is easy to set up, very easy to use and very fast. There is no need to create scripts or any slightly complicated task. On most projects with very limited human interaction and no knowledge of the target of interest we can reach high resolution overnight before the run on the microscope is finished. The CryoSPARC-live features allow us to monitor the quality of the acquisition and make modifications in real time if necessary, whether it is a sample problem (e.g. ice thickness) or an issue with the alignment of the microscope. We are looking forward to extend the use to the core facility users allowing them to get real time feedback on their runs." \*\*- Eric Hanssen, Head, Advanced Microscopy Facility and Associate Professor, University of Melbourne\*\* "CryoSPARC Live is a wonderful tool that not only gets researchers excited about their cryoEM experiments but lets microscope operators know they are acquiring high-quality processable data. With on-the-fly feedback users and staff are able to engage with each other to identify bottlenecks and modify data collection strategies in real-time to conduct optimized experiments for a sample. 3D feedback is critical because it ensures we are able to collect a full dataset of their macromolecule of interest, thereby allowing our users to accelerate their biomedical research." \*\*- Edward Eng, Manager, New York Structural Biology Center (NYSBC)\*\* "Having used CryoSPARC Live at both New York Structural Biology Center (USA) and The Hospital for Sick Children (Canada), it has become an invaluable tool in real time assessment of samples for a frequent user like me. By having the movie frames aligned, CTF estimated, particles picked, 2D classification and ab initio done on the fly allows me to quickly judge which grid/sample is worth collecting on. As all the statistics can also be neatly presented as an overview, it is straightforward too to pick out trends and exceptions in the data: For instance I have noticed cases where a bunch of micrographs had poor estimated resolution, only to pinpoint the problem to a single grid square, allowing me to avoid similar grid squares subsequently. The final big advantage of CryoSPARC Live is that the resulting data can easily be passed onto the conventional CryoSPARC pipeline for further processing – saving both time and energy!" \*\*- Yong Zi Tan, Postdoctoral Fellow, The Hospital for Sick Children\*\* ## History and development CryoSPARC Live was first released as a private beta in May 2019. Based on extensive beta testing at dozens of facilities and labs globally, we have incorporated feedback and worked to improve the workflow, with several iterations already released. CryoSPARC Live will continue to evolve with advancements in data collection, user feedback and automation of the cryo-EM workflow. ### Embedded CryoSPARC Live In 2022, we collaborated with Thermo Fisher Scientific Inc. to make available Embedded CryoSPARC Live, a version of CryoSPARC Live that is designed to seamlessly integrated with Thermo ScientificTM cryo-transmission electron microscope systems. To learn more about the collaboration, please visit: ## Get Started {% content-ref url="/pages/-MNipXzsAHXuvIsuchDY" %} \[Prerequisites and Compute Resources Setup\](/live/prerequisites-and-compute-resources-setup.md) {% endcontent-ref %} {% content-ref url="/pages/-MNipbK81\\\_vhurOSOIVK" %} \[How to Access CryoSPARC Live\](/live/how-to-access-cryosparc-live.md) {% endcontent-ref %} {% content-ref url="/pages/-MNiplu20pJBGs35cP4A" %} \[New Live Session: Start to Finish Guide\](/live/new-live-session-start-to-finish-guide.md) {% endcontent-ref %} {% content-ref url="/pages/-MNiptHMn7c1qhXSGAax" %} \[FAQs and Troubleshooting\](/live/faqs-and-troubleshooting.md) {% endcontent-ref %} --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/live/about-cryosparc-live.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc.md). # All Job Types in CryoSPARC {% content-ref url="/pages/-MUoa87lPm1YW6DAB26E" %} \[Import\](/processing-data/all-job-types-in-cryosparc/import.md) {% endcontent-ref %} {% content-ref url="/pages/-MR1pA2tMmOqig-CU8MC" %} \[Motion Correction\](/processing-data/all-job-types-in-cryosparc/motion-correction.md) {% endcontent-ref %} {% content-ref url="/pages/-MR1qLGDMnyAG7KQCg29" %} \[CTF Estimation\](/processing-data/all-job-types-in-cryosparc/ctf-estimation.md) {% endcontent-ref %} {% content-ref url="/pages/-MUoamuzjZUCsa1m0-zi" %} \[Exposure Curation\](/processing-data/all-job-types-in-cryosparc/exposure-curation.md) {% endcontent-ref %} {% content-ref url="/pages/-MNesJ-dF2JUzMICF7RY" %} \[Particle Picking\](/processing-data/all-job-types-in-cryosparc/particle-picking.md) {% endcontent-ref %} {% content-ref url="/pages/f8sIzFKtFclPgNfk7UX6" %} \[Extraction\](/processing-data/all-job-types-in-cryosparc/extraction.md) {% endcontent-ref %} {% content-ref url="/pages/-M9xp2LTqFA0SqoyLtJy" %} \[Deep Picking\](/processing-data/all-job-types-in-cryosparc/deep-picking.md) {% endcontent-ref %} {% content-ref url="/pages/-MM22IBOzV46KAhZVVtm" %} \[Particle Curation\](/processing-data/all-job-types-in-cryosparc/particle-curation.md) {% endcontent-ref %} {% content-ref url="/pages/-MUob\\\_B0yiYpzeIeiegX" %} \[3D Reconstruction\](/processing-data/all-job-types-in-cryosparc/3d-reconstruction.md) {% endcontent-ref %} {% content-ref url="/pages/-MNf7Nb4weQPD3UpCoQP" %} \[3D Refinement\](/processing-data/all-job-types-in-cryosparc/3d-refinement.md) {% endcontent-ref %} {% content-ref url="/pages/Cxa5mWaTmQxiDmFoytpO" %} \[CTF Refinement\](/processing-data/all-job-types-in-cryosparc/ctf-refinement.md) {% endcontent-ref %} {% content-ref url="/pages/-MUocCsKoGXI9gDlMron" %} \[Conformational Variability\](/processing-data/all-job-types-in-cryosparc/variability.md) {% endcontent-ref %} {% content-ref url="/pages/-MUocLY-Uzy34li2rxCk" %} \[Postprocessing\](/processing-data/all-job-types-in-cryosparc/post-processing.md) {% endcontent-ref %} {% content-ref url="/pages/-MM26LRRNWHmX2Y0ighO" %} \[Local Refinement\](/processing-data/all-job-types-in-cryosparc/local-refinement.md) {% endcontent-ref %} {% content-ref url="/pages/-MNejxB71V\\\_TUPfVwm8E" %} \[Helical Reconstruction\](/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta.md) {% endcontent-ref %} {% content-ref url="/pages/-MM2DB0zuUBFP8-wJoMk" %} \[Utilities\](/processing-data/all-job-types-in-cryosparc/utilities.md) {% endcontent-ref %} {% content-ref url="/pages/IxHDmJ3WEYyXTVKhcFcB" %} \[Simulations\](/processing-data/all-job-types-in-cryosparc/simulations.md) {% endcontent-ref %} --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/job-rebalance-orientations.md). # Job: Rebalance Orientations ## At a Glance

Maps in this figure were produced from EMPIAR 10025 (Campbell et al. 2015)

Remove particles from over-represented viewing directions. ## Description Rebalance Orientations sorts particles into two sets: ‘rebalanced’ and ‘excluded’. The rebalanced particle set will have a more uniform viewing direction distribution (as compared to the input particles) which can help improve downstream refinements. Excluded particles are taken from over-populated viewing direction bins. Viewing direction bins are defined by a set of direction vectors on the unit sphere (generated via Fibonacci sampling). Each particle is sorted into the bin nearest to the particle’s viewing direction. Once the bins are populated, they are sorted by particle count and all bins with a population above the Kth percentile are deemed overpopulated. Each overpopulated bin has particles excluded until it reaches the population of the Kth percentile bin. N.B., the Rebalance Orientations job is \*idempotent,\* in the sense that if two Rebalance Orientations jobs are run in sequence (with the same parameters), the second job will make no change to its inputs. ## Inputs ### Particles Particles to be rebalanced. These particles must have 3D pose estimates, and so should come from a 3D refinement such as \[Homogeneous\](/processing-data/all-job-types-in-cryosparc/3d-refinement/job-homogeneous-refinement.md) or \[Non-Uniform Refinement\](/processing-data/all-job-types-in-cryosparc/3d-refinement/job-non-uniform-refinement-new.md). ## Commonly Adjusted Parameters ### Number of orientation bins Particles will be grouped into this number of bins. When views are rebalanced, they are considered per-bin. Put another way, Rebalance Orientations does not consider a \*single view\* as overrepresented, but rather a bin of similar views. In general we have found the default setting to be sufficient. If orientation bias is focused on a single, narrow region of orientation space, using more bins may focus Rebalance Orientations on only particles in that view. ### Rebalance percentile This number (between 0 and 100) sets the percentile above which bins are rebalanced. Note that this parameter operates on \*bins\*, not \*particles\*. Put another way, setting this parameter to \`80\` will result in bins in the 80th percentile and higher (that is, bins with more particles in them than the bottom 80 percent) having particles removed until they have the same number as the 80th percentile. Note that setting this parameter to \`80\` does not mean that precisely 20% of particles would be removed by this operation; the number of particles removed depends on the orientation distribution of the particles. ### Intra-bin exclusion criterion Once a bin is marked as overrepresented, particles in that bin must be discarded. This parameter provides several choices of how particles are to be selected for exclusion. #### Random Particles are selected at random and discarded until the bin is at the threshold. #### pick\\\_stats/ncc\\\_score Particles with the worst Normalized Cross Correlation score are removed. The Normalized Cross Correlation is calculated during particle picking and measures how well the particle image matches the template, blob, or ring used to select the particle. #### alignments3D/error Particles with the greatest error between the volume projection and the particle image, computed during upstream refinement, are removed. Note that if the volume is highly anisotropic due to orientation bias, the error value may be unreliable. #### alignments3D/alpha Particles with the lowest per-particle scale are discarded. Per-particle scale accounts for local variations in greyscale and is typically taken as a proxy for ice thickness, but would also be affected by overall particle quality and other factors. Note that if the volume is highly anisotropic due to orientation bias, the per-particle scale may be unreliable. #### alignments2D/error Particles with the greatest error between the 2D class average and the particle image are discarded. Note that this error value will capture a combination of \*pose error\* and residual error in the refinement of the class average itself. If relatively few classes were requested during 2D classification, good particles may have high error if their true pose is far from the nearest available class average. ## Outputs ### Rebalanced particles Particles that fell in bins which were below the threshold and particles that fell in overpopulated bins but were selected by the \`Intra-bin exclusion criterion\` are collected together in this output. They are otherwise unchanged from the input. ### Excluded particles Particles that fell in overpopulated bins and were excluded by the \`Intra-bin exclusion criterion\` are collected in this output. They are otherwise unchanged from the input. ### Plots Rebalance Orientations produces\[ Viewing Direction Distribution\](/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots.md#viewing-direction-distribution) plots from before and after the rebalancing operation is performed. These plots are simple heatmaps of particle viewing directions, explained more in the Common Plots guide page. Rebalance Orientations also produces similar plots showing the bin locations and particle counts before and after the rebalancing operation.
Finally, a plot of the rebalancing operation itself is produced. Bins are arranged in increasing order of initial particle count. Bins with a number of particles less than or equal to the threshold are marked with a blue point. Bins with particles in excess of the threshold are represented with two points: a blue point at the threshold, and a red point indicating the number of particles removed from that bin.
Note that each position along the X-axis in this plot represents the \*\*number of particles in an individual bin\*\*, not the cumulative sum of particles in that number of bins. ## Common Next Steps Removal of overrepresented views may help reduce map anisotropy in some cases. However, the overall quality of the map may also be degraded when removing good information in any orientation. Thus, a potentially useful workflow involves using this job and a subsequent refinement to produce a more isotropic map, then using this map to repeat particle picking (e.g., using \[Create Templates\](/processing-data/all-job-types-in-cryosparc/particle-picking/job-create-templates.md) and the \[Template Picker\](/processing-data/all-job-types-in-cryosparc/particle-picking/job-template-picker.md) jobs, or in a neural-network particle picker such as \[TOPAZ\](/processing-data/all-job-types-in-cryosparc/deep-picking/topaz.md)) to attempt to find more of the rare views in the existing data. ## References 1. Campbell, M. G., Veesler, D., Cheng, A., Potter, C. S. & Carragher, B. 2.8 Å resolution reconstruction of the Thermoplasma acidophilum 20S proteasome using cryo-electron microscopy. \*eLife\* \*\*4\*\*, e06380 (2015). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/job-rebalance-orientations.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/cryo-em-foundations/image-formation/aliasing.md). # Aliasing {% hint style="info" %} \*\*Summary\*\*: When a signal is sampled at some sampling rate, the maximum frequency that sample can represent is twice the sampling rate. In images, the sampling rate is the pixel size. Thus, the pixel size sets an upper bound on the best resolution achievable for a given set of images. {% endhint %} ## Aliasing In the physical world, signals are continuous. The pressure waves which make up sound vary smoothly between high and low pressure; light waves which form images vary from light to dark, etc. However, computers must represent these signals with discrete samples. When we sample a signal, we take measurements of that signal at evenly spaced positions in space or time. For a concrete example, consider the samples below:
Seven dots, clearly arranged along a single cycle of a sine wave
By eye, it seems obvious that these samples come from a simple sine wave which oscillates once:
The seven points are connected by a sine wave which oscillates once
However, these samples are explained just as well by a wave which oscillates seven times:
The same seven points are now connected with a sine wave which oscillates seven times
or thirteen times:
The same seven points are connected by a wave which oscillates thirteen times
Indeed, there are in fact an infinite family of waves which perfectly fit \*any finite set of samples\*. There is no way of knowing which of these infinite waves truly gave rise to the samples we observe, so by convention we select the lowest-frequency wave which fits the observed data. However, what happens when there really \*is\* high-frequency information in our images which our samples cannot capture? ### Nyquist Frequency Say you collect an image of some object using a camera sensor which is 12 Å wide and has 6 pixels. Your pixel size is therefore 2 Å. Put another way, you are sampling the incoming image with a \*sampling rate\* of $$\\frac{1}{2 \\AA}$$ , or one sample for every two Å. If your object is a sine wave which oscillates with a frequency of $$\\frac{1}{12 \\AA{}}$$, the image is unambiguous:
The same set of points are shown twice. At top, they are connected by a sine wave which oscillates once. On the bottom, they are connected by steps at the height of each sample, and a dotted wave which oscillates once.
There are plenty of samples along the entire wave to accurate capture its shape. What happens when the wave’s frequency increases?
The wave at the top now oscillates twice, and the steps at the bottom also oscillate twice. The dotted wave is correct, also oscillating twice.
When you image an object with a wavelength of 6 Å, the result is still correct. The exact position of the object relative to the pixels has imposed some asymmetry in the image, but if the object shifted left or right the result would again be symmetric. What about an object with even higher frequency?
The wave at the top now oscillates four times. At the bottom, the steps only oscillate twice, so the dotted wave only oscillates twice as well. The phase of the steps and dotted wave are also flipped.
When the object has a wavelength of 3 Å, the samples line up perfectly with samples from a 6 Å wave as well. Because we always take the lowest frequency wave that explain our data, we incorrectly interpret our image as resulting from a 6 Å wave. This incorrect result is called “aliasing” and is an important effect in SPA. The frequency beyond which aliasing occurs is called the Nyquist frequency, and is half the sampling frequency. $$ f\\\_{\\mathrm{Nyquist}} = \\frac{1}{2} \\times f\\\_{\\mathrm{sampling}} = 2 \\times \\frac{1}{\\mathrm{pixel\\ size}} $$ Put another way, any spatial features in the object which are smaller than twice the pixel size will be aliased in the image. This results in the commonly quoted maximum resolution for a particle stack of twice the image pixel size. Frequencies faster than the Nyquist frequency alias to the frequency below Nyquist and the same distance away from it. This operation is commonly described as “folding over” the Nyquist frequency. Note also that in the 3 Å wavelength example above, the aliased 6 Å wave has opposite phase. This is another property of aliasing: the “folding” operation also flips the phase of the aliased wave.
An arrow pointing up at a frequency of 0.6 times the sampling frequency is "folde" across the Nyquist frequency to become an arrow pointing down (representing a phase shift) at 0.4 times the sampling frequency
{% hint style="info" %} The phase flipping here is a property of all discrete representations of continuous signals. It is distinct from the specific mechanism of phase inversion that occurs in the electron microscope, which is modeled by the CTF. {% endhint %} In the animation below, the moment the true frequency crosses the Nyquist frequency the aliased wave begins to decrease in frequency, rather than increase, since it is folding over the Nyquist frequency.
### CTF Aliasing Just as particle images are represented with pixels in real space, their Fourier transform is represented with pixels in Fourier space. The box in Fourier space is the same size as the box in real space, but each pixel represents a certain range of frequencies rather than a certain region of space. Specifically, $$ \\mathrm{1\\ Fourier\\ pixel} = \\frac{1}{N \\times \\mathrm{pixel\\ size}} $$ where N is the box size in pixels and the pixel size is the size of a pixel in the real space image. For instance, if a particle is represented with a box of 128 pixels which each represent 2 Å, the Fourier space particle image has a Nyquist frequency of $$\\frac{1}{2 \\times 128} \\mathrm{\\AA{}^{-1}} = \\frac{1}{256} \\mathrm{\\AA{}^{-1}}$$. If the CTF oscillates with a frequency greater than this, the contrast transfer function will be aliased. Consider the true contrast transfer function of an image collected in a typical electron microscope used in single-particle analysis (300 kV, 2.7 Cs) with 2 µm defocus.
A graph of the contrast transfer function
However, we cannot capture the full, continuous contrast transfer function. We must model it from our images, which are sampled using discrete Fourier pixels. If the Fourier pixels are small, the resulting image of the contrast transfer function may look like this:
The sane CTF as above, but discretely sampled. A the highest frequencies the amplitude is slightly off, but the overall signal is correct.
At the higher frequencies there is some disruption of the amplitude, but the representation is mostly accurate. However, if the Fourier pixels are too large, the contrast transfer function will be significantly aliased.
The same CTF as above, but now the sampling rate is too low to accurately capture the CTF.
In this rather extreme example, the contrast transfer function modeled from the image (black) is significantly different from the true contrast transfer function (grey), especially past approximately 4 Å. This happens because in this region the contrast transfer function starts oscillating more quickly than the Nyquist limit of the Fourier space image. The high-frequency contrast transfer function oscillations are therefore aliased to incorrect lower frequencies. This effect is more pronounced with: \* higher defocus values, which increase the true oscillation rate of the CTF, \* smaller pixel sizes, which increase the real space Nyquist limit, requiring the Fourier image to represent a larger region of the CTF, and \* smaller box sizes, which reduce the sampling rate in Fourier space. Since defocus cannot be changed after acquisition, if significant CTF aliasing is observed, the remaining means of removing it are therefore \* downsampling, which “zooms in” on a subregion of the CTF by removing higher frequencies, or \* using a larger box which gives more pixels in Fourier space to represent the CTF.
Four graphs of the CTF are shown. At the top is the continuous CTF. Below that, the aliased CTF shown previously. Third from the top, the aliased CTF is shown for a downsampled image. This produces a CTF which only extends to around 4 Å, where aliasing is not significant. Finally, the CTF is shown from a bigger box. The larger box means the CTF has more samples, reducing aliasing.
Of course, the problem of contrast transfer function aliasing is often rendered moot by the poor signal-to-noise ratio of cryo-EM data. For instance, even the severely-aliased example above more-or-less correctly models data up to the resolution to which the CTF fits the data well. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/cryo-em-foundations/image-formation/aliasing.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/deep-picking/topaz/topaz-tutorial.md). # T20S Proteasome: Topaz Particle Picking Tutorial ## Step 1 - Preprocess Data \* Preprocess the T20S subset by completing steps 1-12 in the tutorial found in the \*\*Cryo-EM Data Processing in CryoSPARC: Introductory Tutorial\*\* section found here: {% content-ref url="/pages/-M7DHIKNHCzB43PMPbZD" %} \[Get Started with CryoSPARC: Introductory Tutorial (v3)\](/guides-for-v3/cryo-em-data-processing-in-cryosparc-introductory-tutorial.md) {% endcontent-ref %} \* Ensure that the \*\*Inspect Picks\*\*, and \*\*Select 2D\*\* jobs from the linked tutorial are completed as they will be required for training the Topaz model. The ouputs from these two jobs will be used as inputs for the Topaz-related jobs. ## Step 2 - Create Training Job \* Select \*\*Topaz Train (BETA)\*\* from the Job Builder. Drag and drop the \*\*micrographs\*\* output from the completed \*\*Inspect Picks\*\* job and the \*\*particles\\\_selected\*\* output from the completed \*\*Select 2D\*\* job into the \*\*micrographs\*\* and \*\*particles\*\* inputs respectively. \* Use the file browser (activated by clicking the folder icon) to locate the Topaz executable path found in the Deep Picking section for the \*\*Path to Topaz Executable\*\* field. Instructions on how to find the Topaz executable path can be found above. \* Modify the \*\*Downsampling factor\*\* parameter to 16. This parameter reduces the size of the input micrographs by the factor input and is often necessary to conform to a system's memory constraints. \* Modify the \*\*Expected number of particles\*\* parameter to 300. \* Queue the job. \* The job is training a Topaz model on the subset of 20 micrographs from the T20S tutorial. It is highly recommended to train deep picker models on subsets of micrographs as acquiring training picks for all micrographs takes time and is not required. Once the Topaz model learns on a sufficient subset of the micrographs, it can pick particles from the entire dataset. !\[Passing inputs to the Topaz Train job\](/files/-M7DHJnLyKB3wN-OgmXs) ## Step 3 - Create Topaz Extract Job \* Select \*\*Topaz Extract (BETA)\*\* from the Job Builder. Drag and drop both the \*\*topaz\\\_model\*\* and \*\*micrographs\*\* outputs from the Topaz Train job into the corresponding inputs on the Job Builder. \* Use the file browser (activated by clicking the folder icon) to locate the Topaz executable path found earlier for the \*\*Path to Topaz Executable\*\* field. \* Queue the job. \* The job is using the trained Topaz model to infer picks from the input micrographs. Even though in this tutorial, the job is picking from the same micrographs used to train the model, a properly trained model will infer picks that were not used as training picks from the micrographs. !\[Passing inputs to the Topaz Extract job\](/files/-M7DHJnM7eO3DkWZKEN8) ## Step 4 - Acquire Particles from Topaz Extract \* Select \*\*Extract from Micrographs\*\* from the Job Builder. Drag and drop both the \*\*micrographs\*\* and the \*\*particles\*\* outputs from the Topaz Extract job into the corresponding inputs on Job Builder. \* Queue the job. \* This job will update the particle picks with information required for further processing. \* Select \*\*2D Classification\*\* from the Job Builder. Drag and drop the \*\*particles\*\* output from the \*\*Extract from Micrographs\*\* job into the \*\*particles\*\* input of the \*\*2D Classification\*\* job. \* Queue the job. \* Select \*\*Select 2D classes\*\* from the Job Builder. Drag and drop both outputs of the \*\*2D Classification\*\* job into their corresponding inputs in the \*\*Select 2D classes\*\* job. \* Queue the job. \* Wait for the job status to change to "Waiting" and then select the particle templates that should be kept for further processing. \* The \*\*2D Classification\*\* and \*\*Select 2D classes\*\* jobs serve to filter out unwanted particles from the particle picking. Once the \*\*Select 2D classes\*\* job is complete, the \*\*particles\*\* output from the job can be used as particle picks to process further into the pipeline. ## Next Steps Now that a basic Topaz pipeline has been completed, the more advanced aspects of particle picking with Topaz can be explored. The following are some of these aspects: \* Ideally, deep picking models are trained on a subset of micrographs and then perform inference on an entire dataset, as mentioned before. The Topaz model trained in this tutorial can be applied to the entire T20S dataset rather than the subset used in this tutorial. Potential refinement results will improve with the resultant increased number of picks. \* The \`Topaz Train\` and \`Topaz Cross Validation\` jobs has many training parameters that can be fine tuned to affect the quality of the model. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/deep-picking/topaz/topaz-tutorial.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \# CryoSPARC Guide ## CryoSPARC Guide - \[About CryoSPARC™\](https://guide.cryosparc.com/readme.md): General information about the software platform. - \[Licensing\](https://guide.cryosparc.com/licensing.md): Non-profit and commercial licensing options. - \[Non-commercial license agreement\](https://guide.cryosparc.com/licensing/non-commercial-license-agreement.md): Full text of CryoSPARC non-commercial license agreement. Please send any queries to: info@structura.bio. - \[CryoSPARC Architecture and System Requirements\](https://guide.cryosparc.com/setup-configuration-and-management/hardware-and-system-requirements.md): Description of CryoSPARC HPC software system architecture, typical setups (e.g., workstation, cluster). - \[CryoSPARC Installation Prerequisites\](https://guide.cryosparc.com/setup-configuration-and-management/cryosparc-installation-prerequisites.md): Before installing CryoSPARC, ensure these six requirements are met. - \[How to Download, Install and Configure\](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure.md): Meeting system requirements, obtaining a License ID, and downloading & installing CryoSPARC. - \[Obtaining A License ID\](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/obtaining-a-license-id.md): Fill out the form to obtain a License ID required for installing and using CryoSPARC. - \[Downloading and Installing CryoSPARC\](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/downloading-and-installing-cryosparc.md): Downloading and installing the cryosparc\\\_master and cryosparc\\\_worker packages. - \[CryoSPARC Cluster Integration Script Examples\](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/cryosparc-cluster-integration-script-examples.md): Examples of cluster\\\_info.json and cluster\\\_script.sh scripts for various cluster workload managers - \[Accessing the CryoSPARC User Interface\](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/accessing-cryosparc.md): Viewing the user interface locally and from home - \[(Optional) Hosting CryoSPARC Through a Reverse Proxy\](https://guide.cryosparc.com/setup-configuration-and-management/how-to-download-install-and-configure/optional-hosting-cryosparc-through-a-reverse-proxy.md) - \[Software Updates and Patches\](https://guide.cryosparc.com/setup-configuration-and-management/software-updates.md): How to get the latest CryoSPARC features and fixes or roll back to a previous version. - \[Management and Monitoring (≤v4.7)\](https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring-4.7.md): Instructions for accessing and working in the CryoSPARC command line. - \[Environment variables (≤v4.7)\](https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring-4.7/environment-variables-v4.7.md): (Advanced) Specify additional environment variables in the configuration files to augment CryoSPARC's low-level behaviour. - \[cryosparcm reference (≤v4.7)\](https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring-4.7/cryosparcm-4.7.md): How to use the cryosparcm utility for starting and stopping the CryoSPARC instance, checking status or logs, managing users and using CryoSPARC's command-line interface. - \[cryosparcm cli reference (≤v4.7)\](https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring-4.7/cli-4.7.md): How to use CryoSPARC's low-level command-line interface. - \[cryosparcw reference (≤v4.7)\](https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring-4.7/cryosparcw-4.7.md): How to use the cryosparcw utility for managing CryoSPARC workers - \[Management and Monitoring (v5.0+)\](https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring-v5.0.md): Instructions for accessing and working in the CryoSPARC command line. - \[Environment Variables (v5.0+)\](https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring-v5.0/environment-variables-v5.0.md): (Advanced) Specify additional environment variables in the configuration files to augment CryoSPARC's low-level behaviour. - \[cryosparcm reference (v5.0+)\](https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring-v5.0/cryosparcm-reference-v5.0.md): How to use the cryosparcm utility for starting and stopping the CryoSPARC instance, checking status or logs, managing users and using CryoSPARC's command-line interface. - \[cryosparcm cli reference (v5.0+)\](https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring-v5.0/cryosparcm-cli-reference-v5.0.md): How to use CryoSPARC's low-level command line interface to perform actions that can be performed in the UI. - \[cryosparcw reference (v5.0+)\](https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring-v5.0/cryosparcw-reference-v5.0.md): How to use the cryosparcw utility for managing CryoSPARC workers - \[Software System Guides\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides.md): CryoSPARC software management guides. - \[Guide: Updating to CryoSPARC v5\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/guide-updating-to-cryosparc-v5.md): CryoSPARC v5 is backwards compatible with v4. The update process includes new validation steps that may take some time, up to one hour for larger instances. - \[Guide: Updating to CryoSPARC v4\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/guide-updating-to-cryosparc-v4.md): Installing or updating to CryoSPARC v4 is similar to previous versions of CryoSPARC, but downgrading is not possible past v3.4.0. - \[Guide: Installation Testing with cryosparcm test\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/guide-installation-testing-with-cryosparcm-test.md): This guide covers how to use cryosparcm test to verify your CryoSPARC installation is working properly. - \[Guide: Verify CryoSPARC Installation with the Extensive Validation Job (v4.3+)\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/tutorial-verify-cryosparc-installation-with-the-extensive-workflow-sysadmin-guide.md) - \[Guide: Verify CryoSPARC Installation with the Extensive Workflow (≤v4.2)\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/tutorial-verify-cryosparc-installation-with-the-extensive-workflow-sysadmin-guide-1.md) - \[Guide: Performance Benchmarking (v4.3+)\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/guide-performance-benchmarking-v4.3.md): This guide covers the new benchmarking tool in CryoSPARC that allows for benchmarking a worker’s filesystem, CPUs and GPUs. Available in CryoSPARC v4.3.0+. - \[Guide: Download Error Reports\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/guide-download-error-reports.md): How to download job and system-level error reports from within the application. - \[Guide: Maintenance Mode and Configurable User Facing Messages\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/guide-maintenance-mode-and-configurable-user-facing-messages.md): Pause the job queue during updates and set optional user facing messages. - \[Guide: User Management\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/tutorial-user-management.md): User creation, management, setting roles and password management through the CryoSPARC user interface. - \[Guide: Multi-user Unix Permissions and Data Access Control\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/unix-permissions-and-data-access-control.md): Tips on how to manage permissions and data access control. - \[Guide: Lane Assignments and Restrictions\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/guide-lane-assignments-and-restrictions.md): Assigning CryoSPARC users to specific scheduler lanes. - \[Guide: Priority Job Queuing\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/tutorial-priority-job-queuing.md): How to prioritize jobs to override the CryoSPARC scheduler's default behaviour. - \[Guide: Configuring Custom Variables for Cluster Job Submission Scripts\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/guide-configuring-custom-variables-for-cluster-job-submission-scripts.md) - \[Guide: SSD Particle Caching in CryoSPARC\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/tutorial-ssd-particle-caching-in-cryosparc.md): Overview of how SSD particle caching works, how much SSD space you need, configuration options and troubleshooting. - \[Guide: Data Management in CryoSPARC (v4.0+)\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/guide-data-management-in-cryosparc-v4.0.md): An overview of all data management utilities and common use cases. - \[Guide: Data Cleanup (v4.3+)\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/guide-data-cleanup-v4.3.md): New features in v4.3+ for managing and cleaning up project data. - \[Guide: Reduce Database Size (v4.3+)\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/guide-reduce-database-size-v4.3.md): A guide on reducing the size of large CryoSPARC databases using methods provided by MongoDB. - \[Guide: CryoSPARC Live Session Data Management (≤v4.7)\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/cryosparc-live-session-data-management-4.7.md): How to manage the data created by your cryoSPARC Live Sessions via the user interface and data management API. - \[Guide: Instance Recovery (v5.0+)\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/guide-instance-recovery-v5.0.md): How to recover a CryoSPARC instance if the database directory is corrupted or lost. - \[Guide: Migrating your CryoSPARC Instance\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/tutorial-migrating-your-cryosparc-instance.md): A guide to moving CryoSPARC from one location to another. - \[Deploying CryoSPARC on AWS\](https://guide.cryosparc.com/setup-configuration-and-management/cryosparc-on-aws.md): Version 1.0 (May 10, 2021) - \[Performance Benchmarks\](https://guide.cryosparc.com/setup-configuration-and-management/cryosparc-on-aws/performance-benchmarks-aws.md): Version 1.0 (May 10, 2021) - \[Troubleshooting\](https://guide.cryosparc.com/setup-configuration-and-management/troubleshooting.md): Overview of common issues and advice on how to resolve them. - \[A Tour of the CryoSPARC Interface\](https://guide.cryosparc.com/application-guide/a-tour-of-the-cryosparc-interface.md) - \[Browsing the CryoSPARC Instance\](https://guide.cryosparc.com/application-guide/browsing-the-cryosparc-instance.md) - \[Projects, Workspaces and Live Sessions\](https://guide.cryosparc.com/application-guide/projects-workspaces-and-live-sessions.md) - \[Jobs\](https://guide.cryosparc.com/application-guide/jobs.md) - \[Job Views: Cards, Tree, and Table\](https://guide.cryosparc.com/application-guide/job-views-cards-tree-and-table.md) - \[Creating and Running Jobs\](https://guide.cryosparc.com/application-guide/creating-and-running-jobs.md): Working with the Job Builder, Job Cart and Job Quick Actions. - \[Inspecting Job Data\](https://guide.cryosparc.com/application-guide/inspecting-job-data.md) - \[Low Level Results Interface\](https://guide.cryosparc.com/application-guide/low-level-results-interface.md) - \[Filters and Sorting\](https://guide.cryosparc.com/application-guide/filters-and-sorting.md) - \[View Options\](https://guide.cryosparc.com/application-guide/view-options.md) - \[Tags\](https://guide.cryosparc.com/application-guide/tags.md) - \[Flat vs Hierarchical Navigation\](https://guide.cryosparc.com/application-guide/flat-vs-hierarchical-navigation.md) - \[File Browser\](https://guide.cryosparc.com/application-guide/file-browser.md): When selecting a path to create a project or input for a job parameter, CryoSPARC will display an integrated file browser. - \[Blueprints\](https://guide.cryosparc.com/application-guide/blueprints.md) - \[Workflows\](https://guide.cryosparc.com/application-guide/workflows.md) - \[Managing Jobs\](https://guide.cryosparc.com/application-guide/managing-jobs.md) - \[Interactive Jobs\](https://guide.cryosparc.com/application-guide/interactive-jobs.md) - \[Upload Local Files\](https://guide.cryosparc.com/application-guide/upload-local-files.md): Upload files from your local computer to CryoSPARC directly in the browser - \[Managing Data\](https://guide.cryosparc.com/application-guide/managing-data.md) - \[Downloading and Exporting Data\](https://guide.cryosparc.com/application-guide/downloading-and-exporting-data.md) - \[Instance Management\](https://guide.cryosparc.com/application-guide/instance-management.md) - \[Admin Panel\](https://guide.cryosparc.com/application-guide/admin-panel.md) - \[Keyboard Shortcuts\](https://guide.cryosparc.com/application-guide/keyboard-shortcuts.md) - \[Image Formation\](https://guide.cryosparc.com/cryo-em-foundations/image-formation.md) - \[Contrast in Cryo-EM\](https://guide.cryosparc.com/cryo-em-foundations/image-formation/contrast-in-cryo-em.md): Where does contrast in an electron micrograph come from? What can we do to produce more contrast? What effects do we have to take into account when processing data from electron micrographs? - \[Waves as Vectors\](https://guide.cryosparc.com/cryo-em-foundations/image-formation/waves-as-vectors.md) - \[Aliasing\](https://guide.cryosparc.com/cryo-em-foundations/image-formation/aliasing.md) - \[Expectation Maximization in Cryo-EM\](https://guide.cryosparc.com/expectation-maximization-in-cryo-em.md): An overview of the expectation maximization algorithm. - \[Get Started with CryoSPARC: Introductory Tutorial (v4.0+)\](https://guide.cryosparc.com/processing-data/get-started-with-cryosparc-introductory-tutorial.md): In this tutorial, we will process a small dataset from movies to reconstructed density map. If you are new to data processing in CryoSPARC, we highly recommend following along! - \[Tutorial Videos\](https://guide.cryosparc.com/processing-data/tutorial-videos.md): CryoSPARC tutorial videos. - \[All Job Types in CryoSPARC\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc.md): Details on all of the available job types in CryoSPARC, when to use them, required inputs, parameter explanations, and common next steps. - \[Import\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/import.md): Options for bringing data into CryoSPARC for processing. - \[Job: Import Movies\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/import/job-import-movies.md) - \[Job: Import Micrographs\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/import/job-import-micrographs.md) - \[Job: Import Particle Stack\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/import/job-import-particle-stack.md) - \[Job: Import 3D Volumes\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/import/job-import-3d-volumes.md) - \[Job: Import Templates\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/import/job-import-templates.md) - \[Job: Import Result Group\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/import/job-import-result-group.md) - \[Job: Import Beam Shift\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/import/job-import-beam-shift.md) - \[Motion Correction\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/motion-correction.md) - \[Job: Patch Motion Correction\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/motion-correction/job-patch-motion-correction.md) - \[Job: Full-Frame Motion Correction\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/motion-correction/job-full-frame-motion-correction.md) - \[Job: Local Motion Correction\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/motion-correction/job-local-motion-correction.md): Local motion correction. - \[Job: MotionCor2 (Wrapper) (BETA)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/motion-correction/job-motioncor2-wrapper-beta.md): How to use the wrapper for MotionCor2 available in CryoSPARC. - \[Job: Reference Based Motion Correction (BETA)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/motion-correction/job-reference-based-motion-correction-beta.md) - \[CTF Estimation\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-estimation.md) - \[Job: Patch CTF Estimation\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-estimation/job-patch-ctf-estimation.md): Patch-based CTF estimation. - \[Job: Patch CTF Extraction\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-estimation/job-patch-ctf-extraction.md): Patch CTF extraction. - \[Job: CTFFIND4 (Wrapper)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-estimation/job-ctffind4-wrapper.md) - \[Job: Gctf (Wrapper) (Legacy)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-estimation/job-gctf-wrapper-legacy.md) - \[Exposure Curation\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/exposure-curation.md): Curate micrographs (exposures) to remove low-quality data. - \[Job: Micrograph Denoiser (BETA)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/exposure-curation/job-micrograph-denoiser-beta.md) - \[Job: Micrograph Junk Detector (BETA)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/exposure-curation/job-micrograph-junk-detector-beta.md) - \[Interactive Job: Manually Curate Exposures\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/exposure-curation/interactive-job-manually-curate-exposures.md): Inspect, curate and reject exposures to remove low quality data. - \[Particle Picking\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-picking.md): Identifying and selecting individual particles. - \[Interactive Job: Manual Picker\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-picking/interactive-job-manual-picker.md): Manual picking. - \[Job: Blob Picker\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-picking/job-blob-picker.md): Blob-based picking. - \[Job: Template Picker\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-picking/job-template-picker.md): Template-based picking. - \[Job: Filament Tracer\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-picking/job-filament-tracer-beta.md): Particle picking for filaments. - \[Job: Blob Picker Tuner\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-picking/job-blob-picker-tuner.md): Optimize blob picker parameters. - \[Interactive Job: Inspect Particle Picks\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-picking/interactive-job-inspect-particle-picks.md): Inspect and adjust particle picks. - \[Job: Create Templates\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-picking/job-create-templates.md): Create templates for particle picking. - \[Extraction\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/extraction.md): Extracting picked particles from micrographs and related utilities. - \[Job: Extract from Micrographs\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/extraction/job-extract-from-micrographs.md): Extract picked particles from micrographs using a specified box size. - \[Job: Downsample Particles\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/extraction/job-downsample-particles.md): Downsample particles to save space. - \[Job: Restack Particles\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/extraction/job-restack-particles.md): Restack particles to manage your particle files and speed up caching - \[Deep Picking\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/deep-picking.md): Overview of deep particle picking methods available through CryoSPARC. - \[Guideline for Supervised Particle Picking using Deep Learning Models\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/deep-picking/guideline-for-supervised-particle-picking-using-deep-learning-models.md): Supervised particle picking. - \[Topaz (Bepler, et al)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/deep-picking/topaz.md): Overview of the Topaz wrapper available through CryoSPARC. - \[T20S Proteasome: Topaz Particle Picking Tutorial\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/deep-picking/topaz/topaz-tutorial.md): Topaz particle picking tutorial via the Topaz wrapper available in CryoSPARC. - \[T20S Proteasome: Topaz Micrograph Denoising Tutorial\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/deep-picking/topaz/t20s-proteasome-deep-micrograph-denoising-tutorial.md): Topaz denoising tutorial via the Topaz wrapper available in CryoSPARC. - \[Job: Topaz Train and Job: Topaz Cross Validation\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/deep-picking/topaz/job-topaz-train-beta.md): Topaz job types available via wrapper in CryoSPARC. - \[Job: Topaz Extract\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/deep-picking/topaz/job-topaz-extract-beta.md): Topaz Extract job available via wrapper in CryoSPARC. - \[Job: Topaz Denoise\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/deep-picking/topaz/job-topaz-denoise-beta.md): Topaz Denoise job available via wrapper in CryoSPARC. - \[Deep Network Particle Picker (≤v4.7)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/deep-picking/deep-network-particle-picker.md): Overview of the deep particle picker available through CryoSPARC. - \[T20S Proteasome: Deep Particle Picking Tutorial (≤v4.7)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/deep-picking/deep-network-particle-picker/t20s-proteasome-deep-particle-picking-tutorial.md): Deep particle picking tutorial. - \[Job: Deep Picker Train and Job: Deep Picker Inference (≤v4.7)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/deep-picking/deep-network-particle-picker/job-deep-picker-train-beta-and-deep-picker-extract-beta.md): Deep picking job types. - \[Particle Curation\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation.md): 2D classification and curation of particles following particle picking. - \[Job: 2D Classification\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/job-2d-classification.md) - \[Interactive Job: Select 2D Classes\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/interactive-job-select-2d-classes.md) - \[Job: Reference Based Auto Select 2D (BETA)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/job-reference-based-auto-select-2d-beta.md) - \[Job: Reconstruct 2D Classes\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/job-reconstruct-2d-classes.md) - \[Job: Rebalance 2D Classes\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/job-rebalance-2d-classes-beta.md) - \[Job: Class Probability Filter (Legacy)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/job-class-probability-filter.md): Select a subset of extracted particles. - \[Job: Rebalance Orientations\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/job-rebalance-orientations.md) - \[Job: Subset Particles by Statistic\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/job-subset-particles-by-statistic.md) - \[3D Reconstruction\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-reconstruction.md): Reconstruction of 3D volumes. - \[Job: Ab-Initio Reconstruction\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-reconstruction/job-ab-initio-reconstruction.md): Ab-Initio reconstruction. - \[Job: Homogeneous Ab-Initio Refinement (BETA)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-reconstruction/job-homogeneous-ab-initio-refinement-beta.md) - \[3D Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement.md) - \[Job: Homogeneous Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-homogeneous-refinement.md): Homogeneous refinement. - \[Job: Heterogeneous Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-heterogeneous-refinement.md): Heterogeneous refinement. - \[Job: Non-Uniform Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-non-uniform-refinement-new.md): Non-uniform refinement. - \[Job: Homogeneous Reconstruction Only\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-homogeneous-reconstruction-only.md): Homogeneous reconstruction only (no alignment). - \[Job: Heterogeneous Reconstruction Only\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-heterogeneous-reconstruction-only.md): Reconstruct class volumes from heterogeneous refinement or 3D classification - \[Job: Homogeneous Refinement (Legacy)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-homogeneous-refinement-legacy.md): Homogeneous refinement (Legacy). - \[Job: Non-uniform Refinement (Legacy)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-non-uniform-refinement-legacy.md): Non-uniform refinement (Legacy). - \[CTF Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-refinement.md): CTF refinement jobs and tutorials. - \[Job: Global CTF Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-refinement/job-global-ctf-refinement.md): Per-group beam tilt, trefoil, spherical aberration, tetrafoil. - \[Job: Local CTF Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-refinement/job-local-ctf-refinement.md): Per-particle defocus. - \[Job: Exposure Group Utilities\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-refinement/job-exposure-group-utilities.md): Exposure group utilities for combining or splitting exposure or particle datasets. - \[Conformational Variability\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability.md): Exploring conformational variability. - \[Job: 3D Variability\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-3d-variability.md): 3DVA - \[Job: 3D Variability Display\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-3d-variability-display.md): 3D variability display. - \[Job: 3D Classification\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-3d-classification-beta.md): 3D classification without alignment. - \[Job: Regroup 3D Classes\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-regroup-3d-classes.md): Regroup 3D classes via spectral clustering - \[Job: Reference Based Auto Select 3D (BETA)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-reference-based-auto-select-3d-beta.md) - \[Job: 3D Flexible Refinement (3DFlex) (BETA)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-3d-flexible-refinement-3dflex-beta.md) - \[Postprocessing\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing.md): Postprocessing jobs in CryoSPARC. - \[Job: Sharpening Tools\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-sharpening-tools.md): Re-sharpen and adjust the B-factor of a volume after refinement. - \[Job: DeepEMhancer (Wrapper)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-deepemhancer-wrapper.md): Overview of the DeepEMhancer Wrapper in CryoSPARC. - \[Job: Validation (FSC)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-validation-fsc.md): FSC calculation. - \[Job: Local Resolution Estimation\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-resolution-estimation.md): Local resolution estimation. - \[Job: Local Filtering\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-filtering.md): Locally filter a refined map. - \[Job: ResLog Analysis\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-reslog-analysis.md): ResLog analysis. - \[Job: ThreeDFSC (Wrapper) (Legacy)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-threedfsc-wrapper-legacy.md): Overview of the ThreeDFSC Wrapper in CryoSPARC. - \[Local Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement.md): Local refinement and particle subtraction. - \[Job: Local Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-new-local-refinement-beta.md): Local refinement. - \[Job: Particle Subtraction\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta.md): Particle subtraction (signal subtraction). - \[Job: Local Refinement (Legacy)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-local-refinement-beta.md): Local refinement (Legacy). - \[Helical Reconstruction\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta.md): Helical processing. - \[Helical symmetry in CryoSPARC\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/helical-symmetry-in-cryosparc.md): Explanation of treatment of helical symmetry in CryoSPARC. - \[Job: Helical Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-helical-refinement-beta.md): Helical refinement. - \[Job: Symmetry search utility\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-symmetry-search-utility-beta.md): Symmetry search utility. - \[Job: Average Power Spectra\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta/job-average-power-spectra.md) - \[Utilities\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities.md): Utility jobs in CryoSPARC. - \[Job: Exposure Sets Tool\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-exposure-sets-tool.md): Split an exposure group into subsets or compare two groups. - \[Job: Exposure Tools\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-exposure-tools.md): Modify negative stain and phase plate values. - \[Job: Generate Micrograph Thumbnails\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-generate-micrograph-thumbnails.md): Generate preview thumbnails of micrographs. - \[Job: Cache Particles on SSD\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-cache-particles-on-ssd.md): Cache particles on a worker node so they're ready to be used as soon as resources are available. - \[Job: Check for Corrupt Particles\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-check-for-corrupt-particles.md): Check each particle stack .mrc file for corruption, including checking the header for length mismatch, NaN values check, and checksum verification. - \[Job: Particle Sets Tool\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-particle-sets-tool.md): Split a particle group into subsets or compare two groups. - \[Job: Reassign Particles to Micrographs\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-reassign-particles-to-micrographs.md): Link particles back to the micrographs they came from. - \[Job: Remove Duplicate Particles\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-remove-duplicate-particles.md): Remove duplicate particles. - \[Job: Symmetry Expansion\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-symmetry-expansion.md): Symmetry expansion. - \[Job: Volume Tools\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools.md): Apply various operations including upsample, downsample, crop, pad, flip, etc., to a volume or mask. - \[Job: Volume Alignment Tools\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools.md): Volume alignment tools. - \[Job: Align 3D maps\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-align-3d-maps.md): Align two or more 3D maps. - \[Job: Select Volume\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-select-volume.md): Select a volume and corresponding particle stack based on resolution. - \[Job: Split Volumes Group\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-split-volumes-group.md) - \[Job: Orientation Diagnostics\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-orientation-diagnostics.md): A new job in CryoSPARC v4.4+ to diagnose the presence of preferred orientation - \[Simulations\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations.md): Simulation jobs. - \[Job: Simulate Data (GPU)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-gpu.md): Generate simulated particles from an input volume. - \[Job: Simulate Data (Legacy)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-legacy.md): Generate simulated particles. - \[Automated Workflows\](https://guide.cryosparc.com/processing-data/automated-workflows.md): Get started with automated, end-to-end data processing in CryoSPARC. - \[CryoSPARC Tools\](https://guide.cryosparc.com/processing-data/cryosparc-tools.md) - \[Data Processing Tutorials\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies.md): Tutorials on how to use CryoSPARC jobs and processing cryo-EM data. - \[Case Study: Discrete heterogeneity in a sample of Acetogenin-bound complex I (EMPIAR 10927)\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-discrete-heterogeneity-in-a-sample-of-acetogenin-bound-complex-i-empiar-10927.md): Processing EMPIAR-10927 including separating targets using Per-particle scales or custom settings in Ab-Initio Reconstruction, and using 3DVA to guide sub-classification choices. - \[Case Study: processing of a novel motor-bound nucleosome state (EMPIAR-10739) - part 2\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-processing-of-a-novel-motor-bound-nucleosome-state-empiar-10739-part-2.md): Processing EMPIAR-10739 including using 3DVA to guide classification strategies, separating low population classes, and local refinement of a flexible region. - \[Case Study: End-to-end and exploratory processing of a motor-bound nucleosome (EMPIAR-10739)\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-and-exploratory-processing-of-a-motor-bound-nucleosome-empiar-10739.md): Processing EMPIAR-10739 including handling global pseudosymmetry, using 3DVA to guide classification strategies, separating low population classes, and local refinement of a flexible region. - \[Case study: End-to-end processing of a ligand-bound GPCR (EMPIAR-10853)\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-a-ligand-bound-gpcr-empiar-10853.md): Processing EMPIAR-10853 including Micrograph Denoising, Micrograph Junk Detector, Subset Particles by Statistic, Local Refinement and focussed 3D Classification to improve ligand density. - \[Case Study: DkTx-bound TRPV1 (EMPIAR-10059)\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-dktx-bound-trpv1-empiar-10059.md): Processing EMPIAR-10059 in CryoSPARC v4.6 with a focus on particle curation, developing an intuition for when a domain may be blurred due to flexibility, and how to handle this type of flexibility. - \[Case Study: Pseudosymmetry in TRPV5 and Calmodulin (EMPIAR-10256)\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-pseudosymmetry-in-trpv5-and-calmodulin-empiar-10256.md): Processing EMPIAR-10256 with a focus on handling pseudosymmetry and 3D Classification parameter choices. - \[Case Study: End-to-end processing of an inactive GPCR (EMPIAR-10668)\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-an-inactive-gpcr-empiar-10668.md): Processing EMPIAR-10668 including Local Refinement, 3D Variability Analysis, 3D Flex and classification of continuous heterogeneity. - \[Case Study: End-to-end processing of encapsulated ferritin (EMPIAR-10716)\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716.md): Processing EMPIAR-10716 with a focus on high-symmetry ab-initio reconstruction, local symmetry, non-point-group symmetry, symmetry expansion, and custom geometry operations. - \[Case Study: Discrete and Continuous Heterogeneity in FaNaC1 (EMPIAR-11631 and -11632)\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-discrete-and-continuous-heterogeneity-in-fanac1-empiar-11631-and-11632.md): Processing a combined dataset of an ion channel with a focus on techniques for separating conformational states from a heterogeneous mixture. - \[Case Study: Picking-induced Orientation Bias in HA Trimer (EMPIAR-10096 and -10097)\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-picking-induced-orientation-bias-in-ha-trimer-empiar-10096-and-10097.md): An analysis of the impact of data collection and particle picking techniques on orientation bias, including strategies for improving map quality and isotropy. - \[Case Study: Exploratory data processing by Oliver Clarke\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-exploratory-data-processing-by-oliver-clarke.md) - \[Case Study: Processing EMPIAR-10291 (300 Micrographs) to 3.4Å in 1 hour 25 minutes\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-processing-empiar-10291-300-micrographs-to-3.4a-in-1-hour-25-minutes.md): Processing EMPIAR-10291 from micrographs to a 3D reconstruction. - \[Tutorial: Tips for Membrane Protein Structures\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures.md): Helpful hints for processing cryo-EM data of membrane proteins. - \[Tutorial: Common CryoSPARC Plots\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots.md): A detailed description of the common plots that CryoSPARC makes across multiple job types. - \[Tutorial: Negative Stain Data\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/negative-stain-data.md): Learn how to work with negative stain data in CryoSPARC. - \[Tutorial: Phase Plate Data\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/phase-plate-data.md): Learn how to work with Phase Plate data in CryoSPARC. - \[Tutorial: EER File Support\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-eer-file-support.md): Learn how to work with EER files in CryoSPARC. - \[Tutorial: EPU AFIS Beam Shift Import\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import.md): A tutorial covering how to split exposures into groups based on beam shift values, for data collected in EPU's AFIS mode. - \[Tutorial: Patch Motion and Patch CTF\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-patch-motion-and-patch-ctf.md): This tutorial covers Patch Motion Correction and Patch CTF Estimation. - \[Tutorial: Float16 Support\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-float16-support.md) - \[Tutorial: Particle Picking Calibration\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-particle-picking-calibration.md): In cryoSPARC v2.13+, there is a new feature that directly calibrates pick scores against defocus, making it much easier to set thresholds when using the Inspect Picks job. - \[Tutorial: Blob Picker Tuner\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-blob-picker-tuner.md): Optimize blob picks with the Blob Picker Tuner. - \[Tutorial: Helical Processing using EMPIAR-10031 (MAVS)\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-empiar-10031-mavs.md): Case study on using helical processing tools. - \[Tutorial: Maximum Box Sizes for Refinement\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/performance-metrics.md): Maximum box sizes for various amounts of GPU memory (VRAM). - \[Tutorial: CTF Refinement\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement.md): This tutorial details the background, implementation, and use of CTF refinement in CryoSPARC v2.12+. - \[Tutorial: Ewald Sphere Correction\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ewald-sphere-correction.md): How to work with Ewald Sphere Correction. - \[Tutorial: Symmetry Relaxation\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation.md): A tutorial detailing symmetry relaxation in CryoSPARC, a tool to help improve the refinement of pseudosymmetric particles. - \[Tutorial: Orientation Diagnostics\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-orientation-diagnostics.md): Using the new Orientation Diagnostics job to assess preferred orientation with titled and untilted HA Trimer data - \[Tutorial: BILD files\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-bild-files.md): An overview of BILD files in CryoSPARC - \[Tutorial: Mask Creation\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera.md): Mask Selection and Generation in UCSF ChimeraX - \[Tutorial: Dynamic Masking in Refinements (v5.0+)\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-dynamic-masking-in-refinements-v5.0.md) - \[Case Study: Yeast U4/U6.U5 tri-snRNP\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp.md): Processing EMPIAR-10073 with a focus on Local Refinement. - \[Tutorial: 3D Classification\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification.md): 3D Classification is a new way to perform discrete heterogeneity analysis in CryoSPARC in a manner that complements Heterogeneous Refinement. - \[Tutorial: 3D Variability Analysis (Part One)\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-one.md): Part One of the two-part tutorial on 3D Variability Analysis for exploring heterogeneity. - \[Tutorial: 3D Variability Analysis (Part Two)\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-variability-analysis-part-two.md): Part Two of the two-part tutorial on 3D Variability Analysis for exploring heterogeneity. - \[Tutorial: 3D Flexible Refinement\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement.md): An in-depth guide to 3D Flexible Refinement (3DFlex) in CryoSPARC. - \[Installing 3DFlex Dependencies (v4.1–v4.3)\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement/installing-3dflex-dependencies.md) - \[Tutorial: 3D Flex Mesh Preparation\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation.md) - \[Webinar Recordings\](https://guide.cryosparc.com/processing-data/webinar-recordings.md): CryoSPARC webinar recordings. - \[About CryoSPARC Live\](https://guide.cryosparc.com/live/about-cryosparc-live.md): From microscope to structure, in minutes. The fastest, streamlined path to high-resolution structures, during data collection or afterwards. - \[Prerequisites and Compute Resources Setup\](https://guide.cryosparc.com/live/prerequisites-and-compute-resources-setup.md) - \[How to Access CryoSPARC Live\](https://guide.cryosparc.com/live/how-to-access-cryosparc-live.md): CryoSPARC Live is automatically installed alongside CryoSPARC. - \[UI Overview\](https://guide.cryosparc.com/live/ui-overview.md): CryoSPARC Live UI Overview - \[New Live Session: Start to Finish Guide\](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide.md) - \[CryoSPARC Live Tutorial Videos\](https://guide.cryosparc.com/live/tutorial-videos.md) - \[Live Jobs and Session-Level Functions\](https://guide.cryosparc.com/live/live-jobs-and-session-level-functions.md): Overview of CryoSPARC Live architecture and job types specific to Live. - \[Performance Metrics\](https://guide.cryosparc.com/live/performance-metrics.md) - \[Managing a CryoSPARC Live Session from the CLI (≤v4.7)\](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli.md): This guide details how to automate the creation, configuration and overall management of a CryoSPARC Live session in Python via the CryoSPARC API. - \[Managing a CryoSPARC Live Session from the CLI (v5.0+)\](https://guide.cryosparc.com/live/managing-a-cryosparc-live-session-from-the-cli-v5.0.md) - \[FAQs and Troubleshooting\](https://guide.cryosparc.com/live/faqs-and-troubleshooting.md) - \[v3 User Interface Guide\](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide.md): The information in this section applies to CryoSPARC ≤v3.3. - \[Dashboard\](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/dashboard.md) - \[Project and Workspace Management\](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/project-and-workspace-management.md) - \[Create and Build Jobs\](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/create-and-build-jobs.md) - \[Queue Job, Inspect Job and Other Job Actions\](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/queue-job-inspect-job-and-other-job-actions.md) - \[View and Download Results\](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/view-and-download-results.md) - \[Job Relationships\](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/job-relationships.md) - \[Resource Manager\](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/resource-manager.md) - \[User Management\](https://guide.cryosparc.com/guides-for-v3/user-interface-and-usage-guide/user-management.md) - \[Tutorial: Job Builder\](https://guide.cryosparc.com/guides-for-v3/job-builder-tutorial.md): An in-depth explanation of CryoSPARC's Job Builder, inputs and outputs. - \[Get Started with CryoSPARC: Introductory Tutorial (v3)\](https://guide.cryosparc.com/guides-for-v3/cryo-em-data-processing-in-cryosparc-introductory-tutorial.md): In this tutorial, we will process a small dataset from movies to reconstructed density map. If you are new to data processing in CryoSPARC, we highly recommend following along! - \[Tutorial: Manually Curate Exposures (v3)\](https://guide.cryosparc.com/guides-for-v3/tutorial-manually-curate-exposures.md): Manually Curate Exposures is an interactive job in CryoSPARC that enables a user to visually inspect and curate a set of micrographs or movies. This tutorial refers to CryoSPARC v3.3 and earlier. - \[Guide: Data Management in CryoSPARC (≤v3.3)\](https://guide.cryosparc.com/guides-for-v3/tutorial-data-management-in-cryosparc.md): An overview of all data management utilities and common use cases. - \[Questions and Support\](https://guide.cryosparc.com/resources/questions-and-support.md): How to get help for CryoSPARC. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on a page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/readme.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-dynamic-masking-in-refinements-v5.0.md). # Tutorial: Dynamic Masking in Refinements (v5.0+) {% hint style="info" %} This guide page covers automatic masking in Homogeneous, Non-Uniform, and Heterogeneous refinement jobs created in CryoSPARC versions beginning with v5.0. Older versions of CryoSPARC use an automatic masking algorithm not described here. {% endhint %} ## Masking During Refinement 3D Refinements require two masks: \* a \*refinement mask\*, used during each refinement iteration to remove extraneous density from the current 3D volume which would otherwise degrade particle alignment, and \* a \*resolution mask\*, used to separate target molecule density from solvent during FSC calculation so that the resulting resolution estimate reflects the quality of the target density rather than the background CryoSPARC automatically generates resolution masks during refinement. It also automatically generates a refinement mask if one was not provided by the user. It is important that both masks be appropriately tight, i.e. not too tight, masking too close to target molecule density. It is also important that the masks be appropriately soft, i.e. transitioning smoothly from completely erasing density (mask has value 0.0) to completely retaining density (mask has value 1.0). Masks that are too tight or too hard can lead to refinements where spurious features develop in the density during refinement and/or FSC resolution is overestimated. Users should always inspect both the refinement mask and the resolution mask from a refinement job to confirm that they are not too tight (cutting into the density) or too hard (creating a sharp edge). The \[high resolution phase randomization diagnostic\](/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots.md#high-resolution-phase-randomization) can also be used to detect when a mask is too tight or hard. In CryoSPARC versions before v5.0, automatic masks were generated in refinement jobs based on user-adjustable looseness and softness parameters. These parameters were held fixed during refinement iterations. In some cases, especially for cryo-EM datasets reaching only low or intermediate resolutions (e.g. 4Å or coarser), the default parameters could produce masks that were too tight or too hard. \*\*CryoSPARC v5.0 introduces a new mask generation method\*\* (described below) that is robust across a wide range of resolutions and produces appropriate refinement results and resolution estimates without users having to tune masking parameters. The most significant improvement introduced by the new method is to scale the mask's tightness and softness based on the 3D map's current resolution estimate, rather than using a fixed value over iterations. ## Dynamic masking in CryoSPARC v5.0+ In this section we walk step-by-step through the process by which CryoSPARC produces masks. This process applies to both refinement and resolution masks. Here we create a refinement mask. Consider this 3D map at the end of a particular iteration of a refinement: !\[\](/files/i7NeT43Tn47AzmgKN4bJ) First, we binarize the map. The threshold used is the maximum value in the map times the \`Dynamic mask threshold\` parameter. This has a default value of 0.2 for the refinement mask, but can be changed by the user. The resolution mask uses a fixed value of 0.5. This produces a map with either a zero or one in every voxel. !\[\](/files/WlkDyn3hfWpX1ItaNiL5) Next, the map is dilated by a number of voxels equal to the current resolution times the \`Dynamic mask near multiplier\`. This has a default value of 2.0 for the refinement mask, but can be changed by the user. The resolution mask uses a fixed value of 2.0. !\[\](/files/tOs9DKEqAF3KUVimmYWg) The dilation step creates a mask that is appropriately loose, but it is still has a completely hard edge — voxels are either 1.0 or 0.0. We therefore add a soft edge, which falls gradually from a value of 1.0 at the distance determined by the \`Dynamic mask near multiplier\`, to a value of 0.0 at the distance determined by the \`Dynamic mask far multiplier\`. The width of this soft edge is thus equal to the current resolution multiplied by the \*difference\* between the far and near multipliers. The value of the \`Dynamic mask far multiplier\` is 5.0 by default in refinements, and is fixed to 5.0 for resolution masks. !\[\](/files/wxuXVz9Y37e4tgblc5I1) Now the mask is dilated and soft, as required for refinement masks. When we plot these masks, it’s best to see the underlying volume so that you can ensure that the relevant portions of the volume are contained within the mask. We could simply display the masked volume: !\[\](/files/QcnphJx9dPfnoAZajsAN) But this would hide any density outside the mask which you may wish to see. For example, faint density near the target might indicate a binding partner. If that density was too faint to be included in the initial binarized mask, it would not appear in the masked volume. To avoid this scenario, we instead plot the \*unmasked\* map, but with overlays indicating where the mask has a value of 1.0 (solid lines) and where the soft edge ends (that is, where the mask has a value of 0.0; dashed line). !\[\](/files/iKANR2rJcCNUBW5e3fz7) The region outside the dashed line is slightly shaded to focus the eye on the region inside the mask, but it is still visible to aid downstream analyses of regions outside the automatic mask. Note that both the dilation and padding width depend on resolution. When a map is poor, the mask is therefore very wide and soft. This reduces the possibility of the mask introducing spurious information into the map. As GSFSC resolution improves during refinement, the mask tightens automatically. !\[\](/files/Wvlr1Tz9YKnrNkPmuc6Y) !\[\](/files/Zie68nXIiMXC2Koxxuvc) !\[\](/files/6FSsRHuQLf7d6xsUDIPe) !\[\](/files/an5h4qVFhf6GWXlSwezS) Throughout the examples above, we have been plotting a single mask. CryoSPARC refinements plot both the refinement and resolution masks in the same image, and indicate which is which using the same name as the mask output. !\[\](/files/t84YiYWxQIwp3BVsW081) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-dynamic-masking-in-refinements-v5.0.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement/installing-3dflex-dependencies.md). # Installing 3DFlex Dependencies (v4.1–v4.3) {% hint style="info" %} All 3D Flex requirements are installed with CryoSPARC v4.4+. Skip this section unless you are running v4.1–v4.3. {% endhint %} In CryoSPARC v4.1–v4.3, 3DFlex jobs will not work without first installing the dependencies required to run the jobs. ## Prerequisites {% hint style="warning" %} Please ensure your system meets the following requirements: \* CryoSPARC v4.1 or newer \* Internet connection for downloading additional packages \* Nvidia driver version is 460.32.03 or newer on all GPU machines. Run \`nvidia-smi\` to verify \* No CUDA directories are in your \`PATH\` or \`LD\_LIBRARY\_PATH\` environment variables before running this command. To display the variables set in your environment, run \`export\`. Also ensure that the command \`which nvcc\` does \*\*not\*\* return a path to \`nvcc\`. {% endhint %} ## Installation of Dependencies Log onto each machine where \`cryosparc\_worker\` is installed, and run the command \`cryosparcw install-3dflex\` inside the \`cryosparc\_worker\` folder. For example: \`\`\`bash cd cryosparc\_worker ./bin/cryosparcw install-3dflex \`\`\` {% hint style="success" %} Aside from the Nvidia driver, dependencies for 3DFlex jobs are downloaded during the \`install-3dflex\` commands. There is no need to supply any external dependencies. {% endhint %} The \`install-3dflex\` command does the following: \* Download and install CUDA Toolkit \* Download and install PyTorch with CUDA Toolkit \* Reinstall PyCUDA with CUDA Toolkit \* Verify PyTorch can use CUDA (Requires an NVIDIA GPU) If you run this command on a machine without GPUs, you may see the message \`PyTorch not installed correctly, or NVIDIA GPU not detected.\` You may safely ignore this if there are no other error messages and the remaining verification tests pass. To further verify that your CryoSPARC instance is ready to run 3DFlex jobs, use the \[Installation Tests\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/guide-installation-testing-with-cryosparcm-test#testing-pytorch) to check that PyTorch is working on every worker node connected to your CryoSPARC instance. For example: \`\`\`bash cryosparcm test workers P1 --test gpu --test-pytorch \`\`\` To uninstall the 3DFlex dependencies and return the worker to its original state, run \`cryosparcw forcedeps\`. For example: \`\`\`bash cd cryosparc\_worker ./bin/cryosparcw forcedeps \`\`\` ## Update of Dependencies After updating CryoSPARC to a new full or point release, 3DFlex dependencies can be updated by this sequence of two \`cryosparcw\` commands: \`\`\`bash cd cryosparc\_worker ./bin/cryosparcw forcedeps ./bin/cryosparcw install-3dflex \`\`\` --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement/installing-3dflex-dependencies.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/motion-correction.md). # Motion Correction ## Overview During imaging in the microscope, a cryo-EM sample is irradiated by the electron beam for typically between one and ten seconds. During this time, the sample does not remain perfectly still. Drift of the stage, vibration of the microscope, and deformation of the sample ice all contribute to motion of the particles. These effects are all visible as motion blur in the final recorded image. To allow for correction of this effect, microscope images are captured as multi-frame movies (typically approximately fifty frames over the entire exposure). Because each \*frame\* only captures a short amount of time, the motion blur in each frame is significantly lower than if the entire micrograph was collected at once (i.e., if there was only one frame). Motion correction is the process by which those frames of a raw movie are aligned and averaged to produce a single-frame micrograph. This process significantly improves signal-to-noise ratio over collecting data in a single frame by reducing the cumulative effect of motion blur. In addition to causing motion, the beam interacts directly with the sample. The electron beam is a powerful source of radiation, capable of damaging the sample. High-resolution features are especially sensitive to radiation damage. Since the late frames have received the greatest radiation dose, they also tend to have the lowest quality information at high frequencies. A technique commonly called dose weighting (Grant and Grigorieff, 2015) accounts for the varying information content in each frame by attenuating the high-frequency signal from later frames of movies. All motion-correction jobs in CryoSPARC apply dose weighting. See \[the relevant section of this page\](#dose-weighting) for more information on the specific forms of dose-weighting applied by each job. CryoSPARC provides multiple motion correction methods and workflows. In almost all projects, \[Patch Motion Correction\](/processing-data/all-job-types-in-cryosparc/motion-correction/job-patch-motion-correction.md) should be used in the initial processing steps. It is also used internally by \[CryoSPARC Live\](/live/about-cryosparc-live.md) when performing real-time processing. If the final 3D reconstruction is of high quality, \[Reference Based Motion Correction\](/processing-data/all-job-types-in-cryosparc/motion-correction/job-reference-based-motion-correction-beta.md) may provide additional resolution improvement. ## What is motion correction? Motion correction is the process of algorithmically correcting for motion of the electron microscope stage and the sample ice itself to recover image quality lost by motion blurring.

In this movie of intact SARS-CoV-2 virions (EMPIAR 10492), the particles in total move only approximately 10 Å, but loss of contrast due to motion blurring is still clear. Motion correction recovers signal lost due to motion blurring, increasing contrast and improving the quality of the final results. Though not obvious in individual micrographs, the effect of motion correction is most important in improving high-resolution signal quality.

Cryo-EM data is collected in the form of movies, which are each a series of individual frames. Since a frame is usually between 0.1 and 0.2 seconds, the detector does not accumulate enough electron dose for clear identification of the target. However, the brief length of time significantly reduces the amount of in-frame motion blur.
Since the same physical objects create the image in each frame, we can find a shift to apply to each frame (or sub-region of a frame) that results in the greatest agreement among all frames. The ultimate goal of motion correction is to reduce the total blurring in the final particle images that are extracted — what differs between the different methods and implementations is the type of input data they require and the types of motion they are capable of capturing and correcting. ### What causes motion during data collection? There are two main forces which cause motion during data collection: stage drift and beam-induced ice deformation. The cause of the former is self explanatory: mechanical effects cause drift of the entire stage and grid during movie collection. This motion is observed as a \*shift of the entire image frame\*, and is the easiest type of motion to model. Each frame can simply be translated to create the best match between the previous and the next. This is called \*Rigid\* or \[\*Full Frame\* Motion Correction\](/processing-data/all-job-types-in-cryosparc/motion-correction/job-full-frame-motion-correction.md), because it produces a movement trajectory of the entire frame as a rigid object. Note that rotation of the grid is generally negligible and is not modeled. When the sample is irradiated by the electron beam, the thin layer of ice suspended in the grid hole buckles. This three-dimensional movement appears in the movie as anisotropic (i.e., different at different spatial positions) movement of the ice itself. The exact mechanism behind this effect is not fully understood, but it is suspected that the electron beam allows for relaxation of physical stress built up during sample vitrification (Thorne 2020). Modeling this type of motion is more challenging, since in theory small image regions in each frame might move in a different directions. CryoSPARC models anisotropic motion primarily using the \[Patch Motion Correction\](/processing-data/all-job-types-in-cryosparc/motion-correction/job-patch-motion-correction.md) job, and the theory and method underlying that job is described in the job page. ## What does each Motion Correction job do? As discussed above, the goal of motion correction is to align each frame such that the particles are in the same position throughout the movie. This way, when they are averaged, the high-resolution information is not lost due to motion blur. For example, consider the movement of particles move throughout this movie:
If this movement is not corrected, the particle images would be unacceptably blurry, destroying high-resolution information:
### Full Frame Motion Correction The main source of motion for a given particle is typically concerted motion of the entire stage, or frame. It is relatively straightforward to correct for this type of motion by minimizing the difference between one frame and its neighbors. In doing this, each frame is brought into register with the others. This corrects the rigid movement of the entire frame, while leaving the anisotropic movement unmodelled:
While this unmodelled movement still results in blurry particles, the results of averaging these aligned frames is already much clearer than if the movie had been collected as a single-frame micrograph:
As this is a relatively simple form of motioncorrection, it was one of the earliest forms introduced. Some examples of early implementations of Full Frame Motion Correction are MotionCor (Li et al. 2013) and Unblur (Grant and Grigorieff 2015). ### Local Motion Correction and Reference Based Motion Correction Correcting the anisotropic motion of individual particles is more challenging than the full-frame motion for two main reasons. First, and most importantly, single particle images each contain far less signal than an entire micrograph. Second, correcting for anisotropic motion requires knowing the position of particles to begin with. Once good particle location information is available, there are two types of jobs to correct the movement of individual particles in CryoSPARC. The first is \[Local Motion Correction\](/processing-data/all-job-types-in-cryosparc/motion-correction/job-local-motion-correction.md). In this job, a small patch around each particle is compared in each frame and aligned so as to reduce the total motion, much like the process in Full Frame Motion Correction. The second type is \[Reference Based Motion Correction\](/processing-data/all-job-types-in-cryosparc/motion-correction/job-reference-based-motion-correction-beta.md), which compares a projection of a high-quality 3D Volume to the particle location in each frame to find that particle’s exact location. In general, with a high quality reference volume, we expect Reference Based Motion Correction to perform better than Local Motion Correction. Reference Based Motion Correction also estimates empirical dose weights (see the Dose weighting section) and is based on Bayesian Polishing (Zivanov et al. 2019). However, the requirement of a high-quality volume is significant. Local Motion Correction does not require a reference, and so can be run early on in the processing pipeline if significant anisotropic motion is observed. Local Motion Correction is based on alignparts\\\_bfgs (Rubinstein et al. 2015). ### Patch Motion Correction \[Patch Motion Correction\](/processing-data/all-job-types-in-cryosparc/motion-correction/job-patch-motion-correction.md) is a fourth motion correction job available in CryoSPARC. It provides a means of correcting anisotropic motion without knowing particle positions. To do this, it models the movement of large patches of the micrograph, then describes that movement using a function called a \*spline\*. Importantly, this spline can be evaluated at any pixel location to find the trajectory of that pixel during the recording of the movie. When a particle image must be extracted from a position in a Patch Motion corrected micrograph, the spline function is evaluated at each pixel position to create an aligned average for that pixel. In this way, anisotropic motion is corrected without knowing the particle locations before hand. Therefore, this job typically gives better results than Full Frame Motion Correction, and is the job we recommend for motion correcting any new dataset in CryoSPARC. More information on this algorithm is available in the \[Patch Motion Correction job page\](/processing-data/all-job-types-in-cryosparc/motion-correction/job-patch-motion-correction.md). ## Dose weighting ### What is dose weighting?

A comparison of a map made from the first fifteen or last sixteen frames of a set of movies. The same particles are used in the same poses in both images — only the frames used during motioncorrection differ. The data are from EMPIAR 10424 (Nakane et al. 2020). The EER movies were fractionated into 31 frames with an upsampling factor of 2. The same 500 movies were used in each patch motion correction job. The motion correction jobs were run with default parameters, except F-crop set to 1/2. Published poses were used for homogeneous reconstruction of particles extracted from non-dose-weighted micrographs at a box size of 700 px, downsampled to 320 px.

During image collection, samples are bombarded with an intense electron beam. This beam damages the fragile bonds in the macromolecule, with the amount of damage increasing as electron dose accumulates. In 3D reconstructions, this is visible as a degradation of the high-resolution signal in later frames of the movie — the high resolution information has been burned away by the electron beam. To alleviate the worst effects of this radiation damage, the process of dose weighting involves down-weighting the contribution of late frames to the micrographs. In this way, it is possible to both \* retain the low-frequency signal from late frames, which is less damaged by radiation and useful for particle picking \* discard the high-frequency noise from late frames (since there is almost no useful information at these frequencies). We can plot the dose weights as a series of bar graphs in which the first frame is the topmost bar and the last frame is the lowest bar, and the weight of a frame at a particular resolution is given by the length of the bar.
When plotted this way, the trend aimed for by dose weighting becomes apparent. At low resolution (left panels), all frames are more-or-less equally reliable since the effect of the electron beam is much less noticeable at this resolution. Therefore all frames are treated equally in the final micrograph — they all have a weight of 1.0. However, radiation rapidly damages information at the highest frequencies (right panels). We therefore want to use only information from the early frames at the highest resolution, so early frames have a weight greater than 1.0 and late frames have a weight less than 1.0. Visualizing dose weights in this way can become unwieldy when considering all frequencies in a movie. We therefore typically present them as a heatmap instead, where the columns correspond to a resolution, the rows correspond to frames, and the color denotes the weight associated with that frame at that resolution.
### Dose weighting in CryoSPARC The heatmap above shows an example of the default dose weights applied to movies during motion correction. These default dose weights are calculated in the same way as described by Grant and Grigorieff. These dose weights are based on a model of exponential decay of the signal-to-noise ratio and are applied without any knowledge of the underlying sample or movies. Default dose weights work well in most cases, but if more information about the system is available a better estimate of radiation damage can be derived. More specifically, if the position of particles in each frame, high-quality pose estimates, and high-resolution reference volumes are available for each particle, it is possible to calculate the correlation between the reference volume and the particle image. From these correlations we can deduce the appropriate dose weights for each frame and resolution. These calculated dose weights are called \*\*empirical dose weights\*\* and can be calculated by comparing a 3D reference with the particles in the movie. This process is performed, for example, by Bayesian Polishing in RELION (Scheres 2014; Zivanov et al. 2019) and \[Reference Based Motion Correction\](/processing-data/all-job-types-in-cryosparc/motion-correction/job-reference-based-motion-correction-beta.md) in CryoSPARC. More information about the procedure is available in that job page. ## References 1. Thorne, R. Hypothesis for a mechanism of beam-induced motion in cryo-electron microscopy. \*IUCrJ\* vol. 7 416–421 (2020). 2. Li, X. \*et al.\* Electron counting and beam-induced motion correction enable near-atomic-resolution single-particle cryo-EM. \*Nature Methods\* \*\*10\*\*, 584–590 (2013). 3. Grant, T. & Grigorieff, N. Measuring the optimal exposure for single particle cryo-EM using a 2.6 Å reconstruction of rotavirus VP6. \*eLife\* \*\*4\*\*, e06980 (2015). 4. Zheng, S. Q. \*et al.\* MotionCor2: anisotropic correction of beam-induced motion for improved cryo-electron microscopy. \*Nature Methods\* \*\*14\*\*, 331–332 (2017). 5. Nakane, T. \*et al.\* Single-particle cryo-EM at atomic resolution. \*Nature\* \*\*587\*\*, 152–156 (2020). 6. Scheres, S. H. Beam-induced motion correction for sub-megadalton cryo-EM particles. \*eLife\* \*\*3\*\*, e03665 (2014). 7. Zivanov, J., Nakane, T. & Scheres, S. H. W. A Bayesian approach to beam-induced motion correction in cryo-EM single-particle analysis. \*IUCrJ\* vol. 6 5–17 (2019). 8. Rubinstein JL, Brubaker MA. Alignment of cryo-EM movies of individual particles by optimization of image translations. \*J Struct Biol\* (2015). ## Motion Correction Jobs {% content-ref url="/pages/-MR1pPxz8oV8vD7hR-oy" %} \[Job: Patch Motion Correction\](/processing-data/all-job-types-in-cryosparc/motion-correction/job-patch-motion-correction.md) {% endcontent-ref %} {% content-ref url="/pages/-MR1pK52kx\\\_tgOuk3GoV" %} \[Job: Full-Frame Motion Correction\](/processing-data/all-job-types-in-cryosparc/motion-correction/job-full-frame-motion-correction.md) {% endcontent-ref %} {% content-ref url="/pages/kNgvuDHqQUT39iBVJjSv" %} \[Job: Local Motion Correction\](/processing-data/all-job-types-in-cryosparc/motion-correction/job-local-motion-correction.md) {% endcontent-ref %} {% content-ref url="/pages/9yzqa8SeDKMwjIEMn7Vt" %} \[Broken mention\](broken://pages/9yzqa8SeDKMwjIEMn7Vt) {% endcontent-ref %} {% content-ref url="/pages/dtTZBOpS2X7VWo87kp5D" %} \[Job: MotionCor2 (Wrapper) (BETA)\](/processing-data/all-job-types-in-cryosparc/motion-correction/job-motioncor2-wrapper-beta.md) {% endcontent-ref %} {% content-ref url="/pages/OdXx0PGmfMOyzBAgHbJN" %} \[Job: Reference Based Motion Correction (BETA)\](/processing-data/all-job-types-in-cryosparc/motion-correction/job-reference-based-motion-correction-beta.md) {% endcontent-ref %} ## Motion Correction Tutorials {% content-ref url="/pages/-MNeA4BFZRumxWAGw8dk" %} \[Tutorial: Patch Motion and Patch CTF\](/processing-data/tutorials-and-case-studies/tutorial-patch-motion-and-patch-ctf.md) {% endcontent-ref %} --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/motion-correction.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-estimation.md). # CTF Estimation \[\*Jump to the CTF Job Types\*\](#ctf-estimation-jobs) ## At a Glance The Contrast Transfer Function (CTF) models the effect of defocus and microscope aberrations on single particle images. These effects must be corrected before the images can be used to reconstruct a 3D Volume. This article contains a broad overview of what the contrast transfer function is and where it comes from. Advanced pages provide more detail on waves, contrast, and aliasing. These topics are important both for a deeper theoretical understanding of the technique and also for practical considerations, such as the choice of box size during extraction. Finally, there is a list of useful external resources for interested readers at the end of this page. {% hint style="info" %} Developing even an incomplete understanding of the sources of contrast in cryo-EM is a significant undertaking — we intend this section to serve as a reference over the course of one’s cryo-EM education rather than a prerequisite to performing one’s first CTF job. {% endhint %} ## The Contrast Transfer Function ### The CTF and the PSF When modeling image aberrations, it may be most intuitive to consider the question, “what does the image of a single point look like when imaged by this system”. This concept is known as the Point Spread Function (often abbreviated PSF). In theory, one might expect that an image of a single point would be a simple projection of that point into a 2D image:
On the left, the "object" is a blue dot. On the right, the "image" is a red dot. An arrow labeled "Point Spread Function" points between the two.
However, various imperfections and complications in electron microscopes means this is not the case. The deviations of the true image from this “ideal” projected point are collectively called \*aberrations\*. The largest aberration is due to the fact that images are collected out of focus (also known as “with defocus”) to improve contrast (see \[Contrast in Cryo-EM\](/cryo-em-foundations/image-formation/contrast-in-cryo-em.md)). The sum total of these aberrations result, generally for cryo-EM, in a point spread function that consists of oscillating bands of dark and light rings around a single point:
The red dot at the right now has oscillating rings of red and white around it.
To model what would happen for an arbitrary object shape, we could break apart the object into a set of many individual points, and apply the PSF at each point to determine the resulting image.
At the top, a continuous object on the left has an image with rings around it on the right. At the bottom, the object is split into points, each of which has rings around them in the image.
If the object were split into infinitely many points, applying the point spread function to each point would produce an identical image. This operation is called a convolution, and can be computationally expensive. However, a mathematically useful property of convolution is that it is equivalent to a simple multiplication when working in Fourier space. Thus, to simplify convolution, we can: 1. take the Fourier transform of our object, 2. multiply it by the Fourier transform of the point spread function, 3. and then perform an inverse Fourier transform on the result. This process produces the image we would expect to see in the microscope.
This image is a flowchart. At the left, we see the structura logo (labeled "object") and the PSF. Both are Fourier transformed. These Fourier transforms are multiplied together, then the inverse Fourier transform is taken. This results in the final image: the Structura logo with rings around each part.
The Fourier transform of the point spread function is so useful and commonly used that it has its own name: the Contrast Transfer Function (CTF). In addition to its usefulness in convolving the object and the point spread function, the contrast transfer function is also much easier to estimate directly from image data than the point spread function. ## The CTF in Practice The CTF has several practical implications on the processing of cryo-EM data. #### Particle visibility
A grid of images. The left images have zero defocus, the right images have -2 micron defocus. The top row shows a plot of the CTF at these defoci. The middle images show simulated particles with these defoci. The bottom images show simulated particles plus noise. The images with greater defocus are much easier to see, especially with the added noise.
Images collected at or near focus will have very little contrast at low resolutions because protein and buffer scatter electrons with approximately the same intensity (see \[Contrast in Cryo-EM\](/cryo-em-foundations/image-formation/contrast-in-cryo-em.md) for more details on this topic). This makes them difficult to pick against a noisy background. For example, consider the simulated images above. The image collected with no defocus is faintly visible without noise, but becomes almost impossible to see with even modest amounts of noise. Compare this to the image simulated with significant (2 µm) defocus. This level of defocus introduces contrast at low frequencies, making the particle clearly visible even when noise is present. However, the image is more obviously corrupted by oscillations in contrast at high frequencies (visible in the simulated image as black and white rings). Most, but not all, of the image corruption can be modeled and recovered computationally. Thus, micrographs are typically collected with the least defocus for which particle images can still be reliably picked against the noisy background. #### Zero crossings
A plot of the CTF is shown. Three frequencies are highlighted with points. Below the plot, examples of a wave with this frequency and the result of applying the CTF to this wave are shown.
As the contrast transfer function oscillates between -1 and 1, it crosses 0 several times. Frequencies for which the contrast transfer function is 0 have no contrast — they are absolutely invisible in the image. If all images were collected at the same defocus, all of the images would have 0 contrast at the same spatial frequencies. This would result in a specific band of frequencies having little to no contrast, making the map useless. Data is therefore collected over a range of defocus values such that, taken together, each frequency is represented in a sufficient number of particle images to be properly accounted for. #### Signal delocalization
The same particle image is simulated with 0 and -1.5 micron defocus. A zoomed region of the particle shows that the defocused particle has signal in a region that is empty in the no-defocus image.
Finally, collecting images with defocus delocalizes signal away from its true position. In the simulated comparison above, note that the high-frequency features (loops in the fabs, helical pitch, etc.) are clearly visible in the left images, where no CTF is applied. Note, of course, that images like this are impossible to collect, since with no defocus, they would not have any contrast. On the right, these same high-frequency features have spread away from their true positions and now overlap, making the image impossible to directly interpret. This effect is more significant at higher frequencies and at higher defocus values. Particle images which were collected at high defocus and which refine to high resolutions must therefore be extracted with a larger box size to capture information moved away from the particle center. ### Sample Flatness Cryo-EM samples are never perfectly "flat". Particles tend to concentrate near the air-water interfaces prior to the sample being frozen, and the ice surface itself is often nonplanar. Recalling that defocus affects the contrast transfer function, this means that a single image can contain particles with different defoci and therefore different contrast transfer functions. CryoSPARC provides a patch-based contrast transfer function estimation method in the \[Patch CTF Estimation\](/processing-data/all-job-types-in-cryosparc/ctf-estimation/job-patch-ctf-estimation.md) job type which examines many different areas in the micrograph to compute a "defocus landscape", to combat this issue. Patch CTF requires no prior information about particle locations within the micrograph, and can be used immediately after motion correction. It can even work on tilted samples without knowing about the tilt beforehand. ## The effects of the CTF illustrated It can be difficult to understand the major practical effects of the contrast transfer function in abstract. We therefore present a useful test object here, inspired by examples first presented in Downing and Glaeser (2008). In the following figures, we mathematically apply the contrast transfer function to a test object. This test object is a wedge with horizontal stripes. The wedge gets narrower, and the stripes closer together, as we move from left to right. In this way, the wedge comprises a smooth range of (vertical) spatial frequencies, starting with the low frequencies at the left and finishing with the high frequencies at the right. In these simulations, each pixel represents 1 Å. The wedge is therefore approximately 20 nm tall at its tallest side, and contains frequencies corresponding to the resolutions from approximately 26 Å to 6 Å. Additionally, we apply the contrast transfer function only in the vertical direction to prevent information from one spatial frequency spilling into adjacent columns.
Top: a wedge with alternating black and white stripes. No CTF is applied to this wedge. On the left hand side, the stripes are wide, so their spatial frequency is low. On the right, the stripes are small, so the spatial frequency is high. Middle: the same wedge with a -1.5 micron defocus. The region in the middle of the wedge is flat grey, since there is no contrast at this frequency. Signal is delocalized away from its position in the wedge with no CTF. Bottom: a graph of the CTF with zero crossings marked.
At this defocus, the CTF has a negative value for low frequencies. This means that low frequencies (e.g., the left side of the wedge) have negative contrast. Approximately halfway along the wedge, the CTF crosses zero (dashed vertical lines indicate zero-crossings). The frequencies at and near this point have no contrast — they are the same flat grey as the background. The zero-crossings of the CTF are why it is important to collect data at a range of defocus values. If all of these zero crossings were at the same point, that frequency would never have any contrast, and the images would therefore be incapable of producing a 3D volume. After the first zero crossing, the CTF has a positive value. This means the wedge is again visible against the grey background, but black has become white and vice versa. This is another effect of the CTF that must be corrected for. In reality, the stripes have the same “density” all the way along the wedge. This flipping is purely an effect of the CTF. The image above shows the CTF at only a single defocus (-1.5 µm). In the below animation, the defocus varies smoothly from 0.0 to 3.0 µm. The top pane shows the frequency wedge image while the bottom pane shows the CTF. Both are at the same defocus, and the frequencies are aligned in each.
An animation of the wedge from the previous image. Defocus increases from 0 to -3 microns.
Note first that as defocus increases, so does the contrast at the low frequencies (left end of the wedge). At the beginning of the animation, when the defocus is 0, the left side of the wedge is nearly invisible. As defocus increases, so does the contrast of the low-frequency, left-hand side of the wedge. Next, observe that the zero crossings move up and down along the wedge as the defocus changes, since the contrast transfer function crosses zero at different frequencies depending on the defocus. Zero crossings are indicated in the bottom pane with dashed lines. Finally, watch the high-frequency (right-hand) tail of the wedge as defocus changes. The information from this high-frequency region is displaced by a significant fraction of the total size of the object at high defocus! ## Useful Resources For an approachable discussion of the contrast transfer function, we recommend Grant Jensen’s lecture on the topic, available \[on YouTube\](https://youtu.be/mPynoF2j6zc). Readers interested in the math behind phase contrast and the contrast transfer function may find the \[notes from Fred Sigworth and Hemant Tagare\](https://cryoemprinciples.yale.edu/chapters) interesting. \[Marin van Heel’s notes\](https://www.singleparticles.org/methodology/MvH\_Phase\_Contrast.pdf) also provide interesting discussion of the topic. Finally, \[Transmission Electron Microscopy by Kohl and Reimer\](https://link.springer.com/book/10.1007/978-0-387-40093-8) provides a thorough and detailed reference for motivated readers. To check whether CTF aliasing occurs for a given set of experimental parameters in 2D, consider using \[this online script\](https://3dem.github.io/relion/ctf.html), written by Takanori Nakane. \[This tool\](https://ctfsimulation.streamlit.app/) from the Jiang lab simulates a CTF from a variety of user-selected parameters. It can be a helpful way to build intuition about the effects of various microscope parameters on the final CTF. Another useful tool for developing intuition about the CTF’s effect on images is available from \[Johannes Elferich\](https://jojoelfe.github.io/webgl-ctf/image\_abb). ## References 1. Downing, K. H. & Glaeser, R. M. Restoration of weak phase-contrast images recorded with a high degree of defocus: The “twin image” problem associated with CTF correction. \*Ultramicroscopy\* \*\*108\*\*, 921–928 (2008). ## CTF Estimation Jobs {% content-ref url="/pages/-MR1qOsE3pv\\\_NChlhkcR" %} \[Job: Patch CTF Estimation\](/processing-data/all-job-types-in-cryosparc/ctf-estimation/job-patch-ctf-estimation.md) {% endcontent-ref %} {% content-ref url="/pages/-MSTwb\\\_Lb0qdUOYUKrjD" %} \[Job: Patch CTF Extraction\](/processing-data/all-job-types-in-cryosparc/ctf-estimation/job-patch-ctf-extraction.md) {% endcontent-ref %} {% content-ref url="/pages/ImRIxxhDrKW4PJmyJ3M0" %} \[Job: CTFFIND4 (Wrapper)\](/processing-data/all-job-types-in-cryosparc/ctf-estimation/job-ctffind4-wrapper.md) {% endcontent-ref %} {% content-ref url="/pages/T9YbsSqKDdR42S8P0v9x" %} \[Job: Gctf (Wrapper) (Legacy)\](/processing-data/all-job-types-in-cryosparc/ctf-estimation/job-gctf-wrapper-legacy.md) {% endcontent-ref %} ## CTF Estimation Tutorials {% content-ref url="/pages/-MNeA4BFZRumxWAGw8dk" %} \[Tutorial: Patch Motion and Patch CTF\](/processing-data/tutorials-and-case-studies/tutorial-patch-motion-and-patch-ctf.md) {% endcontent-ref %} --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-estimation.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/exposure-curation/job-micrograph-denoiser-beta.md). # Job: Micrograph Denoiser (BETA) ## At a Glance

A micrograph from EMPIAR-10424 (Nakane et al. 2020)

Produce enhanced micrograph images to aid particle picking and visual inspection. ## Description Micrograph Denoiser takes micrographs as input and produces denoised versions of those micrographs. These denoised versions can be used downstream to help pick particles and to aid in visual inspection of the micrographs. Note that particles are extracted from the raw micrographs and not the denoised micrographs even if the latter are used for particle picking. The denoiser works by first learning repeated patterns in the data during the training step. Then, to perform the denoising, it considers each region of the input micrograph. The denoiser is trained to match patterns it has learned and it amplifies those patterns in the denoised output micrograph. Features and noise which do not match are dimmed, thereby enhancing the visual quality of the denoised output micrograph. For a more thorough explanation of how the Micrograph Denoiser works, see the\[ Denoiser Training section\](#denoiser-training). Denoised micrographs have significantly higher contrast than, for example, a lowpass filtered micrograph because image content is \*added\* by the denoiser. Of course, this contrast is not truly present in any single given region of the data — it is learned in aggregate by the denoiser. As such, the denoised images can be quite helpful for visual inspection or particle picking, but they do not contain increased signal for 2D or 3D reconstruction and therefore particles are not extracted from these denoised micrographs for downstream processing. This job comprises both the training and application of the denoiser model. Input micrographs with training data are used to train the denoiser model, then \*all\* connected micrographs are denoised to produce the output. ## Inputs ### Exposures Micrographs to be denoised, with background subtracted and CTF estimates available. The pixel size of these micrographs must be smaller than 3 Å. {% hint style="info" %} Currently, only Patch Motion Correction performs background subtraction, so the input micrographs must come from movies which were motion corrected by Patch Motion Correction. {% endhint %} If a new Denoising model will be trained (which is the recommended workflow), the input micrographs must also have training data. This data is only generated by Patch Motion Jobs run with CryoSPARC version \*\*4.5 or later\*\*. If you wish to perform denoising on data motion corrected in prior versions of CryoSPARC, see \[Denoising Data from Existing Patch Motion Jobs\](#denoising-data-from-existing-patch-motion-jobs). Thus, a typical preprocessing workflow for CryoSPARC v4.5 or later might be 1. Import Movies 2. Patch Motion Correction 3. Patch CTF 4. Micrograph Denoiser 5. Blob Picker, etc… ### Denoise Model If a Micrograph Denoiser has previously been run on this data, you may connect the Denoiser model output of the previous job to use that model instead of training a new one. ## Commonly Adjusted Parameters ### Renormalize input greyscale In some situations, the denoiser must (or should) re-estimate the range of pixel values that likely correspond to particles (as opposed to empty ice or contaminants like crystalline ice or carbon). This estimation is called \*greyscale normalization\*, and \`Renormalize input greyscale\` controls whether or not this occurs. For more information on greyscale normalization, see \[that section of this guide page\](#greyscale). When using the pre-trained model or when training a new model, this parameter is not displayed. This is because, in both of these cases, the model was not (or has not yet been) trained on the data and so must have a new greyscale normalization estimate. When using a model trained by a previous Micrograph Denoiser job, this parameter becomes visible. If the input model was trained on the same data it will be denoising, it is typically not necessary to renormalize the greyscale and so this parameter can be kept off. If, however, it was trained on different data (even a different dataset of the same particle), it is likely worth renormalizing the greyscale and this parameter should be turned on. ### Greyscale normalization factor Although the greyscale normalization procedure typically finds the correct range for the normalized greyscale, it is not perfect. This parameter (1.0 by default) is a multiplicative scale applied after the greyscale normalization process. For example, if the automated normalization determines that training should take place using a greyscale ranging from 0 to 200, but the \`Greyscale normalization factor\` is set to \`1.5\`, the final greyscale used during training and denoising will be 0 to 300. Typically, this parameter can be left at the default value of 1.0. One notable exception is HexAuFoil grids, which often require a lower value for this parameter due to the significant fraction of the micrograph occupied by gold. \[The first plot produced by Micrograph Denoiser\](#diagnostic-plots) is an example micrograph with the greyscale normalization applied. If this plot appears flat and grey or completely blown out, this parameter may need to be adjusted to a lower or higher value, respectively. ### Use pretrained model If no input model is connected, you can either train a new model based on the input data or use the pre-trained model that is packaged with CryoSPARC. Generally, we recommend that this setting is left off, since a model trained directly from the data is expected to perform better and is relatively fast.

The same micrograph (from EMPIAR 10335; Han et al. 2019) denoised using the pretrained model (left) or a model trained on this dataset (right). Training took eight minutes to complete 200 epochs with 100 training micrographs.

### Number of mics for training How many micrographs are used during denoiser training. Micrographs are selected randomly from the input for training, up to this number. If fewer than this number have training data available, the job will produce a warning but proceed with the training. At least 10 micrographs are required to train the denoiser; if fewer than 10 micrographs have training data, the job will fail. We have generally found that 100 micrographs are sufficient to train a high-quality denoiser. ### Train from scratch If this parameter is true (default), a new denoiser will be trained starting from a random initialization. This generally produces better results. If this parameter is false, the training will instead start with an initialization using the pre-trained denoiser as the starting point. ### Num training epochs This parameter controls the number of times the denoiser is trained on the selected training subset. If the results of a denoiser job are still noisy or difficult to interpret, re-running the job with a greater number of epochs can produce better results at the cost of increased training time. ### Crop micrograph edges (fraction) In some datasets, the edges of micrographs may have artifacts due to aberrations in the microscope or significant full-frame drift. These high-contrast artifacts can degrade denoiser training performance. In these cases, it is beneficial to ignore the edges of the micrograph during training. This parameter sets a fraction of the micrograph to ignore \*on each edge\*. For example, setting \`Crop micrograph edges\` to 0.01 will crop 1 pixel off each side of a 100 x 100 pixel image, resulting in a 98 x 98 pixel training image. For rectangular images, the largest dimension is used to calculate the number of pixels: the same factor of 0.01 would trim a 50 x 100 pixel image to 48 x 98 pixels during training. ## Outputs ### Denoised micrographs Denoised micrographs are output in this slot. Note that the original, non-denoised micrographs are also included in this same output. If this output is ultimately connected to an Extract Particles job, the non-denoised micrographs will be used automatically. ### Diagnostic plots #### Normalization The first plot produced by the Micrograph Denoiser is an example micrograph with normalization applied. See the \[greyscale section of this page\](#greyscale) for more information on how and why input micrographs are normalized before training and denoising. If this micrograph has little to no contrast, the normalization factor must be reduced. If this micrograph appears blown-out, with most pixels either white or black, the normalization factor must be increased.
#### Training and Validation Micrograph Denoiser also produces a plot of training and validation curves. However, unlike some machine learning tools, the weight of the various components of the Denoiser model change over the course of the training. We generally do not expect these plots to be informative, and users should focus on the denoised results to assess training quality.
## Common Problems ### Denoised micrographs are blurry or still noisy
If the denoiser model has not yet converged, it will not be able to accurately model noise in the micrographs. This will result in blurry or still-noisy results. Increasing \`Num training epochs\` often helps in these cases. ### Denoiser produces images of empty ice or large, blotchy images Flat, grey images or blotchy, high-contrast images, like those shown in the second row of the Diagnostic Plots above, are typically due to a Greyscale normalization factor that is too high or low, respectively. Changing this parameter should improve results. ## Common Next Steps Denoised micrographs are especially helpful when performing and evaluating particle picking. Thus, a typical next step would be \[Blob Picking\](/processing-data/all-job-types-in-cryosparc/particle-picking/job-blob-picker.md) or \[Template Picking\](/processing-data/all-job-types-in-cryosparc/particle-picking/job-template-picker.md), followed by \[Inspect Picks\](/processing-data/all-job-types-in-cryosparc/particle-picking/interactive-job-inspect-particle-picks.md). The Inspect Picks job allows users to toggle between viewing the raw and denoised micrographs to evaluate pick locations. In CryoSPARC v4.6+, Inspect Picks is able to automatically cluster and select particles when picking is done on denoised micrographs (see \[Interactive Jobs\](/application-guide/interactive-jobs.md#interactive-job-inspect-particle-picks)). After particles are picked, micrographs can be plugged directly into \[Extract Micrographs\](/processing-data/all-job-types-in-cryosparc/extraction/job-extract-from-micrographs.md) — the raw micrographs will automatically be used during extraction even if denoised micrograph images are available or were used for picking. {% hint style="warning" %} At this time, TOPAZ does not perform well on micrographs denoised with the Micrograph Denoiser. If particle picking will be performed with TOPAZ, we recommend using \[TOPAZ Denoise\](/processing-data/all-job-types-in-cryosparc/deep-picking/topaz/job-topaz-denoise-beta.md) instead. {% endhint %} ## Denoising Data from Existing Patch Motion Jobs Patch Motion Correction jobs run in versions of CryoSPARC prior to 4.5 do not generate the data necessary to train a denoiser model. To denoise these micrographs, two workflows are available: 1. \*(recommended)\* Performing Patch Motion Correction on a small subset of the movies to generate the necessary data, or 2. Using the pre-trained model ### Generate new training data (recommended) In our testing, a denoiser trained on the input data typically outperforms the pre-trained model. We therefore recommend that the necessary training data is generated for a subset of micrographs and used to create the denoising model. This model can then denoise the entire set of micrographs, including those for which were not re-motion corrected. 1. Create a Patch Motion Correction job with the same settings as the existing job, except set \`Only process this many movies\` and \`Num. movies for denoiser training data\` both to \`100\`. Plug the micrographs from the initial Patch CTF Estimation job into the Exposures input. 2. Run the resulting micrographs through a Micrograph Denoiser job. The CTF estimates from the initial job will be used along with the training data from the new Patch Motion Correction job. 3. Set up a new Micrograph Denoiser job with 1. the original, full set of motion-corrected and CTF-estimated movies, and 2. the denoise model from the first Micrograph Denoiser job This takes advantage of the benefits of training the denoiser on the data, while avoiding the need to re-motion correct the entire dataset. ### Use the pretrained method If time is at a premium, plugging the existing movies into the Micrograph Denoiser and turning \`Use pretrained model\` on skips model training and uses the pretrained model. The results with the pretrained model are typically not as good as those trained on the data, but will likely still represent an improvement over simple lowpass filtering. ## Denoiser Training The Micrograph Denoiser in CryoSPARC uses a specialized neural network architecture to produce denoised micrographs from training data. The neural network is trained using a \*Nosie2Noise\* methodology (Lehtinen et al. 2018) to predict what parts of an image are noise and what parts are signal. The basic principle behind Noise2Noise training is very similar to that of GSFSC validation. First, training data is generated by splitting each movie into odd and even frames, then creating half-micrographs from only those frames. Any signal in the movie should be the same in both half-micrographs, but the noise is treated as entirely random and independent.
Next, the neural network is trained to predict half-micrograph B from half-micrograph A. The only information that is the same between the two micrographs is the signal. Thus, as this neural network improves its ability to predict half B from half A, it is in effect learning patterns present only in the signal — modeling noise would not improve its ability to predict half B.
This setup has been explored in several denoiser methods for cryo-EM data, including Warp (Tegunov and Cramer 2018), TOPAZ denoise (Bepler et al. 2020), and Sphire (Wagner et al. 2020). In addition to the \*Noise2Noise\* training setup, CryoSPARC’s Micrograph Denoiser pre-corrects for the CTF in training data and input data, causing the denoised micrographs to be as visually consistent as possible across a range of defocus values. Furthermore, traditional denoising metrics are used to augment the training objective function to encourage the denoiser to quickly learn to produce visually clear micrographs emphasizing repeated signal such as particles. ## Greyscale Each pixel in a cryoEM micrograph contains a numeric value representing the electron dose received at that pixel. It can have, essentially, any value. Typically, these numbers are represented by making the highest value pure black and the lowest value pure white, with intermediate values linearly scaled to an intermediate grey. This mapping from values to darkness is called the “greyscale” of the image. Each dataset will have a slightly different greyscale, depending on the electron dose, pixel size, aperture settings, etc. Particles typically fall within a relatively narrow band of values across a dataset. What’s more, they are typically much closer in value to empty ice than to very dark objects like carbon or crystalline ice. Thus, if we trained the denoiser on the raw greyscale it may not even be able to detect true particles, since they would have essentially the same value as empty ice.
To avoid this flattening effect, the Micrograph Denoiser first estimates the greyscale band most likely to contain particles and produces a \*normalized greyscale\* covering only that range. Any values outside this range are clipped to black or white. This dedicates the greatest dynamic range to the values most likely to contain particles.
Note in the figure above that, after normalization, the values corresponding to empty ice are all white, and any values outside the expected range for a particle are all black, regardless of their true value. All the variance of the greyscale is focused on the region in which particles are expected to lie. Normalizing the greyscale in this way focuses the denoiser on detecting patterns from the particles rather than empty ice or contaminants. \`Greyscale normalization factor\` adjusts the estimated greyscale by multiplying the limits, moving them further from the mean. For example if, in the initial greyscale, any values below -10 are pure white and any values above 10 are pure black, then setting the \`normalization factor\` to \`1.5\` would result in a final greyscale (used for training) in which -15 and below is white and 15 and above is black.
This means the model would be trained with a wider range of values; this may improve or degrade performance, depending on whether or not those values are useful for learning about the particles in the dataset. ## References 1. Nakane, T. \*et al.\* Single-particle cryo-EM at atomic resolution. \*Nature\* \*\*587\*\*, 152–156 (2020). 2. Han, Y. \*et al.\* High-yield monolayer graphene grids for near-atomic resolution cryoelectron microscopy. \*Proceedings of the National Academy of Sciences\* \*\*117\*\*, 1009–1014 (2019). 3. Lehtinen, J. \*et al.\* Noise2Noise: Learning Image Restoration without Clean Data. \*arXiv\* (2018) doi:\[10.48550/arXiv.1803.04189\](https://doi.org/10.48550/arXiv.1803.04189). 4. Tegunov, D. & Cramer, P. Real-time cryo-EM data pre-processing with Warp. \*bioRxiv\* (2018) doi:\[10.1038/s41592-019-0580-y\](https://doi.org/10.1038/s41592-019-0580-y). 5. Bepler, T., Kelley, K., Noble, A. J. & Berger, B. Topaz-Denoise: general deep denoising models for cryoEM and cryoET. \*Nature Communications\* \*\*11\*\*, 5208 (2020). 6. Wagner, T. & Raunser, S. The evolution of SPHIRE-crYOLO particle picking and its application in automated cryo-EM processing workflows. \*Communications Biology\* \*\*3\*\*, 61 (2020). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/exposure-curation/job-micrograph-denoiser-beta.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation.md). # Tutorial: Symmetry Relaxation ## Introduction {% hint style="info" %} Symmetry relaxation is an option available in Homogeneous and Non-Uniform refinement jobs that can help resolve pseudosymmetry and symmetry-mismatched complexes. This tutorial describes the specific case that symmetry relaxation attempts to address, and walks through how it can be applied to a dataset. {% endhint %} ### Alignment and Symmetry Mismatches All refinement jobs in CryoSPARC require estimating the orientation from which each particle is viewed, relative to the density reference. This “alignment” step enables iterative refinement, and allows for successive reconstruction of an updated density map. This step is fairly costly though, as it requires solving a 5-dimensional search problem (3 pose dimensions and 2 shift dimensions) \*independently for every particle\*, via matching projections of the rotated and shifted density to particle images. In all refinements with global pose search, CryoSPARC uses a technique known as \*Branch and Bound\* (BnB) to accelerate the alignment step. This was described in our 2017 Nature Methods publication, \[cryoSPARC: algorithms for rapid unsupervised cryo-EM structure determination\](https://www.nature.com/articles/nmeth.4169) (Punjani et al., 2017). Since poses and shifts range over a continuum of possible values, the search space of poses and shifts must be discretized, so that a projection-matching based likelihood calculation can be performed at all poses in the discrete space. Discretization of poses usually does not cause issues for asymmetric structures, for which each particle will have only one correct pose. Discretization isn’t problematic with perfectly symmetric structures either: each particle has a set of multiple correct poses, the number of which is equal to the symmetry order of the imposed symmetry group. Each of these correct poses is related to each other via symmetry transform. In this case, the alignment algorithm can find any of the optimal poses, and the entire set of symmetry-related poses will be used to reconstruct the molecule. ![](https://guide.cryosparc.com/files/KAIef5qcxuueukGxQlOY) An illustration of the pitfalls of enforcing symmetry on pseudosymmetric molecules. Where this strategy encounters difficulties is with nearly symmetric molecules. There are many terms used to refer different forms of approximate symmetries, including pseudosymmetry, symmetry-mismatches, or symmetry-breaking features. An excellent characterization of different forms of these symmetries is presented in \[Huiskonen, 2018\](https://portlandpress.com/bioscirep/article/38/2/BSR20170203/57149/Image-processing-for-cryogenic-transmission). Pseudosymmetric particles may have several plausible poses at low resolution, and the correct pose may only be distinguishable if each of the symmetry-related poses are compared directly. The discretization of the pose search space used by CryoSPARC provides no guarantee that the symmetry-related poses are compared directly, unless symmetry relaxation is activated (as described in the subsequent section). {% hint style="info" %} Note that there is no single processing strategy that will best resolve all types of symmetry-mismatched complexes, and optimal reconstruction of symmetry-mismatched complexes is an active area of research that is usually heavily informed by the specific sample being solved. For a thorough catalogue of different processing strategies used for resolving symmetry-mismatched molecules, refer to, \[Abrishami et al., 2021\](https://www.sciencedirect.com/science/article/pii/S0079610720300390?via%3Dihub) published in Progress in Biophysics and Molecular Biology. {% endhint %} With symmetry-mismatched structures, keen attention to map details and quality is required. Simple resolution indicators like FSC are usually not the best indicator of optimal final results, and manual map inspection is strongly recommended. ### Symmetry Relaxation In effort to make CryoSPARC more robust to symmetry-mismatched complexes, we have updated the Homogeneous and Non-Uniform Refinement job types to include a tool known as s\*ymmetry relaxation\*. Symmetry relaxation is a modification to the orientation search procedure, which forces CryoSPARC to be more thorough during orientation search, to avoid placing particles into incorrect poses that are related to the true pose by a symmetry transform. It is recommended to enable symmetry relaxation when dealing with pseudosymmetric or symmetry-breaking molecules. When symmetry relaxation is enabled, regular BnB alignment proceeds as normal for each particle, and the BnB-optimal pose is stored. After the BnB-optimal pose is found, symmetry relaxation imposes an extra step in which the alignment objective function is evaluated at \*all poses that are symmetry-related to the current optimal pose\*. The symmetry-related pose angles are computed analytically, and the objective function is evaluated at the current FSC resolution of the density. If any of the symmetry-related poses are found to have a better objective value, these new poses will be used to reconstruct the next iteration’s density map. In CryoSPARC v4.4’s Homogeneous and Non-Uniform Refinement jobs, symmetry relaxation is made available via the \`Symmetry relaxation method\` parameter. There are three options available: \* \`none\`: This disables symmetry relaxation. The input symmetry group will be enforced as usual, and each particle will be used during reconstruction N times, where N is the order of the symmetry group \* \`maximization\`: This option enables symmetry relaxation. Once the BnB-optimal pose is found, the alignment objective will be evaluated over each of the N symmetry-related poses. The single best pose will carry forward to the reconstruction (”backprojection”) step. \* \`marginalization\`: This option enables symmetry relaxation via marginalization. Once the BnB-optimal pose is found, the alignment objective will be evaluated over a small search radius covering each of the N symmetry-related modes. The single optimal mode will be carried forward to the reconstruction step, and each pose within the search radius will be weighted during backprojection by its normalized posterior probability. ![](https://guide.cryosparc.com/files/VHVVMuKPP2RwtwMyCA46) A comparison of the three methods of pose search: Standard alignment (”None”), Maximization, and Marginalization Whether maximization or marginalization should be used depends on the size of the protein, size of the mask, and overall signal-to-noise ratio (SNR) of the dataset. For larger proteins, masks, and higher SNRs, maximization may be sufficient. For smaller proteins, masks, and lower SNRs, marginalization may produce better results. This advice is congruent with our general recommendation that marginalization is preferred when working with smaller proteins and lower SNRs. ### Additional refinement iterations ![](https://guide.cryosparc.com/files/YnhktuA3clmlpoMHPvFy) Several iterations of a Non-Uniform Refinement of EMPIAR 10256 are shown. Homogeneous and Non-Uniform Refinements end when the GSFSC resolution stops improving. In the typical case, this is desirable — further refinements will have little to no effect, so additional iterations are wasted. However, when performing symmetry relaxation, the particles may still move between symmetry-related poses (and therefore improve the quality of the map) even when the GSFSC resolution has stopped improving. We thus typically recommend increasing the \`Number of extra final passes\` when performing symmetry relaxation, perhaps starting with a value between five and ten. ## Symmetry Relaxation Walk Through In this walk-through, we will use a dataset with pseudo-icosahedral symmetry to illustrate how symmetry relaxation can help resolve symmetry breaking features. The density map we’ll be using is available on the Electron Microscopy Data Bank under entry \[#8254\](https://www.ebi.ac.uk/emdb/EMD-8254), “\*\*Phage Qbeta asymmetric reconstruction\*\*”. This structure was solved in \[Gorzelnik et al, 2016\](https://www.pnas.org/doi/full/10.1073/pnas.1609482113) and subsequently referred to in \[Huiskonen, 2018\](https://portlandpress.com/bioscirep/article/38/2/BSR20170203/57149/Image-processing-for-cryogenic-transmission) as an example of a dataset containing a symmetry mismatch. Since the raw movies or particles were not released, we will be using the solved density map to generate simulated particles within CryoSPARC v4.4 using the \[Simulate Data job\](/processing-data/all-job-types-in-cryosparc/simulations/job-simulate-data-gpu.md), and then using these simulated particles to reconstruct the pseudo-icosahedral density. While the use of synthetic data is not representative of most workflows used in practice, we’re presenting this walk-through as it allows us to directly compare the faithfulness of each refinement method (with or without symmetry relaxation) both in terms of map quality and in terms of pose discrepancy from ground truth. With real datasets, we do not have access to ground truth orientations and thus cannot provide as robust of an analysis of the estimated latent variables. ### Synthetic Data Generation To begin, we’ll download the \[EMD-8254\](https://www.ebi.ac.uk/emdb/EMD-8254) volume and run an Import 3D Volumes job in CryoSPARC to import the volume into CryoSPARC. After the volume is imported, we need to ensure that the volume is aligned to CryoSPARC’s icosahedral symmetry axes conventions, which can be done via the symmetry alignment feature in the Volume Alignment Tools job. Even though we are working with an asymmetric volume, we intend on using symmetry relaxation under the icosahedral symmetry group, thus we require the volume to be (as best as possible) aligned to the icosahedral symmetry axes. This step can be done via connecting the imported volume to the Volume Alignment Tools job, and setting the following parameters: \* Do symmetry alignment: True \* Symmetry string: I The output volume is now aligned to CryoSPARC’s icosahedral symmetry axes convention. Next, we’ll generate the synthetic data using the \[Simulate Data job\](/processing-data/all-job-types-in-cryosparc/simulations.md). Connect the aligned volume to Simulate Data, and simulate 20,000 particles at the default signal-to-noise ratio. We will also use \[Volume Tools\](/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools.md) to generate a mask (via thresholding, dilating, and padding) the aligned volume to be used downstream during refinement. Our workflow thus far is included below. ![](https://guide.cryosparc.com/files/ULLUE4bomHBtu7XCQVBT) Workflow (tree view) for particle and mask generation. \### Initial 3D Reconstruction Next, we will proceed with 3D reconstruction as usual: Ab-initio reconstruction followed by refinement. We run Ab-initio reconstruction with the following parameters: \* Maximum resolution (Angstroms): 8 \* Initial resolution (Angstroms): 20 \* Symmetry: I\\\* {% hint style="info" %} \\\*Icosahedral symmetry is applied at this point to ensure Ab-initio finds an output volume that is aligned to the symmetry axes, and to avoid the problem of “flattened” models with high-symmetry data. {% endhint %} In the next step, we’ll create an asymmetric reference from the Ab-initio output. Using Homogeneous Reconstruction Only, we’ll connect the particles from Ab-initio and set the following parameters: \* Symmetry: I \* Break Symmetry: True This will take the input particle stack and reconstruct a volume where each particle’s pose has been randomly permuted by one of the icosahedral symmetry group’s operations. This step effectively “breaks” the perfect symmetry that may be present in the initial set of particle poses, and can be used on the outputs of a symmetry-enforced ab-initio reconstruction or refinement job. ### Comparing Refinement Methods Now we’ll compare the different methods of refinement to see how symmetry relaxation can help resolve symmetry mismatches. The three methods we’ll compare are: \* Standard asymmetric refinement (SAR) \* Symmetry relaxation (SR) via maximization \* Symmetry relaxation (SR) via marginalization Build three Homogeneous Refinement jobs. In all jobs, set the following parameters: \* \*\*Initial lowpass resolution (A): 20\*\* \* \*\*Force re-do GS split: False\*\* \* \*\*Initialize noise model from images: True\*\* For the standard asymmetric refinement, leave all other parameters as default. For the symmetry relaxation via maximization and marginalization, set the following additional parameters: \* Symmetry: I \* Symmetry relaxation method: \`maximization\` or \`marginalization\` Once the three jobs have completed, we can see that both symmetry relaxation (SR) methods recovered the correct asymmetric volume, whereas the SAR volume still appears fully symmetric. ![](https://guide.cryosparc.com/files/4rCMRlVpBRinJLlHNaij) Standard Asymmetric Refinement (SAR, in gray) compared to symmetry relaxation (SR) via maximization (in yellow) and via marginalization (in blue). Note how the volume reconstructed using symmetry relaxation has resolved the internal RNA helices stretching across the inside of the viral capsid. Since we’re working with synthetic data, we can compare the estimated particle poses to their ground truth values, for each of the different methods. Below is a set of histograms in polar coordinates, displaying the number of particles with a given error between the ground truth pose and the pose found by Branch and Bound. We also investigated whether adding extra iterations to SAR helped; the plot below shows the results with 0, 1, and 2 extra iterations added. ![](https://guide.cryosparc.com/files/rEAwi2GLLERAzK4k5RvB) Comparison of the magnitude of the pose error between the ground truth pose and the pose found by Branch and Bound. Each polar-plot histogram displays the number of particles with a given pose error, which starts at 0º at the 12 o’clock position, and increases clockwise. Pose errors are displayed for standard asymmetric refinement (SAR) with 0, 1, and 2 extra iterations (in purple, orange, and yellow, respectively). Both symmetry relaxation methods are displayed in the lower right hand, showing that either symmetry relaxation method better recovers the ground truth for the majority of particles, compared to asymmetric refinement. Histograms concentrated near 0º (12 o’clock position) indicate the majority of particles had their poses correctly recovered. Manually forcing extra refinement iterations helped somewhat to reduce the number of misaligned particles (compare the orange and yellow histograms to the purple one), but both symmetry relaxed methods performed even better. The tree view for this workflow is displayed below. ![](https://guide.cryosparc.com/files/UFsN08kC93enlBRhfKda) \## References \* A. Punjani, J. L. Rubinstein, D. J. Fleet, and M. A. Brubaker, “\[CryoSPARC: Algorithms for rapid unsupervised cryo-em structure determination\](https://www.nature.com/articles/nmeth.4169),” \*Nature Methods\*, vol. 14, no. 3, pp. 290–296, 2017. doi:10.1038/nmeth.4169 \* J. T. Huiskonen, “\[Image processing for cryogenic transmission electron microscopy of symmetry-mismatched complexes\](https://portlandpress.com/bioscirep/article/38/2/BSR20170203/57149/Image-processing-for-cryogenic-transmission),” \*Bioscience Reports\*, vol. 38, no. 2, 2018. doi:10.1042/bsr20170203 \* V. Abrishami \*et al.\*, “\[Localized reconstruction in scipion expedites the analysis of symmetry mismatches in cryo-EM data\](https://www.sciencedirect.com/science/article/pii/S0079610720300390?via%3Dihub),” \*Progress in Biophysics and Molecular Biology\*, vol. 160, pp. 43–52, 2021. doi:10.1016/j.pbiomolbio.2020.05.004 \* K. V. Gorzelnik \*et al.\*, “\[Asymmetric cryo-EM structure of the canonical allolevivirus qβ reveals a single maturation protein and the genomic ssrna in situ\](https://www.pnas.org/doi/full/10.1073/pnas.1609482113),” \*Proceedings of the National Academy of Sciences\*, vol. 113, no. 41, pp. 11519–11524, 2016. doi:10.1073/pnas.1609482113 --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/live/performance-metrics.md). # Performance Metrics ## CryoSPARC Live Performance Metrics CryoSPARC Live is built and tuned for high performance pre-processing and streaming reconstruction of single particle data, and can use multiple concurrent GPUs for to maximize throughput. {% hint style="info" %} \*\*CryoSPARC Live preprocessing includes four steps: (1) motion correction, (2) CTF estimation, (3) particle picking and (4) extraction.\*\* {% endhint %} CryoSPARC Live can sustain a throughput of 450 or more exposures per hour, per GPU, for K3 data. On a 4-GPU machine, that can scale to 1800+ exposures per hour! For K2 or Falcon data, performance can be even higher, upwards of 650 exposures per hour per GPU. Depending on your hardware configuration (particularly raw data storage disk access speed), each preprocessing worker can sustain a throughput of at least one movie every 30 seconds, which is equal to \\~2,500 movies per day per GPU. In our internal tests, we have seen performance on well-tuned systems (like the testing hardware below) reaching up to 8,000 movies per GPU per day. See the \[Hardware Configurations used for Benchmarking section\](#hardware-configurations-used-for-benchmarking) to see details on what hardware was used to run the benchmarks. All 3D renderings were captured in ChimeraX from maps created by cryoSPARC Live. ## Hardware Configurations Used for Benchmarking All pre-processing timings were measured with \*\*Configuration 1\*\*, unless otherwise noted. | Component | Configuration 1 | Configuration 2 | Configuration 3 | | ---------------- | ---------------------------- | ---------------------------- | ---------------------------- | | CPU | AMD Ryzen Threadripper 2950x | AMD Ryzen Threadripper 3960x | AMD Ryzen Threadripper 3960x | | Memory Bandwidth | 128 GB/s | 144GB/s | 144GB/s | | RAM | 128GB DDR4 2666MHz | 256GB DDR4 2933MHz | 256GB DDR4 2933MHz | | GPU 0 | Quadro GV100 | Quadro RTX 8000 | GeForce RTX 3090 | | GPU 1 | Quadro GV100 | Quadro RTX 8000 | GeForce RTX 3090 | | GPU 2 | Quadro RTX 5000 | GTX 1080Ti | - | | GPU 3 | GTX 1080Ti | Tesla K40c | - | {% hint style="info" %} Fast CPU memory bandwidth is a major contributing factor to high performance in cryoSPARC Live. Please make note of this metric when selecting your system's CPU and RAM. {% endhint %} ## K2 MRC (HA Trimer) Benchmark results for 668 MRC-format uncompressed movies from a GATAN K2 4k × 4k detector. The first 40 of 100 frames were used. Exposures from this dataset were captured with the stage tilted 40º. Particles were selected with the Template Picker strategy. Streaming 2D Classification, Ab-initio Reconstruction and Streaming Refinement yielded 3.0Å resolution map from \\~230,000 particles. !\[\](/files/-MNyVWvrHZxYh4vGxUtd) {% embed url="" %} Dataset on EMPIAR {% endembed %} ### Data Properties | Property | Value | | ---------------------------- | ------------- | | Detector | Gatan K2 | | Number of Movies | 668 | | File Format | MRC | | Frame Size | 3838 x 3710 | | Frames per Movie | 100 (40 used) | | Pixel Size | 1.13Å | | Particle Extraction Box Size | 144 × 144 | ### Benchmarks | Metric | Value | | ------------------------------------- | ----- | | Movies Pre-processed Per Hour Per GPU | 430 | | Movies Pre-processed Per Day Per GPU | 10290 | | Average Pre-processing Time Per Movie | 8.4s | ## K2 TIFF (Nav1.7) Benchmark results for \\~24,000 TIFF-LZW compressed movies from a GATAN K2 4k × 4k detector. Particles were selected with the Blob Picker strategy. Streaming 2D Classification, Ab-initio Reconstruction and Streaming Refinement yielded 3.3Å resolution map from \\~300,000 particles. !\[\](/files/-MNyVZm9MBUjGXptF4Dy) {% embed url="" %} Dataset on EMPIAR {% endembed %} ### Data Properties | Property | Value | | ---------------------------- | --------------- | | Detector | Gatan K2 Summit | | Number of Movies | 25084 | | File Format | TIF-LZW | | Frame Size | 3838 × 3710 | | Frames per Movie | 40 | | Pixel Size | 0.85Å | | Particle Extraction Box Size | 512 × 512 | | Particle Extraction Bin Size | 256 x 256 | | Applied Symmetry | C2 | ### Benchmarks | Metric | Value | | ------------------------------------- | ----- | | Movies Pre-processed Per Hour Per GPU | 650 | | Movies Pre-processed Per Day Per GPU | 15600 | | Average Pre-processing Time Per Movie | 5.5s | ## K2 super-res TIFF (T20S) Benchmark results for \\~200 TIFF-LZW compressed movies from a GATAN K2 detector with super-resolution capture. Particles were selected with the Template Picker strategy. Streaming 2D Classification, Ab-initio Reconstruction and Streaming Refinement yielded a 2.5Å resolution map from \\~130,000 particles. The target T20S Proteasome has D7 symmetry. !\[\](/files/-MNyVbBGUUzv05C9BNWi) {% embed url="" %} Dataset on EMPIAR {% endembed %} ### Data Properties | Property | Value | | ---------------------------- | -------------------- | | Detector | Gatan K2 (super-res) | | Number of Movies | 196 | | File Format | TIFF-LZW | | Frame Size | 7420 × 7676 | | Frames per Movie | 38 | | Pixel Size | 0.6575Å | | Particle Extraction Box Size | 448 × 448 | | Applied Symmetry | D7 | ### Benchmarks | Metric | Value | | ------------------------------------- | ----- | | Movies Pre-processed Per Hour Per GPU | 254 | | Movies Pre-processed Per Day Per GPU | 6096 | | Average Pre-processing Time Per Movie | 14.2s | ## K3 TIFF (TT-OAD2) Benchmark results for \\~200 TIFF-LZW compressed movies from a GATAN K3 detector. The first 40 of 64 frames were used. Particles were selected with the Blob Picker strategy. Post-processing (2D Classification, Refinement, etc.) was not run on this dataset. Exposures in this dataset were captured with beam-induced tilt. {% embed url="" %} Dataset on EMPIAR {% endembed %} ### Data Properties | Property | Value | | ---------------------------- | ------------ | | Detector | Gatan K3 | | Number of Movies | 3159 | | File Format | TIFF-LZW | | Frame Size | 5760 × 4092 | | Frames per Movie | 62 (40 used) | | Pixel Size | 0.826Å | | Particle Extraction Box Size | 144 × 144 | ### Benchmarks | Metric | Value | | ------------------------------------- | ----- | | Movies Pre-processed Per Hour Per GPU | 420 | | Movies Pre-processed Per Day | 10050 | | Average Pre-processing Time Per Movie | 8.6s | ## K3 super-res TIFF (TT-OAD2) Benchmark results using super-resolution variants from super-resolution variant of \[previous dataset\](#k3-tiff-tt-oad-2). Only the first 40 frames of each exposure were used. ### Data Properties | Property | Value | | ---------------------------- | -------------------- | | Detector | Gatan K3 (super-res) | | Number of Movies | 4259 | | File Format | TIFF-LZW | | Frame Size | 11520 × 8184 | | Frames per Movie | 67 (40 used) | | Pixel Size | 0.413Å | | Particle Extraction Box Size | 288 × 288 | ### Benchmarks | Metric | Value | | ------------------------------------- | ----- | | Movies Pre-processed Per Hour Per GPU | 192 | | Movies Pre-processed Per Day Per GPU | 4608 | | Average Pre-processing Time Per Movie | 18.7s | ## Falcon III TIFF (PAC1) Benchmark results for \\~3000 TIFF-LZW compressed movies from a Falcon III detector. Particles were selected with the Blob Picker strategy. Post-processing (2D Classification, Refinement, etc.) was not run on this dataset. {% embed url="" %} ### Data Properties | Property | Value | | ---------------------------- | -------------- | | Detector | TFS Falcon III | | Number of Movies | 2895 | | File Format | TIFF-LZW | | Frame Size | 4096 × 4096 | | Frames per Movie | 64 | | Pixel Size | 0.835Å | | Particle Extraction Box Size | 420 × 420 | ### Benchmarks | Metric | Value | | ------------------------------------- | ----- | | Movies Pre-processed Per Hour Per GPU | 493 | | Movies Pre-processed Per Day Per GPU | 11832 | | Average Pre-processing Time Per Movie | 7.3s | ## Falcon IV EER (Apoferritin) Benchmark results for \\~3000 Electron Event Representation (EER) movies from a Falcon IV detector. The particle is highly symmetric. The target apoferritin is highly symmetric. Enough information is present in the dataset to approach atomic resolution. Particles were selected with the ring template picker strategy. Streaming 2D Classification, Ab-initio Reconstruction and Streaming Refinement yielded a 1.9Å resolution map from \\~700,000 particles without any additional processing. !\[\](/files/-MNyVf1R\_km5Xpru4zSC) {% embed url="" %} Dataset on EMPIAR {% endembed %} ### Data Properties | Property | Value | | ---------------------------- | ------------- | | Detector | TFS Falcon IV | | Number of Movies | 3370 | | File Format | EER | | Frame Size | 8192 × 8192 | | Frames per Movie | 434 (40 used) | | Pixel Size | 0.457Å | | Particle Extraction Box Size | 512 × 512 | ### Benchmarks | Metric | Value | | ------------------------------------- | ----- | | Movies Pre-processed Per Hour Per GPU | 303 | | Movies Pre-processed Per Day Per GPU | 7272 | | Average Pre-processing Time Per Movie | 11.9s | ## K2 TIFF with Preprocessing + 2D/3D Streaming (CB1 Gi) Benchmark results for \\~3000 TIFF-LZW compressed movies from a GATAN K2 detector. The target complex is a small, flexible membrane protein. Particles were selected with the Template Picker strategy. Streaming 2D Classification, Ab-initio Reconstruction and Streaming Refinement yielded a 3.9Å resolution map from \\~700,000 particles. !\[\](/files/-MNyVhuQRXkF0IkFeyi3) !\[\](/files/-MNyVk4K9Pqv5YopMyYQ) {% embed url="" %} Dataset on EMPIAR {% endembed %} ### Data Properties | Property | Value | | ---------------------------- | ----------- | | Detector | Gatan K2 | | Number of Movies | 2756 | | File Format | TIFF-LZW | | Frame Size | 3838 × 3710 | | Frames per Movie | 40 | | Pixel Size | 0.86Å | | Particle Extraction Box Size | 360 × 360 | | Particle Extraction Bin Size | 256 × 256 | ### Benchmarks {% hint style="info" %} Pre-processing and streaming results for this dataset measured with \[Hardware Configuration 3\](#hardware-configurations-used-for-benchmarking) {% endhint %} | Metric | Value | | ------------------------------------- | ----- | | Movies Pre-processed Per Hour Per GPU | 870 | | Movies Pre-processed Per Day Per GPU | 20880 | | Average Pre-processing Time Per Movie | 4.13s | #### Post-Processing: 256 × 256 box size, 50 2D Classes | Metric | Value | | ---------------------------------------------- | ---------------------- | | Particles Extracted for 2D Classification | 79,278 | | Time to 2D Classify Extracted Particles | 4 minutes, 16 seconds | | Particles Used for Reconstruction | 100,000 | | Time to Reconstruct Initial Volume (Ab-initio) | 5 minutes, 11 seconds | | Particles Selected for Refinement | 278,312 | | Time to Refine Final Volume | 22 minutes, 55 seconds | --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/live/performance-metrics.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/guide-data-management-in-cryosparc-v4.0.md). # Guide: Data Management in CryoSPARC (v4.0+) {% hint style="warning" %} The information in this guide applies to CryoSPARC v4.0+. For information about managing project directories in older versions, see \[Guide: Data Management in CryoSPARC (≤v3.3)\](/guides-for-v3/tutorial-data-management-in-cryosparc.md) {% endhint %} {% hint style="danger" %} Do not remove from the filesystem any directory that is managed by an\[ attached CryoSPARC project\](#2.-attaching-detaching-archiving-and-unarchiving-projects). First, either \* perform the \*Detach Project\* CryoSPARC GUI action for unwanted project(s) \* or, to delete an unwanted project from the CryoSPARC GUI \*\*and erase the project's data from disk\*\*, perform the \*Delete Project\* GUI action {% endhint %} {% hint style="info" %} For additional data cleanup utilities available in v4.3+, please see: \[Guide: Data Cleanup (v4.3+)\](/setup-configuration-and-management/software-system-guides/guide-data-cleanup-v4.3.md) {% endhint %} ## Overview Single particle cryo-EM projects and labs continue to operate at increasing scales. In CryoSPARC v4.0, we introduce improved workflows and tools for dealing with archiving, transferring, exporting, and importing projects. Tools introduced in previous versions of CryoSPARC for exporting and importing individual jobs and individual results of various types (particle stacks, exposure stacks, volumes, etc.) remain available. In CryoSPARC v4.0, the most important changes that have been made are: \* CryoSPARC projects are now explicitly \*\*locked\*\* (or \*\*attached\*\*) to a single CryoSPARC instance at a time. In previous versions, it was possible for a project directory to be imported into and accidentally modified by two instances at the same time, causing metadata corruption. Now, each project directory contains a lock file that marks the projects as in-use by (i.e., \*\*attached to\*\*) a particular instance. \* CryoSPARC project directories are now named based on the project title at creation time, rather than a numeric project-UID (e.g., "P12"). Numeric UIDs are only used to refer to each project within a single instance. This serves to ensure that project directories are user-recognizable, and that the numeric UID is not retained when a project moves from one instance to another. \* The life-cycle of a CryoSPARC project is now separated from the CryoSPARC instance(s) that interact with the project. The following lifecycle actions can be taken on a project: \* \*\*Create:\*\* an instance can create a new project, and that project lives in a unique and self-contained project directory on disk. The project directory is created at creation time of the project. The project is attached to the instance that created it (and therefore there is a lock file present in the project directory). Within the instance to which the project is attached, the project has a unique numeric UID. \* \*\*Detach:\*\* a user can opt to detach a project from the instance to which it is currently attached. This action ensures that no jobs or background processes are running in the project, and then removes the lock file from the project directory. In the UI, the project that was previously attached displays as "Detached" and can no longer be interacted with. \* \*\*Attach\*\*: a project directory that has previously been detached (and therefore has no lock file present) can be attached to an instance. When attachment is performed, all the workspaces, jobs, and Live sessions within the project directory are imported into the attaching instance, and the project becomes usable within this instance and is given a new numeric UID. A lock file is written to the project directory. Attach takes the place of the previous \*\*Import\*\* action. \* \*\*Archive:\*\* A project that is attached to an instance can be "archived" without detaching the project. This instructs CryoSPARC that the project directory is no longer available for reading and writing at it's current location, but will become available (possibly at a new location) at some future time. A user should archive a project before moving the project directory to a different location on disk, for example a different filesystem, a backup location, tape archive, or cold-storage. Archiving ensures that no jobs or background processes are running in the project and marks the project as archived, but does not remove the lock file from the project. Once archived, the project can still be browsed in the UI, but can not be modified. \* \*\*Unarchive:\*\* A project that has been archived can be resurrected in the same instance from which it was archived. When unarchiving, the user is prompted to provide the (possibly changed) location of the project directory on disk. For example, a project can be archived and then the project directory moved to a cold unaccessible backup. Later, when needed, the project directory can be restored to an accessible filesystem, and the project can be unarchived pointing at the new project directory location. This makes the project available once again for further processing. \* As a minor change, output files of CryoSPARC jobs that are stored in job directories no longer have the \`cryosparc\_PXX\_\` prefix, since the numeric project UID can change when a project moves from one instance to another. In order to retain the existing behaviour of the CryoSPARC UI and limit confusion between different files, when CryoSPARC results are downloaded through the browser in the UI, the prefix \`cryosparc\_PXX\_\` is added to the local filename of the download in the browser, using the then-current numeric project UID. These changes make it much simpler to perform the following actions: \* \*\*Detaching\*\* a project from one CryoSPARC instance and \*\*attaching\*\* it to another instance \* \*\*Sending\*\* a project initially started at a centralized facility to a user who is going to continue processing at home in their own instance, by detaching and attaching \* \*\*Archiving\*\* a project to remote/slow storage for later retrieval and resurrection \* \*\*Copying\*\* a project directory to make a complete clone \* \*\*Changing\*\* the name or location of a project directory on disk, by archiving and unarchiving The following actions remain possible, with no change in behaviour in v4.0: \* \*\*Reducing\*\* the disk space used by a project by removing intermediate results created by jobs \* \*\*Uploading\*\* the final results of a CryoSPARC job to an online repository \* \*\*Advanced manipulation\*\* of CryoSPARC metadata at a low level or programatically, by exporting a results (e.g., a particle stack), manually modifying the associated .cs files, and importing again The following sections describe specific aspects of data management in CryoSPARC in more detail. ## 1. CryoSPARC Projects and Project Directories ### Projects and "Continuous Export" CryoSPARC workflows are naturally divided into projects. Each project should contain the work and jobs for one or more related data collection sessions that are associated with a given sample/target. Project boundaries are strict, in the sense that files and results from one project cannot be directly used in another project. Project directories are self-contained, and all image processing data (except for imported raw data, see \[#7.-imported-data-in-project-directories\](#7.-imported-data-in-project-directories "mention")) pertaining to a project is written to the project directory. A project directory always contains all the information needed to define that project. The project directory is written to every time certain actions are taken within CryoSPARC, for example changing project, workspace, or job metadata (titles, descriptions, etc), and when jobs complete processing. This \*\*"continuous export"\*\* model ensures that at any time, a project directory is self-contained and if anything goes wrong with a CryoSPARC instance or database, the projects remain intact and up-to-date, without the user having to manually trigger an export action. As a safety feature, a project directory can, at any time, be transferred/renamed/copied and \*\*attached\*\* as a valid project in any (other) CryoSPARC instance that can read the files. \*\*This is true even if the original CryoSPARC instance that created the project is no longer functional.\*\* See \[#use-case-rescuing-a-project-from-an-inoperable-instance\](#use-case-rescuing-a-project-from-an-inoperable-instance "mention") for details on how to rescue a project from a failed or inoperable instance. Similar to projects, jobs inside the project are stored in a self-contained format, and job directories are updated whenever the jobs are created, modified, or completed. \*\*Note:\*\* jobs that are in \`launched\`, \`started\`, \`running\`, \`waiting\`, \`killed\` or \`failed\` status will not be updated on disk until they enter \`completed\` status - either by actually completing, or by the user choosing the \`mark as completed\` option in the Job Details panel. ### \*\*Project directories and lock files\*\* For projects created in CryoSPARC v4.0+, the project directory is initially named based on the title of the project at creation time. For example, if a project is titled\\ "My Protein, Data Collection (October 1 2022)"\\ the project directory will be created as\\ \`CS-my-protein-data-collection-october-1-2022\`.\\ The project directory will be created within the container directory indicated at creation time. The project directory can be changed later on (see \[#use-case-renaming-a-project-directory\](#use-case-renaming-a-project-directory "mention")). Inside each project directory in CryoSPARC v4.0+ (including existing projects), there will be a lock file present called \`cs.lock\`. \*\*This file should not be removed or changed.\*\* ### ## 2. Attaching, Detaching, Archiving, and Unarchiving Projects ### \*\*Attach\*\* Detached projects can be \*\*attached\*\* to a CryoSPARC instance. Attaching a project creates a new project in the instance using the indicated project directory. All project details, workspaces, jobs, and sessions in the detached project will be made accessible, and the project directory will be treated as an active project directory by the instance. Any intact CryoSPARC project directory that does not contain a lock file (including from a previous CryoSPARC version) can be attached. {% hint style="warning" %} Note: Projects already belonging to another CryoSPARC instance \*\*cannot\*\* be attached until they are detached from their original instance. When this is not possible see \[#use-case-rescuing-a-project-from-an-inoperable-instance\](#use-case-rescuing-a-project-from-an-inoperable-instance "mention") {% endhint %} Projects can be attached under the “New Project” dropdown menu: ![](https://guide.cryosparc.com/files/WH3UWhDpXG91zd0NlEgQ) Once the attach process begins, you will see a new project appear in the projects page. This project will have a new numeric UID (distinct from the numeric UID of the project in the instance where the project previously was attached). Once attachment is complete, users can begin to interact with the project and continue processing. ### Detach Projects can be \*\*detached\*\* from their CryoSPARC instance. Detaching a project unlocks the project from its instance, allowing the project folder to be moved to another location or attached to another instance. In the UI of the instance where the project is being detached, the project will also display as ‘detached’ and no longer be usable. A detached project’s details, workspaces, jobs, and sessions are saved to the project directory. A project can be detached using the “Detach Project” button in the project’s “Actions” menu: ![](https://guide.cryosparc.com/files/9SyAwxteFIGaqClSL7pr) Detached projects will show an icon on their cards: ![](https://guide.cryosparc.com/files/t9uEgf2qKsasXJjmTeMQ) Upon detaching a project, the project is no longer associated with the CryoSPARC instance, but some project information is retained in the CryoSPARC database. As of v4.1.2, the “Delete Project from Database” action can be used to remove the remaining database entries associated with the project. Performing this action on a detached project hides it in the UI and removes large database files, potentially freeing up space on disk. ### Archive {% hint style="warning" %} This section describes the CryoSPARC \*Archive Project\* function. \*Archive Project\* does \*\*not\*\* copy the project data. Ensure that \*Archive Project\* is followed by copy/transfer of the project directory to long-term storage \*outside\* CryoSPARC, as needed. {% endhint %} {% hint style="warning" %} One must not change the contents of a CryoSPARC project directory outside CryoSPARC. \*Unarchiving\* a project whose project directory has been modified since archiving will lead to inconsistencies between the project directory and the project's database records. Such inconsistencies can lead to CryoSPARC malfunction and data loss. {% endhint %} CryoSPARC projects can be \*\*archived\*\* to allow their project folder to be moved on disk and un-archived at a later date. Archiving sets the project to read-only mode, where it can be seen in the UI but cannot be modified. All project details, workspaces, jobs, and sessions will be maintained in the CryoSPARC database as well as in the project directory on disk. CryoSPARC does not expect the project directory to be available for read or write while a project is archived. When moving a project directory, be sure to consider moving the raw data that was imported into the project as well (see \[#7.-imported-data-in-project-directories\](#7.-imported-data-in-project-directories "mention")) A project can be archived using the “Archive Project” button in the project’s “Actions” menu: ![](https://guide.cryosparc.com/files/nUGT0ximPd43YAPqywGK) Archived projects will show an icon on their card: ![](https://guide.cryosparc.com/files/fIrdf75c0zAjhazUdNks) The archived status can also be seen in the project details: ![](https://guide.cryosparc.com/files/zWz9oDQFLDtNPhE0PgF0) \### Unarchive Archived projects can be unarchived back into the CryoSPARC instance, removing the read-only status and allowing the project to be modified again. Projects can be unarchived from a different project directory location than the location at time of archive. {% hint style="warning" %} Note: Archiving and unarchiving should only be used with the intention of keeping the project tied to the current instance of CryoSPARC. Users looking to transfer projects between CryoSPARC instance should refer to the Attach and Detach features instead. {% endhint %} {% hint style="warning" %} \*Unarchiving\* a project whose project directory has been modified since archiving will lead to inconsistencies between the project directory and the project's database records. Such inconsistencies can lead to CryoSPARC malfunction and data loss. {% endhint %} A project can be unarchived using the “Unarchive Project” button in the project’s “Actions” menu: ![](https://guide.cryosparc.com/files/MqER2rdTB77xpqPAok1P) When unarchiving a project, the project directory must be specified: ![](https://guide.cryosparc.com/files/TdRMNhby0bVnaMf6IErD) \## 3. Ability to view instance storage statistics {% hint style="info" %} This functionality has not changed in v4.0. See \[/pages/-MNiDCV\\\_2xPWJWCTbMK7#3.-ability-to-view-instance-storage-statistics\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/pages/-MNiDCV\_2xPWJWCTbMK7#3.-ability-to-view-instance-storage-statistics "mention") {% endhint %} ## 4. Ability to clear intermediate results {% hint style="info" %} This functionality has not changed in v4.0. See\[/pages/-MNiDCV\\\_2xPWJWCTbMK7#4.-ability-to-clear-intermediate-results\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/pages/-MNiDCV\_2xPWJWCTbMK7#4.-ability-to-clear-intermediate-results "mention") {% endhint %} Several job types (2D Classification 3D Classification, and 3D Variability Analysis) have an option to control whether the job will save intermediate results at all. By default, jobs will save intermediate results. However, this can be turned off on a per-job level using job parameters, or it can be turned off at the project level for each job type. To do so, select the project and at the bottom of the details panel, set job-specific defaults under the 'Generate Intermediate Results' module: ![](https://guide.cryosparc.com/files/Dab8BHao8GKoyPNHR2ST) Project-level defaults for generation of intermediate results \## 5. Ability to export and import individual jobs {% hint style="info" %} This functionality has not changed in v4.0. See \[/pages/-MNiDCV\\\_2xPWJWCTbMK7#5.-ability-to-export-and-import-individual-jobs\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/pages/-MNiDCV\_2xPWJWCTbMK7#5.-ability-to-export-and-import-individual-jobs "mention") {% endhint %} ## 6. Ability to export and import low-level output groups {% hint style="info" %} This functionality has not changed in v4.0. See \[/pages/-MNiDCV\\\_2xPWJWCTbMK7#6.-ability-to-export-and-import-low-level-output-groups\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/pages/-MNiDCV\_2xPWJWCTbMK7#6.-ability-to-export-and-import-low-level-output-groups "mention") {% endhint %} ## 7. Imported data and symlinks in project directories When raw data is imported into a CryoSPARC project (using an \`Import\` Job), the raw data is not copied into the project directory. Rather, symlinks are created within the Import Job directory pointing to the raw data. Aside from these symlinks, CryoSPARC jobs do not create symlinks that point to locations that are outside the project directory. This keeps project directories self-contained. The symlinks within import jobs can be changed if the position of the raw data on disk changes. For example, when Archiving a project, if the raw data (e.g., raw movies) are also archived to a different location than where they were imported from, the import symlinks must be updated. See the guide here for more details:\[/pages/-MNeMxOeXMsIWu8jTuYb#a.-moving-only-raw-particle-micrograph-or-movie-data-already-imported-into-cryosparc\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/pages/-MNeMxOeXMsIWu8jTuYb#a.-moving-only-raw-particle-micrograph-or-movie-data-already-imported-into-cryosparc "mention") Due to the use of symlinks, it is important that when copying or moving a project directory, symlinks \*\*NOT\*\* be dereferenced (i.e., do not use the \`-h\` flag with \`tar\` ). If symlinks are dereferenced, the new copy of the project directory will also contain copies of all the raw data files, as well as potentially multiple copies of intermediate and output files that are internally symlinked within the project directory. Instead of dereferencing symlinks, raw data should be archived separately from project directories. ## Use Cases and Examples ### Use Case: Moving a project directory from one storage location to another Sometimes, you may need to move a project directory on disk. You may have created it in the wrong place accidentally, you may have a full disk, or if you have tiered storage, e.g., a fast SSD-backed storage system for active projects and a slower HDD-backed storage array for bulk storage, you may wish to move a project directory from the fast filesystem to the slower filesystem once most processing is complete. In these cases, you can simply: 1. Archive the project 2. Move the project directory to its new location 3. Unarchive the project using the path to the project directory at its new location The project will now be usable once again, and all reads/writes will happen to the new project directory location. ### Use Case: Renaming a project directory In CryoSPARC v4.0+, project directories are named based on the project title entered at creation time. If you later change the title, you can rename the project directory using the following steps: 1. Archive the project 2. Rename the project directory on disk, but leave it in it's original location 3. Unarchive the project using the path to the project directory with its new name ### Use Case: Transfer a project from one CryoSPARC instance to another When you need to move a CryoSPARC project between instances, for example when transferring a project from a data collection facility to a user's home facility, use the following steps: 1. Detach the project from its original CryoSPARC instance 2. Copy the project directory to a location accessible by the new CryoSPARC instance 3. Attach the project to the new CryoSPARC instance using the path to the project directory at its new location 4. (Optional, available in v4.1.2+) Use the “Delete Project from Database” action on the detached project to remove and remaining database entries relating to this project from its original CryoSPARC instance ### Use Case: Archive or Detach a project directory and consolidate it for long term storage Once processing in a project is complete, the project can be either detached (if it is unlikely to be brought back to the same CryoSPARC instance) or archived. Either action will allow the project directory to be moved or compressed without causing errors in the CryoSPARC instance. Be sure to separately archive/copy/move the raw data that was imported into the CryoSPARC project, as raw data is not stored inside the project directory. See \[#7.-imported-data-and-symlinks-in-project-directories\](#7.-imported-data-and-symlinks-in-project-directories "mention") The project directory can be copied as-is, and stored on a backup, remote, or cold-storage filesystem. In some cases it may help to \`tar\` the project directory into a single file. An example command to consolidate a project directory is: \`\`\` cd /u/cryosparcuser/cryosparc\_projects/ tar -cvf P47.tar ./P47 \`\`\` Note that you can use any method you choose to archive/transfer/store the project directory, as long as the entire contents remain intact. If you need to access the project at a later date, you can un-\`tar\` the bundle to any accessible filesystem. Then, if the project was archived (you can tell by checking that the \`cs.lock\` file is still present inside the project directory), you can un-archive it to the same instance from where it was archived. Otherwise if it was detached, you can attach it in any instance. ### Use Case: Rescuing a project from an inoperable instance If a CryoSPARC v4.0+ instance is no longer operable (due to database corruption or other issue), a project that was attached to that instance can be rescued by attaching to a new instance. Use the following steps: 1. Ensure that that inoperable instance is completely shut down, and that there are no remaining "zombie" processes associated with that instance still running. 2. For additional safety, make a copy of the project directory to be rescued and use the copy for subsequent steps. 3. Delete the \`cs.lock\` file in the project directory. 4. In the new instance, use \*\*Attach Project\*\* and point to the project directory where the lock file was removed. 5. The new instance should import all available workspaces, jobs, and sessions and make the project directory available for use once again. ### Use Cases that are unchanged in v4.0 The following use cases remain unchanged in v4.0+: \* \[Guide: Data Management in CryoSPARC (≤v3.3)\](/guides-for-v3/tutorial-data-management-in-cryosparc.md#use-case-share-a-particular-job-with-another-user) \* \[Guide: Data Management in CryoSPARC (≤v3.3)\](/guides-for-v3/tutorial-data-management-in-cryosparc.md#use-case-upload-your-particle-stack-to-empiar) \* \[Guide: Data Management in CryoSPARC (≤v3.3)\](/guides-for-v3/tutorial-data-management-in-cryosparc.md#use-case-manually-modify-cryosparc-outputs-and-metadata-for-continued-experimentation) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/guide-data-management-in-cryosparc-v4.0.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation.md). # Tutorial: 3D Flex Mesh Preparation It can often be important to the success of 3D Flexible Refinement that a mesh with the proper topology is created prior to training. The mesh defines what types of motion are “allowable”. In this article, we first highlight the theoretical grounding for 3D Flex meshes and why custom meshes may be necessary in many cases. We then walk through the process of creating an example mesh. More detail is available in the original 3D Flexible Refinement publication (\[Punjani and Fleet, 2023\](#references)). The material covered in this page is also available in video form: {% embed url="" fullWidth="true" %} ## Summary 3D Flex Meshes are used to regularize the 3D Flex deformation model during training. Custom meshes are most useful when discontinuous motion, such as separation or sliding, are expected. When generating a custom mesh, special care must be taken when deciding on a fusion strategy to allow the model to capture the full range of target motion. ## 3D Flex Mesh Overview ### Why model deformation with a mesh? At its core, CryoSPARC’s 3D Flex is attempting to determine the conformation of a given single particle using just one image. The algorithm treats conformational change as movement of electron potential from one voxel to another. An algorithm to directly solve this problem by checking various mixtures of density in various voxels would be both slow and sensitive to overfitting. We have therefore implemented 3D Flex to instead model movement of the vertices of a mesh. This reduces the number of degrees of freedom in the model and imposes smoothness on the resulting flexibility because nearby regions of the map must move together. The reduced number of degrees of freedom also result in increased speed during training. To explain how the use of a deformation mesh achieves these goals, consider a theoretical 1D particle. This particle can stretch or squish to various degrees along its length. Rather than directly try to model the movement of intensity from one pixel to another, we instead create a mesh along the particle, like so. ![](https://guide.cryosparc.com/files/s52kYUcoCrGJ3WRmiKqI) Then, rather than applying a deformation to each pixel individually, we instead move just the points of the mesh. Everything between the mesh points is squished or stretched linearly as the points on either side move closer or further apart: ![](https://guide.cryosparc.com/files/nhEXt3GM6FnkCUSkdJhq) The same principle can be applied in higher dimensions — the important point is that the algorithm is only modeling deformation of the \*mesh\*. The deformation of the underlying volume is, in turn, a function of the deformation of the mesh. ![](https://guide.cryosparc.com/files/rnZ4n7Qkcxt9f1wzL438) Meshes are created with the \[3D Flex Mesh Prep\](/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement.md#mesh-generation) job. This job requires a consensus volume and a solvent mask. When you run this job, a number of tetrahedral cells (hereafter just “tetras”) will be created to span the entirety of the box. The resulting tetra mesh can then be used by a \[3D Flex Training\](/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement.md#parameter-tuning) job to model the movement of the particles. It is during the training job that the tetra cell vertices are moved to model deformation — 3D Flex Mesh Prep jobs produces the vertices in their evenly-spaced “starting” positions. This is an important point — both vertices and tetras are important in 3D Flex. When a vertex moves, the tetra deforms. This deformation will move any voxels inside the tetra in a linear fashion. For a simpler example, consider again the two-dimensional case. We start with a consensus reconstruction, a solvent mask, and a set of “tetras” (triangles in 2D). When we move individual vertices along the right-hand side of our mesh, the rectangular particle is deformed linearly within each tetra. ![](https://guide.cryosparc.com/files/I0cmV7vNBoU8p8sFm9BT) Using this technique we can model most forms of \*\*continuous\*\* motion as movement of the vertices which make up the mesh (or, equivalently, the expansion and contraction of tetras). Note, however, that \*\*moving a vertex deforms all adjacent tetras in the same way\*\*. All of these tetras share their vertices, so they cannot move independently from each other. We thus cannot model discontinuous motion (i.e., domains which slide past each other or move apart from one another) using a single tetra mesh. We therefore must \*segment\* the mesh into separate parts which can move independently. ### Discontinuous Motion Consider the case in which we have two domains which “slip” past each other: ![](https://guide.cryosparc.com/files/UCPulfimJb1rEpq0E9YV) To model the movement of the purple arrow downward we could move the vertices enclosing it downward. Similarly, to move the orange arrow upward we could move the vertices surrounding the orange arrow upward. However, this creates a problem at the interface between the two arrows — the vertices there want to move downward and upward at the same time. ![](https://guide.cryosparc.com/files/9hRKKvWwfXrYFgVT6uwn) This is not possible with a single mesh — the vertex must move either up or down. Therefore, the sliding movement of these two arrows cannot be modeled with this mesh topology. To resolve this conflict, we instead create two overlapping meshes and assign each arrow to its respective mesh. These meshes \*start\* with vertices and tetras in the same position, but there are now \*multiple vertices and tetras\* where the meshes overlap. ![](https://guide.cryosparc.com/files/zOhPumA0YO5HTu6W34PI) In this image, we have segmented our starting mesh into two distinct meshes, one for each arrow. As before, each mesh is defined by a set of vertices, indicated with blue circles. The vertices are annotated with a symbol corresponding to the mesh to which they belong. At the border between the two arrows, where the two meshes meet, there are two vertices which start in the same position. These vertices are thus annotated with both an up- and down-arrow symbol. This vertex duplication allows each mesh to deform the same region of space in its own direction, since it models that region with vertices and tetras which are independent of the other mesh. ![](https://guide.cryosparc.com/files/c383rPI1J6VOzYX9eCEW) Note that the formerly-overlapping vertices are no longer in the same position. The orange mesh moved its vertices up and to the right, while the purple mesh moved its vertices down and to the left. We have successfully modeled sliding movement of the arrows, even though they are positioned in a region of space which fits inside a single tetra. ## When is a custom mesh required? As demonstrated above, domains which are expected to slide past each other or move apart from each other \*require\* custom meshes. Other types of flexibility, or specific targets, may benefit from a custom mesh even in the absence of sliding or separating movements. For instance, proteins with a micelle may benefit from a custom mesh in which the micelle is segmented into a separate mesh which is then marked as rigid, so as not to waste latent space modeling uninteresting deformation of the micelle. In the remainder of this page, we walk through the process of segmenting and creating a mesh. How does CryoSPARC assign voxels to one or the other overlapping mesh? When you create a segmented 3D Flex mesh, two objects are actually created. The first is the mesh (or set of meshes), which itself comprises a \*\*list of vertices\*\* and the \*\*connections between those vertices\*\*. The connections are listed in out in sets of four, with each set therefore defining a tetrahedral cell. Each of these tetra cells are then assigned to a mesh. The second object is a \*\*tetra mask\*\*, which is a 3D volume the same size as the input volume used for segmentation and training. Each voxel of the tetra mask stores an integer value corresponding to the mesh to which that voxel “belongs”. During training and reconstruction, the density in a particular voxel is flowed along the vector described by the mesh indicated by its coordinate in latent space and the tetra to which it is mapped. !\[\](/files/u5WycrdT2RXMqEPVx9sM) In this way, tetra meshes can overlap while still only moving map density from “their own” voxels, and voxel-level segmentation is retained while the tetras themselves are significantly larger than a single voxel. \## Creating a custom mesh Creating a custom mesh from prepared data comprises three main steps: 1. Create the segmentation 2. Decide on a mesh fusion strategy 3. Run the 3D Flex Mesh Prep job This tutorial will walk through the details of each of these steps, working through the process of creating a custom 3D Flex Mesh for NaV 1.7 (EMPIAR 10261). The data used in this tutorial were originally collected and processed by \[Xu and colleagues (2019)\](#references). ### Creating a segmentation 3D Flex Mesh Prep jobs need a way to assign each voxel in the consensus map to a particular tetrahedron. In the default case, there is only one tetra segment which covers the entire map. Assignment is thus trivial — voxels belong to the lone tetra that encloses that voxel. However, in the case of the custom mesh, voxels which are on the border between two segments may be enclosed by two or more tetra cells, so their assignment is uncertain. The segmentation we create in this step explicitly assigns every voxel inside the mask to one and only one tetra. {% hint style="info" %} \*\*Segmenting the mesh using another tool\*\*\\ We recommend creating a segmentation using the Segger tool in ChimeraX, and we cover that technique in this tutorial. If you prefer another tool, 3D Flex Mesh Prep jobs also accept an \`.mrc\` file with an integer value in each voxel corresponding to that voxel’s segment. The segmentation \`.mrc\` file should have the same dimensions as the map from 3D Flex Data Prep. In each voxel, store an integer value to indicate the segment ID for that voxel. All voxels of a given segment should contain the same segment ID value. The segment IDs you use should start from zero and be contiguous (i.e., for \`S\` segments, use the IDs \`0, 1, 2,..., S-1\`) . Indicate solvent voxels (i.e., "No segment") with \`-1\`. When identifying segments for fusion, use the integers in the \`.mrc\` file instead of using ChimeraX segment ID numbers. {% endhint %} First, download the consensus volume from the 3D Flex Data Prep job. This volume has already been cropped and downsampled to match the training box size. For Nav 1.7, the consensus volume looks like this: ![](https://guide.cryosparc.com/files/tNEeNPPFkGf7FQnQwQnL) From the lower contour (transparent in this image) it is clear that the C-terminus of the channel (bottom of the image) is poorly aligned, perhaps due to flexibility. However, it should never slide along or separate from the well-aligned transmembrane domain (TMD), so the TMD and C-terminus should be one segment. Two fabs are bound to the channel at the top of the image. A 3D Variability Analysis job shows that these Fabs flex toward and away from each other, so they should be separated into individual mesh segments. ![](https://guide.cryosparc.com/files/7Fad1aHq3L1u5kt03QkT) 3D Variability Analysis of the NaV 1.7 channel. Finally, the micelle is visible in the low contour. There is not a clear best method of treating the micelle in 3D Flex Meshes. We generally recommend starting by segmenting the micelle into its own mesh and marking it as rigid, but there are several alternate treatments of the micelle which may produce better results with some targets. See the \[Micelles, fusion, and rigidity\](#micelles-fusion-and-rigidity) section of this page for more discussion of this topic. The custom mesh for this target will therefore have four mesh segments: 1. The “root” segment of the TMD and C-terminus 2. The micelle, which should be treated as if it is rigid 3. The “left” Fab 4. the “right” Fab Following the guidance in the \[Mask Creation tutorial\](/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera.md#method-one-volume-segmentation), the Segger segmentation for such a mesh looks like this: ![](https://guide.cryosparc.com/files/LeE6n0LxvP6p7c7FPeFv) At this point you should make note of the segment numbers for your segmentation. You can find the segment number by hovering over a region of the segmentation and reading the number out of the tooltip. In this example, the “left” Fab is segment \`570\`, the “right” fab is segment \`569\`, the micelle is segment \`572\`, and the TMD/C-terminus segment is number \`571\`. {% hint style="info" %} The hover tooltip displays both the segment number and ChimeraX’s model ID. \*\*The segment number is the one you will enter into CryoSPARC\*\* and usually appears first. It does not have a prefix and is typically a larger integer, e.g., \`570\`. The ChimeraX model ID, which is not what you want, is instead prefixed with a \`#\` and is typically two small integers separated by a \`.\`, e.g., \`#2.2\`. {% endhint %} Save the segmentation as a \`.seg\` file using File > Save segmentation in the Segger pane and upload the resulting file to the CryoSPARC system. ### Decide on a fusion strategy #### Why fuse segments? Recall that the ultimate goal of this job is to create separate mesh segments with overlapping tetra cells to allow for discontinuous movement of domains which are not physically attached to each other. When you import your segmentation, the creation of these independent meshes is automatic, since they are fully defined by the base tetra grid and the segmentation file. However, there is no way to know from this information alone which vertices should be physically coupled and which should be allowed to move freely. For instance, using the above segmentation, the mesh creation job will create four tetra mesh segments. ![](https://guide.cryosparc.com/files/KRmXTKyqGgj6pCRJX9q2) In these images, tetras are colored by which mesh segment they belong to. They have also been slightly scaled inward for visibility — the true tetra cells share edges with no gap between them. Each of these mesh segments comprise individual tetras which use the same vertex grid. Additionally, each segment contains some tetra cells which enclose the same physical space as other grids. For instance, the green and yellow grids both contain tetra cells which enclose the interface between the two fabs. ![](https://guide.cryosparc.com/files/j4zXQJEeSmmhFGHF5kZT) We can show this in another way by marking the vertices with symbols based on which grids use that vertex, as we did for the 2D example above: ![](https://guide.cryosparc.com/files/aRQLZcui4qv3FBmTF9A0) In this image, the base tetra mesh vertices are marked with transparent spheres. If a tetra mesh contains any tetra cells which use a vertex, that vertex is marked with a symbol corresponding to that mesh. Vertices between the two fabs are used by both the green and yellow meshes, so they are marked with a green cube and a yellow cone. As with the 2D example, \*\*the green and yellow meshes each have their own “copy” of these vertices\*\* that they can move independently during training and reconstruction — they only overlap now because both meshes start in their consensus, undeformed state. Additionally, each Fab also has overlapping tetra cells with the TMD/C-terminal segment: ![](https://guide.cryosparc.com/files/FfQ4MaE73gnDdcnTl02H) The Fabs are bound tightly to the TMD, and so the 3D Flex model should not be allowed to move voxels which belong to the fab away from voxels which belong to the TMD. However, if each Fab is in its own mesh, there is no reason for the model to keep them attached to the TMD. We must impose this mesh fusion on the model. ![](https://guide.cryosparc.com/files/5sdme270Aey1q8UrJpDL) Consider the 2D cartoon above. If each mesh is allowed to move independently (top), the arrows can separate from each other instead of sliding. If, however, the overlapping tetras are removed and the border vertices are forced to occupy the same position, the orange arrow can no longer move away from the purple arrow. The process of: 1. Deleting one copy of tetras shared by two meshes, then 2. Fixing the position of the border vertices to be the same in both meshes is called a mesh fusion. Tuning these fusions is an important component of generating a custom mesh. Typically, several fusion strategies should be tried to find the optimal parent/child order and final mesh topology. In this example, we will fuse each fab and the micelle to the TMD and no other fusions. #### Mesh Fusion and Rigidity It is important to be careful when choosing a mesh fusion strategy when some segments will be marked rigid. ![](https://guide.cryosparc.com/files/BlyOaGQolkUyBtymEchT) In this cartoon example, the blue segment is fused to yellow, yellow to orange, etc. The pink segment is the only rigid segment. It may be obvious that the vertices that purple and red share with pink will be rigid, since those mesh segments are fused. However, the central vertex of \*all\* of the mesh segments will be rigid, since blue cannot move its central vertex without moving that of yellow, etc., until we consider that red’s central vertex cannot move without moving the rigid vertex of pink. Although this fact may be readily apparent in this simple example, results of segment fusion can be less obvious when considering a three-dimensional mesh. ### Running the Job Create the 3D Flex Mesh Prep job and input the prepared solvent mask and consensus volume and enter the path to the \`.seg\` file. In the example segmentation, the TMD/C-terminus segment is \`571\`, the two Fabs are \`569\` and \`570\`, and the micelle is \`572\`. The \`Segment connections\` field (which controls segment fusions) should therefore read \`571>569,571>570,571>572\` to produce our desired topology of TMD > Fab1, TMD > Fab2, TMD > micelle. {% hint style="info" %} Currently, 3D Flex Mesh Prep cannot create a mesh with a base num. tetra cells greater than 40. {% endhint %} Finally, we set the micelle segment fully rigid by entering \`572\` in \`Rigid segments\`. This allows the 3D Flex Train job to correctly model the micelle without trying to move density in or out of this region. ## Results ![](https://guide.cryosparc.com/files/ottKAC18jvs4wk9kquuA) Trajectories along each of the three latent space coordinates when the micelle is marked rigid and fused to the TMD. The three components of the 3D Flex Train results are displayed above. The constant regions (at the top of the image) of the fabs are able to move closer together and further apart with the custom mesh than the default mesh, because they are not fused together. Note, however, that the TMD is held relatively steady because of the rigidity of the micelle segment. Directly investigating the mesh can be overwhelming, but provides insight into how tetra cells are allowed to deform when one or more of their vertices are fused with other tetra segments. ![](https://guide.cryosparc.com/files/tICTZi1NSGgughJ8gpI7) \## Micelles, fusion, and rigidity Consider the motions detected by 3D Variability again, paying close attention to the transmembrane domain: ![](https://guide.cryosparc.com/files/7Fad1aHq3L1u5kt03QkT) 3DVA modeled a significant flexing motion of the TMD, but the 3D Flex results captured only a slight deformation of this region. While it is possible that 3D Flex more accurately modeled the data and the channel truly is rigid, it is also possible that this rigidity is an artifact of marking the micelle segment rigid. As discussed above, if a vertex belongs to two fused meshes, it must occupy the same position in both meshes. This means that the edges of the TMD are treated as rigid, since they must be in the same position as the rigid micelle. Additionally, because the mesh is extended to fill the mask, the C-terminus moves in our custom mesh example than the default mesh, because it is also fused to the rigid micelle. It is undesirable for the micelle to hold parts of the target in a single rigid conformation. However, there is not yet an obvious best treatment of micelles in 3D Flex. Below we present a non-exhaustive collection of alternate fusion strategies, each of which have their own advantages and drawbacks. Aside from the fusion parameter, these training jobs all used the same parameters. Optimization of the other training parameters for each fusion strategy may further improve results. Which of these strategies (or a strategy of your own creation) works best depends on the sample. The best results typically require trying a few different fusion strategies, especially for new targets with as-yet unknown motion. {% hint style="info" %} Note that it is not currently possible to make a one-to-one comparison between different 3D Flex training jobs, as the latent spaces are almost certainly different. For these examples we display a trajectory along each coordinate (the default behavior of 3D Flex Generate). {% endhint %} ### Alternate strategy 1: Do not mark the micelle as rigid Another approach would be to simply allow the micelle to move as much as any other segment. This approach allows the Flex model to “see” micelle density and so prevents it from filling the micelle with density from other regions, and allows nearby regions like the TMD to move freely. However, in some cases, the model will dedicate an undesirable amount of focus to modeling micelle noise rather than biologically-relevant flexibility. Here is the result of using the same mask and mesh as above, but without setting the micelle as rigid. ![](https://guide.cryosparc.com/files/gfXarExipEhsZi9Txbwi) Trajectories along each of the three latent space coordinates when the micelle is fused to the TMD without being marked as rigid. The flexibility of the fabs remains clear, but the movements of the TMD and C-terminus have also been captured, unlike the rigid micelle example. Clearly, fusing a rigid micelle to the TMD had a detrimental effect on the ability of 3D Flex to model those domains. ### Alternate strategy 2: Mask out the micelle entirely One obvious approach is to mask the micelle out entirely, as one would for 3DVA or Local Refinement. In some cases, this does produce good results. In others, the model moves density which belongs to the helices to fill the empty space where the micelle should be. This happens because no matter how we mask the volume, the micelle is still present in the images, and the model has no way of knowing which parts of the image we may want it to ignore. Here is the result of using a mask during the mesh prep job that does not include the micelle, then creating a custom mesh with three segments: one for each fab and one for the channel. ![](https://guide.cryosparc.com/files/aTnhZlToPCP1LI75IGhy) Trajectories along each of the three latent space coordinates when the micelle is masked out of the consensus volume prior to training. This mesh produced a model in which the C-terminus and TMD are both highly flexible — more than either the the rigid or non-rigid micelle models. However, the movement of the fabs toward and away from each other is no longer clear. Unlike all other forms of refinement currently available in CryoSPARC, 3D Flexible Refinement performs volume reconstruction in real space rather than Fourier space. This can introduce surprising and significant artifacts when the mask cuts through significant map density. If large streaks of map density are observed, we recommend increasing the size of the mask. ![](https://guide.cryosparc.com/files/lstqrlFdoVE0w543wF0s) Masking artifacts in 3D Flexible Refinement. \### Alternate strategy 3: Fuse the rigid micelle to a less flexible region Recall that when we fuse regions, we do not necessarily need to follow the physical linkages of the sample. In the case of the NaV 1.7 channel, the micelle truly is fused (in some sense) to the TMD — but for the purposes of 3D Flex, we could fuse it to one of the Fabs instead. This would allow the TMD to move independently of the micelle, and could allow rigid body movement of the entire channel/fab complex relative to the micelle. Below is an example of this fusion strategy. ![](https://guide.cryosparc.com/files/OuQhjFMKPJx3xvdvcbmw) Trajectories along each of the three latent space coordinates when the micelle is marked as rigid and fused to a fab (the fab on the right in these maps). Treating the micelle in this way also failed to capture the flexibility of the TMD, and also may have held the Fabs more rigid. However, the model is able to move the C-terminus of the channel significantly more in this topology than when the micelle was fused to the ion channel. Recalling that each mesh segment is expanded to fill the mask this result is not too surprising, since the rigid micelle and C-terminus share a border. The movement of the C-terminus in this example may therefore also be captured if a fifth mesh segment was added surrounding the C-terminus and fused to the TMD. ## References 1. Ali Punjani and David J. Fleet, “3DFlex: Determining Structure and Motion of Flexible Proteins from Cryo-EM,” \*Nature Methods\* 20, no. 6 (June 1, 2023): 860–70, . 2. Hui Xu et al., “Structural Basis of Nav1.7 Inhibition by a Gating-Modifier Spider Toxin,” \*Cell\* 176, no. 4 (February 7, 2019): 702-715.e14, . --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/tutorial-verify-cryosparc-installation-with-the-extensive-workflow-sysadmin-guide.md). # Guide: Verify CryoSPARC Installation with the Extensive Validation Job (v4.3+) {% hint style="info" %} The "Extensive Workflow" job has been renamed to "Extensive Validation" in CryoSPARC v4.3.0+. For the version of this guide applicable to CryoSPARC versions ≤v4.2, please see: \[Extensive Workflow\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/tutorial-verify-cryosparc-installation-with-the-extensive-workflow-sysadmin-guide-1) {% endhint %} {% hint style="success" %} In CryoSPARC v5.0+, many job types have been added to the Extensive Validation Job's test set, and in advanced mode (see below), nearly all job types that exist are launched, providing for a comprehensive test of CryoSPARC. {% endhint %} ## Introduction CryoSPARC's "Extensive Validation" job orchestrates a full 3D target reconstruction for two datasets: \* T20S Proteasome (\[EMPIAR-10025\](http://pdbe.org/EMPIAR-10025)) from a small subset of movies (\\~8GB) \* Tobacco Mosaic Virus (\[EMPIAR-10305\](https://www.ebi.ac.uk/empiar/EMPIAR-10305/)) CryoSPARC's engineering team uses this job to automatically test and benchmark CryoSPARC between releases. System Administrators may use the Extensive Validation job to verify that CryoSPARC is correctly configured following a fresh installation or an update. !\[Table view of a workspace with an Extensive Validation run\](/files/k26yaPzYSrcvMPTUU5O0) ### Benchmarking vs. Testing The Extensive Validation job has two modes: Benchmark mode and Testing mode. ![](https://guide.cryosparc.com/files/Eu7lgVvdwpNbIhWYZecp) Switch the Extensive Workflow job from "Testing" mode to "Benchmark" mode using the drop-down menu. In Benchmark mode, jobs with pre-defined parameters run sequentially on the worker node in order. Each job accesses available system resources independently of other jobs to collect accurate runtime statistics. \*\*Benchmark mode is useful for evaluating the overall performance of a worker node.\*\* In Testing mode, jobs run in parallel when possible. Multiple parameter combinations of each job are dispatched. \*\*Testing mode is useful for ensuring that a CryoSPARC instance is installed correctly.\*\* Both Benchmark and Testing modes verify the following system requirements: \* CryoSPARC system and license installation \* Worker/Cluster configuration \* GPU and CUDA driver installation \* SSD caching ### Datasets Available {% hint style="info" %} CryoSPARC downloads the selected dataset into the project directory when the Extensive Validation job first runs in the current project. {% endhint %} #### \[EMPIAR-10025\](https://www.ebi.ac.uk/empiar/EMPIAR-10025/) \* \*\*Number of movies:\*\* 20 \* \*\*Frames per movie:\*\* 38 \* \*\*Movie size:\*\* 7420 × 7676 (K2 Super Resolution) \* \*\*Pixel size:\*\* 0.86 Å \* \*\*Particles processed:\*\* 10,000 \* \*\*Particle box size (pixels):\*\* 448 Jobs run in Benchmark Mode (up to CryoSPARC v4.7) 1. Import Movies 2. Patch Motion Correction 3. Patch CTF Estimation 4. Manually Curate Exposures 5. Blob Picker 6. Inspect Particle Picks 7. Extract From Micrographs (CPU) 8. 2D Classification 9. Select 2D 10. Template Picker 11. Inspect Particle Picks 12. Extract From Micrographs (CPU) 13. 2D Classification (50 Class) 14. 2D Classification (100 Class) (All Job Types Enabled) 15. 2D Classification (200 Class) (All Job Types Enabled) 16. Select 2D 17. Particle Sets Tools 18. Ab-Initio Reconstruction 19. Ab-Initio Reconstruction (3 Class) (All Job Types Enabled) 20. Homogeneous Refinement 21. Non-Uniform Refinement 22. Homogeneous Refinement Legacy (All Job Types Enabled) 23. Non-Uniform Refinement Legacy (All Job Types Enabled) 24. 3D Classification 25. 3D Variability (3 Mode) 26. 3D Variability (6 Mode) (All Job Types Enabled) Jobs run in Testing Mode (up to CryoSPARC v4.7) 1. Import Movies 2. Patch Motion Correction 3. Full-Frame Motion Correction 4. Patch CTF Estimation 5. CTFFIND4 6. Curate Exposures (Stream A) 7. Curate Exposures (Stream B) 8. Blob Picker (Stream A) 9. Blob Picker (Stream B) 10. Inspect Picks (Stream A) 11. Inspect Picks (Stream B) 12. Extract From Micrographs (Stream A) 13. Local Motion Correction (Stream B) 14. 2D Classification (Stream A) # all jobs past this point are in Stream A 15. Select 2D 16. Template Picker 17. Inspect Picks 18. Extract From Micrographs 19. 2D Classification (50 Class) 20. 2D Classification (100 Class) 21. 2D Classification (200 Class) 22. Select 2D 23. Particle Sets Tools 24. Ab-Initio Reconstruction 25. Ab-Initio Reconstruction (3 Class) 26. Homogeneous Refinement 27. Non-Uniform Refinement 28. Homogeneous Refinement (Legacy) 29. Non-Uniform Refinement (Legacy) 30. Heterogeneous Refinement (3 Class) 31. Heterogeneous Refinement (6 Class) 32. 3D Classification (Simple mode) 33. 3D Classification (PCA mode) 34. 3D Variability (3 mode) 35. 3D Variability (6 mode) 36. Sharpening Tools 37. Validation (FSC) 38. Global CTF Refinement 39. Local CTF Refinement 40. 3D Variability Display Jobs run in Benchmark Mode (CryoSPARC v5.0+) 1. Import Movies 2. Patch Motion Correction 3. Patch CTF Estimation 4. Manually Curate Exposures 5. Blob Picker 6. Inspect Particle Picks 7. Extract From Micrographs (CPU) 8. Micrograph Junk Detector (v5.0+) 9. 2D Classification 10. Downsample Particles (v5.0+) 11. Select 2D 12. Template Picker 13. Inspect Particle Picks 14. Extract From Micrographs (CPU) 15. 2D Classification (50 Class) 16. 2D Classification (100 Class) (All Job Types Enabled) 17. 2D Classification (200 Class) (All Job Types Enabled) 18. Select 2D 19. Particle Sets Tools 20. Ab-Initio Reconstruction 21. Ab-Initio Reconstruction (3 Class) (All Job Types Enabled) 22. Heterogeneous Refinement (v5.0+) 23. Homogeneous Refinement 24. Non-Uniform Refinement 25. Subset Particles by Statistic 26. Volume Tools 27. Orientation Diagnostics 28. 3D Variability (3 Mode) 29. 3D Variability (6 Mode) (All Job Types Enabled) 30. Local Refinement Jobs run in Testing Mode (CryoSPARC v5.0+) 1. Import Movies 2. Import Micrographs 3. Import 3D Volumes 4. Import Particle Stack 5. Import Templates 6. Patch Motion Correction 7. Full-Frame Motion Correction 8. Import Beam Shift 9. Patch CTF Estimation 10. CTFFIND4 11. Check For Corrupt Micrographs 12. Generate Micrograph Thumbnails 13. Curate Exposures (Stream A) 14. Curate Exposures (Stream B) 15. Exposure Tools 16. Exposure Sets Tool (Split) 17. Exposure Sets Tool (Intersect) 18. Blob Picker (Stream A) 19. Blob Picker (Stream B) 20. Inspect Picks (Stream A) 21. Remove Duplicate Particles 22. Blob Picker Tuner 23. Inspect Picks (Stream B) 24. Reassign Particles to Micrographs 25. Patch CTF Extraction 26. Local Motion Correction (Stream B) 27. Local Motion Correction (Multi) (Stream B) 28. Extract From Micrographs (CPU) (Stream A) 29. Extract From Micrographs (GPU) 30. Micrograph Junk Detector 31. Restack Particles 32. 2D Classification (Stream A) # all jobs past this point are in Stream A 33. Cache Particles on SSD 34. Check For Corrupt Particles 35. Class Probability Filter 36. Downsample Particles 37. Select 2D 38. Average Power Spectra 39. Template Picker 40. Inspect Picks 41. Extract From Micrographs 42. Exposure Group Utilities (Combine Particles) 43. Exposure Group Utilities (Split Particles) 44. Exposure Group Utilities (Combine Exposures) 45. Exposure Group Utilities (Split Exposures) 46. 2D Classification (50 Class) 47. 2D Classification (100 Class) 48. 2D Classification (200 Class) 49. Rebalance 2D Classes 50. Select 2D 51. Particle Sets Tools 52. Reconstruct 2D Classes 53. Ab-Initio Reconstruction 54. Ab-Initio Reconstruction (3 Class) 55. Split Volumes Group 56. Simulate Data 57. Reference Based Auto Select 3D 58. Volume Tools 59. Create Templates (Non-Helical) 60. Create Templates (Helical) 61. Heterogeneous Refinement 62. Homogeneous Refinement 63. Non-Uniform Refinement 64. Subset Particles by Statistic 65. Volume Alignment Tools 66. Volume Tools (Mask) 67. Reference Based Auto Select 2D (Sobel) 68. Reference Based Auto Select 2D (Cluster) 69. Reference Based Auto Select 2D (Thresholds) 70. Rebalance Orientations 71. Orientation Diagnostics 72. Heterogenous Reconstruction Only 73. Local Resolution Estimation 74. Symmetry Search Utility 75. Apply Helical Symmetry 76. Reference Based Motion Correction 77. 3D Flex Data Prep 78. Patch Motion to Local Motion 79. 3D Classification 80. Sharpening Tools 81. Validation (FSC) 82. Global CTF Refinement 83. 3D Variability (3 mode) 84. 3D Variability (6 mode) 85. Local Refinement 86. Recenter Trajectories 87. Apply Trajectories 88. Local Filtering 89. 3D Flex Mesh Prep 90. Align 3D Maps 91. Regroup 3D Classes 92. Particle Subtraction 93. 3D Flex Training 94. Local CTF Refinement 95. ResLog Analysis 96. 3D Variability Display 97. 3D Flex Generator 98. 3D Flex Reconstruction \#### \[EMPIAR-10305\](https://www.ebi.ac.uk/empiar/EMPIAR-10305/) \* \*\*Number of movies:\*\* 62 \* \*\*Frames per movie:\*\* 20 \* \*\*Movie size:\*\* 7420 × 7676 (K2 Super Resolution) \* \*\*Pixel size:\*\* 0.32 Å \* \*\*Particles processed:\*\* \\~30,000 \* \*\*Particle box size (pixels):\*\* 512 Jobs run in Benchmark Mode (up to CryoSPARC v4.7) 1. Import Movies 2. Patch Motion Correction 3. Patch CTF Estimation 4. Curate Exposures 5. Blob Picker 6. Inspect Picks 7. Extract From Micrographs 8. 2D Classification 9. Select 2D 10. Template Picker 11. Inspect Picks 12. Extract From Micrographs 13. 2D Classification 14. Select 2D 15. Particle Sets Tools 16. Ab-Initio Reconstruction 17. Homogeneous Refinement 18. Non-Uniform Refinement 19. 3D Classification 20. 3D Variability Jobs run in Testing Mode (up to CryoSPARC v4.7) 1. Import Movies 2. Patch Motion Correction 3. Patch CTF Estimation 4. Filament Tracer 5. Inspect Picks 6. Extract From Micrographs 7. 2D Classification 8. Select 2D 9. Helical Refinement 10. Local CTF Refinement 11. Global CTF Refinement 12. Symmetry Expansion 13. Homogeneous Reconstruct Only Jobs run in Benchmark Mode (CryoSPARC v5.0+) 1. Import Movies 2. Patch Motion Correction 3. Patch CTF Estimation 4. Filament Tracer 5. Inspect Particle Picks 6. Extract From Micrographs (CPU) 7. 2D Classification 8. Select 2D Classes 9. Helical Refinement 10. Local CTF Refinement 11. Global CTF Refinement 12. Symmetry Expansion 13. Homogeneous Reconstruction Only Jobs run in Testing Mode (CryoSPARC v5.0+) 1. Import Movies 2. Patch Motion Correction 3. Patch CTF Estimation 4. Filament Tracer 5. Inspect Particle Picks 6. Extract From Micrographs (CPU) 7. 2D Classification 8. Select 2D Classes 9. Helical Refinement 10. Local CTF Refinement 11. Global CTF Refinement 12. Symmetry Expansion 13. Homogeneous Reconstruction Only \## Prerequisites {% content-ref url="/pages/-M7DHIJsaWhZ95BFAmfp" %} \[How to Download, Install and Configure\](/setup-configuration-and-management/how-to-download-install-and-configure.md) {% endcontent-ref %} ## Creating and Running the Extensive Validation Job 1. Open the CryoSPARC web interface 2. In the dashboard, create a new Project from the navigation bar and create an initial workspace. !\[\](/files/6cQbGcdlyv6FyhfdYpNs) Specify a descriptive title such as "Extensive Validation Testing" and directory for the project to store its data. ![](https://guide.cryosparc.com/files/zSxsPu5mmdW6W75Vb0W4) \*\*Best practices:\*\* Create a new workspace and run the Extensive Validation in that workspace each time CryoSPARC updates and restarts. Name each workspace with the latest installed version of CryoSPARC that the job runs on. For example, when testing CryoSPARC v4.3.0, name the workspace "v4.3.0 Benchmark and Validation" 4\\. Select the Job Builder from the sidebar and select the \*\*"Extensive Validation"\*\* job (under the Validation category). !\[\](/files/DlF3WPH3nzplWw6fE0bD) \*5. (Optional)\* If desired, change the job parameters. 6\\. Select the node or cluster that the Extensive Validation jobs should run on. !\[\](/files/GmpOPDLtRFToqmbFH5zX) !\[\](/files/34wWzotnPo7iPCKCiUXt) Queue the job from the Job Builder sidebar. Open the job's Event log (either click/tap the Job card header or select the Job card and press the \`Space\` key) to monitor its progress. The Validation job logs each spawned job as it is queued and logs how long it takes to complete. !\[\](/files/ZqXkgKgfcnHlcZXfy5pV) Close the Job modal with the \`×\` button. This shows the workspace overview with CryoSPARC jobs spawned by the Extensive Validation job Once all jobs successfully complete, the Extensive Validation job status changes to "Completed". This means the installation was successful. Users may now be notified to start or resume processing! ## Troubleshooting Failed Jobs Extensive Validation will fails if any spawned job fails. Scroll through the workspace to find jobs with the "Failed" status. Open the failed job's Event Log. Scroll to the bottom to see why the job failed. Common failure reasons include \* \[Cannot verify license\](https://discuss.cryosparc.com/t/new-install-error-connecting-to-cryosparc-license-server/3825) or license key entered incorrectly \* \[Incorrect filesystem permissions\](https://discuss.cryosparc.com/t/file-r-w-permissions/3544) \* \[CUDA not set up correctly\](https://discuss.cryosparc.com/t/3d-variability-analysis-errors-v2-9-0/3110) \* \[SSD cache full or not set up correctly\](https://discuss.cryosparc.com/t/changing-ssd-no-run-directory/2916) \* \[Worker registration issue\](https://discuss.cryosparc.com/t/exit-status-255-error/2177) \* \[Not enough GPU memory available\](https://discuss.cryosparc.com/t/patchmotion-failure-2-13-2/3971) Once the configuration issue is resolved, restart the Extensive Validation job: Either create a new workspace and job as already noted, or clear the existing Extensive Workflow job and re-queue. ## Additional Extensive Testing For an even \*more\* extensive system test, the Extensive Validation job provides the parameter "Run Advanced Jobs" !\[Turn on the "Run Advanced Jobs" radio button to run the full workflow.\](/files/A8E8n5os4f2PWp5AirHR) With "Run Advanced Jobs" enabled, additional validation jobs run in parallel. Use this to verify multi-GPU performance on a single node. Advanced jobs available for each dataset are listed in the "Datasets Available" section above. ## Expected Results To compare the results of Extensive Validation runs, use "Benchmark" mode. This locks in the parameters and runs each job serially to ensure all system resources are available independently. The benchmark results may be viewed in the "Manage" panel, under the "Benchmarks" tab. ![](https://guide.cryosparc.com/files/iiGdjfx5aoKewXAZkWJM) ![](https://guide.cryosparc.com/files/5pdcvX22UDUANSOtrjmM) There are several reference benchmarks available for comparison with your CryoSPARC installation, including benchmarks completed on AWS EC2 instances. Select one or more benchmark rows and click "Compare" to compare benchmarks. ![](https://guide.cryosparc.com/files/t03gEWdx3WoyFWEYFxBn) \--- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/tutorial-verify-cryosparc-installation-with-the-extensive-workflow-sysadmin-guide.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-homogeneous-refinement.md). # Job: Homogeneous Refinement ## At a Glance Perform a global alignment of input particles to a reference volume, filtered by the GSFSC resolution. ## Description Homogeneous Refinement improves the input volume by iteratively aligning particles to the volume, then using the new \[poses\](/processing-data/all-job-types-in-cryosparc/3d-refinement.md#aligning-images-to-a-volume) to improve the volume. For more information on this algorithm (called Expectation Maximization), see the \[Expectation Maximization in Cryo-EM\](https://guide.cryosparc.com/expectation-maximization-in-cryo-em) page. Homogeneous Refinement is a global refinement, meaning that prior pose estimates are neither used nor required. It uses half sets and its output volume is GSFSC-filtered. Homogeneous Refinement will generally produce a high-quality volume for most samples, but we encourage users to read the \[Recommended Alternatives\](#recommended-alternatives) section since other types of refinement are better suited to some goals or samples. ## Inputs ### Particles As Homogeneous Refinement is a global refinement, no prior pose information will be used or is needed. Particles must have CTF information in order for refinement to proceed. ### Initial Volume An initial volume is used the first iteration of a Homogeneous Refinement, since the images do not have 3D pose estimates and therefore cannot be used to create a reference volume. In subsequent iterations, the volume created by backprojection of the particles is used for alignments. The initial model will be low-pass filtered before alignment as specified by the \[\`Initial lowpass resolution\`\](#initial-lowpass-resolution-a) parameter. {% hint style="warning" %} The initial volume has a significant impact on the end result of a refinement job. A poor input volume (e.g., a noisy or anisotropic volume, or a volume that is too dissimilar from the target) will produce poor results. See \[Ab-Initio Reconstruction\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-reconstruction/job-ab-initio-reconstruction) for information on how CryoSPARC can generate initial volumes. {% endhint %} ### Mask (optional) If a mask is provided, at each iteration of refinement the volume will be masked using this mask instead of the \[dynamic masking\](#dynamic-masking) routine. This can be helpful if dynamic masking fails, or if unstructured regions (e.g., micelles) interfere with alignment. If the mask is being used to focus refinement on a particular region, Local Refinement may perform better — see \[Recommended Alternatives\](#recommended-alternatives). Note that the provided mask is only used for alignment. FSC is always calculated with a dynamic mask. You can calculate the FSC with your own mask using \[Validation (FSC)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-validation-fsc) once the refinement finishes. For masking behaviour in CryoSPARC v5+, see \[the dedicated guide page\](/processing-data/tutorials-and-case-studies/tutorial-dynamic-masking-in-refinements-v5.0.md). ## Commonly Adjusted Parameters ### Window dataset (real-space) !\[\](/files/alze8AczHB9Z1miakSge) In general, particle images are expected to be well-centered. This means that (ignoring signal delocalization due to the Contrast Transfer Function) the edges and corners of the particle image do not contain information about the particle. They may, however, contain noise or adjacent particles which interfere with alignment. Refinement algorithms therefore typically window these particle images before comparing them to reference projections. The window gently transitions from 1.0 at the \`Window inner radius\` to 0.0 at the \`Window outer radius\`. In very crowded grids, it may help to use a tighter window (i.e., reduce both radii) to exclude neighboring particles. Note that windowing is performed \*before\* the images are centered, so if particle images are not well centered windowing may remove particle information. ### Symmetry The symmetry operator entered here is used to enforce (or relax) symmetry during refinement. By default it is C1 (i.e., no symmetry). Allowed symmetry operators are: ![](https://guide.cryosparc.com/files/AalNZnMb3LyeDNzKOZxB) Any cyclic group, e.g., C4. Cyclic groups have N-fold symmetry around a single axis. By convention in CryoSPARC this axis is aligned to the Z-axis. ![](https://guide.cryosparc.com/files/NK9iRu6SNQjfziWWRukL) Any dihedral operator, e.g. D5. Dihedral groups have the following properties: \* N-fold symmetry around one axis. By convention in CryoSPARC this axis is aligned to the Z-axis. \* 2-fold symmetry about one axis. By convention in CryoSPARC, this axis is aligned to the Y-axis. \* Symmetry order of 2N ![](https://guide.cryosparc.com/files/CyjMVXrdzzstrNPNj0Ps) The tetrahedral group T, which has the following properties: \* 3-fold symmetry around four axes. By convention in CryoSPARC one of these 3-fold symmetry axes is aligned with the Z-axis. \* Symmetry order of 12 ![](https://guide.cryosparc.com/files/7VzYUNdpNh4Rt9kruG82) The octahedral group O, with the following properties: \* 4-fold symmetry along three orthogonal axes. By convention in CryoSPARC these are aligned to the X, Y, and Z axes. \* Symmetry order of 24 ![](https://guide.cryosparc.com/files/aKVOegr2l67HzjZhfXJE) The icosahedral group I with the following properties: \* Six 5-fold axes \* Symmetry order of 60 \* CryoSPARC defines two icosahedral conventions: \* \`I1\` (or just \`I\`): 2-fold axes on X, Y, Z; the 5-fold axis point with greatest Z value in the YZ plane; the 3-fold axes with greatest Z value in the XZ plane \* \`I2\`: 2-fold axes on X, Y, Z; the 5-fold axis point with greatest Z in the XZ plane; the 3-fold axes with greatest Z value in the YZ plane ### Symmetry relaxation method This parameter can take one of three possible values: “none”, “maximization”, or “marginalization”. For more information on symmetry relaxation, see the \[symmetry relaxation tutorial\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation). #### None \*\*The point group is used to enforce symmetry\*\*. Each particle is inserted in each of the N different symmetry-related poses, where N is the symmetry order. This effectively increases the signal to noise ratio by a factor of N, but can produce invalid maps or other poor results if the target is not truly symmetric. #### Maximization Each particle is aligned to the reference as in an asymmetric reconstruction, but a small neighborhood each of the N symmetry-related poses is then checked. \*\*Only the best of all symmetry-related poses is used.\*\* Note that this means particle images are only used once — the map is not forced to be symmetric. #### Marginalization Each particle is aligned to the reference as in an asymmetric reconstruction, but a small neighborhood of each of the N symmetry-related poses is then checked. \*\*Particles contribute to the reconstruction in each symmetry-related pose, weighted by the probability of that pose being correct\*\*. Note that this means particle images are only used once — the map is not forced to be symmetric. ### Do symmetry alignment !\[\](/files/QVcgL5idVCiP27ANitKO) If this parameter is on, the input volume is transformed and shifted such that the symmetry axes are aligned to map axes (e.g., the four-fold axis of a C4 symmetric input map is aligned to the Z axis). ### Re-estimate greyscale level of input reference !\[\](/files/lu1XVUQHLWDeieFMUtT7) Cryo-EM maps comprise a grid of voxels, with each voxel containing some value which is related to the Coulomb potential of the target at that position. However, these values only provide information about the \*relative\* potential within a single map, not the absolute potential of the target. In general, maps created from different sets of images will not have the same values in the same voxels. The range of values across all voxels is called the \*greyscale\*. Since alignments are scored by assessing the difference between each image and the volume, a difference in greyscale leads to poor alignments. If this parameter is on, the greyscale of the input map will be adjusted to match those of the input particles. In general, we recommend that this parameter is on for Homogeneous Refinements. This parameter ensures that the volume starts near the mean particle’s greyscale. Each particle will have slightly different contrast due to ice thickness, beam effects, etc. These per-particle differences in scale are fit by \[Per-Particle Scale\](#minimize-over-per-particle-scale). ### Number of extra final passes This many EM iterations will be performed with the full particle stack after the GSFSC resolution stops improving. By default, refinement is considered complete after the first iteration in which the GSFSC does not improve. In most cases, this is sufficient. However, the GSFSC is only one measure of map quality. In some cases, continuing refinement after GSFSC resolution stops improving can still result in an overall higher-quality map. The most common situation in which extra final passes improves the final result is symmetry relaxation. As of yet, there is not a good automated metric by which the refinement can validate whether the symmetry-relaxed poses of the particles have converged. As such, terminating the refinement upon GSFSC convergence may prevent the algorithm from sufficiently breaking pseudosymmetry. For example, consider data from EMPIAR-10256 (Dang et al. 2019). !\[\](/files/tiBahRhqJjUa2yTsGeCN) GSFSC stops improving after the second iteration. However, signal from the symmetry-breaking CaM molecule is not fully resolved until iteration 32. ### Adaptive Marginalization When this parameter is turned on, particle poses will be marginalized, meaning that each particle image contributes to the 3D reconstruction from multiple poses, each weighted by their probability of being correct. Marginalization can improve the results of refinement, with small particles or low-SNR images benefiting the most. For medium and large particles or high-SNR images, maximization (\`Adaptive Marginalization\` off, the default) works just as well and is computationally less expensive. ### Maximum align resolution (A) During alignment (not reconstruction) the map uses frequencies only up to this resolution. If left blank, the map uses all frequencies up to the current resolution. Keep in mind that in both cases the map is also filtered by the GSFSC curve, so in practice maps may use coarser resolutions than this parameter dictates. Setting this parameter to a higher numeric value (lower resolution) may reduce overfitting due to high-frequency noise for some datasets. Note that much of the \*alignable information\* in an individual particle image comes from the low frequencies. Thus, the \[\*reconstruction\*\](/processing-data/all-job-types-in-cryosparc/3d-refinement.md#backprojection) may achieve a higher resolution than that of the alignment limit. For example, consider data from EMPIAR-10424 (Nakane et al. 2020). !\[\](/files/hXDiFkWODsczngww1O8m) The map produced with this parameter is left empty (left) is of slightly higher quality (for example, the indicated histidine appears slightly more isotropic), but both maps achieve better than 1.5 Å resolution. In addition to potentially preventing overfitting, setting the Maximum align resolution to a higher numeric value (lower resolution) may help symmetry relaxation converge if the asymmetric feature is large and the reconstruction goes to high resolution. !\[\](/files/oK206Re7txSbr0rBsDzE) In this example again using data from EMPIAR-10256 (Dang et al. 2019), setting the maximum alignment resolution to 6 Å provided a significantly improved breaking of the C4 psuedosymmetry after the same number of iterations. Note that when using the GSFSC resolution (blue, left) significant density remains in all four positions (indicated with arrows). When limiting alignment to 6 Å, pseudosymmetry is successfully broken. Both maps are lowpass filtered to 6 Å to aid comparison. ### Initial lowpass resolution (A) Before the first iteration, the input volume is lowpass filtered to this resolution in Å. Typically, the default value of 20 Å does not need to be changed. For highly symmetric or very small particles, a finer resolution may improve results. ### GSFSC split resolution (A) \[Half sets\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#fourier-shell-correlation-plots) share information during refinement, up to this resolution. Put another way, half sets are only truly independent at frequencies finer than this value. If a refinement was run with two completely independent half maps, over iterations the two maps might adopt different orientations in 3D space. The correlation between two half maps in different orientations would be very low, meaning that the GSFSC resolution would be extremely poor even if the half maps were identical. !\[\](/files/EagfwWUvB7p40D47bF0u) To avoid this scenario, the components of each half map below the resolution specified by this parameter are averaged together in every iteration. This forces the half maps to adopt the same overall pose in 3D space, but retains their independence at higher resolutions. This parameter should almost always be left at its default setting of 20 Å. If the GSFSC resolution for a highly symmetrical particle is surprisingly poor and the particles generate good 2D classes, you should first download and inspect the half maps. If they are clearly in different orientations, setting this parameter to a higher resolution may help. Keep in mind that the half-sets are not independent at resolutions coarser than this parameter, so it should be kept as coarse as possible. ### Auto batchsize For a typical dataset, the map used for alignment in the first iterations of a refinement is a poor estimate of the true, final volume. Poses aligned to this reference therefore also poor. It is thus wasteful to align \*every\* particle to these early volumes. CryoSPARC therefore, by default automatically estimates the number of particles (called a batch) to align before generating a new reference for following iterations. For small particles or particles with poor signal-to-noise ratio, larger batch sizes may be necessary for optimal reconstruction. The automatic estimate of the optimal batch size can be changed using two similar but distinct parameters, described below. In general, adjusting the batch size with \`Batchsize snrfactor\` is recommended, since the effect of changing it to a specific value is more predictable across datasets. \* \`Batchsize epsilon\` controls the estimated proportion of Fourier pixels which will be missed by the minibatch. Setting this value \*higher\* allows for \*fewer\* particles in the minibatch, while a \*lower\* value creates minibatches with \*more\* particles. Note that this parameter should always remain above 0. \* \`Batchsize snrfactor\` directly multiplies the batch size calculated using \`Batchsize epsilon\`. Setting this parameter higher by a factor of 2 (i.e., 100 instead of 50) doubles the number of particles in the minibatch. Auto batch sizing can be disabled entirely by turning \`Disable auto batchsize\` on. In this case, the entire particle stack is used in each iteration. In general, this significantly slows jobs without appreciable improvement in the final result. ### Minimize over per-particle scale !\[\](/files/1wsBSicdAXHgoilQWcy3) If this parameter is on, each particle’s optimal scale is calculated at each iteration. If this parameter is off, the particles’ input scales are used during each iteration. The per-particle scale is a value for \*each particle image\* which adjusts the contrast of the reference volume to the contrast in the individual particle image. For example, consider two particles produced by the same volume in the same pose, but in different ice thicknesses. The particle in thinner ice will have more contrast than the particle in thick ice, but the reference volume should have the same voxel values for both. Per-particle scale is used to adjust the greyscale of \*individual particle images\* to account for this fact. As the name implies, each particle has a scale value which relates its greyscale to that of the volume. While per-particle scale in theory corrects for each image’s greyscale, particles with a low per-particle scale tend to be poorer quality than particles with high per-particle scale. For this reason, it may be beneficial to filter out particles with low scale. See the \[Subset Particles by Statistic\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/job-subset-particles-by-statistic#subsetting-by-per-particle-scale) job page for more information on this process. ### Reset input per-particle scale If this parameter is on, all particles’ scale is set to \`1.0\` at the beginning of the refinement. If this parameter is off, particles’ scales are retained from the input. Note that if particles have not yet been refined, their starting scales are all \`1.0\`. If \`Minimize over per-particle scale\` is off and per-particle scales have previously been fit (in an earlier refinement, for example), you may wish to turn this parameter off to retain the previously-found scales. ### Initialize noise model from images If this parameter is on, the noise model is directly estimated from the images. If this parameter is off, a constant value is used to initialize the noise model. In theory, a noise model inferred directly from particle images may help when \`Adaptive marginalization\` is on, since marginalization tends to be more sensitive to the choice of noise model. In practice, the noise model typically converges during the first or second iteration, so this setting has little impact on the final result. ### Dynamic masking {% hint style="info" %} Starting in CryoSPARC v5, the dynamic mask near/far parameters are multiples of the current resolution instead of raw physical units. See \[3D Masking in Refinement\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-dynamic-masking-in-refinements-v5.0#dynamic-masking-in-cryosparc-v5.0) for more information. {% endhint %} If a static mask is provided to the Mask input, that mask will be applied at each iteration. If a mask is not provided and \`Use dynamic refinement mask\` is turned on, a mask will be dynamically generated by the following process. !\[\](/files/J5mAbdLrPR1Dw6uf3GJo) Once the GSFSC resolution is the same as or finer than \`Dynamic mask start resolution (A)\`, a mask will be generated by thresholding each half map at the \`Dynamic mask threshold (0-1)\`. If \`Dynamic mask use absolute value\` is turned on, this thresholding is performed on the absolute value of the map (i.e., a threshold of 0.2 would include voxels with value less than -0.2 or greater than 0.2). This is useful if there are regions of the map which are expected to be empty. Since empty pockets will have lower density than the corners of the image (which may have neighboring particles or contaminants), they will tend to have negative map values. However, these pockets are typically small and near regions of high density, so this parameter is rarely required in practice. The mask is then padded with \`1.0\` for a distance of \`Dynamic mask near (A)\` and a soft edge is added, reaching \`0.0\` at a distance of \`Dynamic mask far (A)\`. Dynamic masking can effectively be disabled by setting \`Dynamic mask start resolution (A)\` to an unrealistically low value, such as 0.1 Å. Starting in CryoSPARC v5, dynamic masking can be disabled by turning off \`Use dynamic refinement mask\`. {% hint style="info" %} Cryo-EM maps can have very different absolute voxel values. To account for this, the \`Dynamic mask threshold (0-1)\` parameter is a \*relative threshold\*. The map is thresholded at a voxel value of \`Dynamic mask threshold\` times the maximum voxel value in the map. For instance, consider a map with voxels ranging from \`-0.10\` to \`0.23\`. If \`Dynamic mask threshold (0-1)\` is set to \`0.5\`, all values greater than \`0.115\` are set to \`1.0\` and all values less than or equal to \`0.115\` are set to \`0.0\`. The mask is then dilated and padded using the \`Dynamic mask {near, far}\` parameters. {% endhint %} ### GPU batch size of images Reading images from the filesystem is slow. To speed up refinement, CryoSPARC will try to load as many images into the GPU at once as it can. However, it is challenging to precisely determine the space required by a given refinement, so the number of images that fits can only be estimated. If you run out of GPU memory during a refinement, you \*may\* be able to complete the refinement by manually setting this parameter to a low number of images. Note that GPU batch size is a purely computational consideration — it will not have an effect on the final result. It differs in this way from the batch size of the \*refinement\*, which controls the number of images seen in each iteration and is controlled by the \[\`Batchsize epsilon\` and \`Batchsize snrfactor\`\](#auto-batchsize) parameters. ### Defocus Refinement and Global CTF refinement CryoSPARC can estimate per-particle defocus and per-group higher-order CTF aberrations during a refinement. On-the-fly CTF estimation is controlled by \`Optimize per-particle defocus\` (for \[Local CTF Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-refinement/job-local-ctf-refinement)) and \`Optimize per-group CTF params\` (for \[Global CTF Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-refinement/job-global-ctf-refinement)). See the associated pages for information about the other CTF refinement parameters. We recommend first performing separate Local and Global CTF Refinements and assessing whether the datasets benefit from these optimizations before performing them on-the-fly during refinement. For more information, see the guide page on \[CTF refinement\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement). ### Do EWS correction Whether to correct for the curvature of the Ewald sphere. This typically produces moderate resolution improvements for large particles which are already at high resolution without Ewald sphere correction. If this option is turned on, ensure that the correct curvature is selected in \`EWS curvature sign\`. For more information on these parameters, see the \[Ewald Sphere Correction section of this page\](#ewald-sphere-correction), or the \[dedicated guide page\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ewald-sphere-correction). ## Outputs ### All particles Particles are output with updated poses. If CTF parameters were refined during the job, these output particles also have updated CTF estimates. Note that this may mean that exposure group parameters for these particles differ from those of the micrographs. If the particles are re-extracted, ensure that \`Force re-extract CTFs\` from micrographs is off (the default setting) to retain these refined CTF parameters. ### Refined volume The final volume produced by the refinement is output as \`map\`. It is filtered to the GSFSC resolution. Additionally, a sharpening B-factor is automatically estimated and applied to the volume to produce a sharp volume (\`map\_sharp\`). The B-factor is estimated using the \[Guinier plot\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#guinier-plot). Masks used for refinement and FSC calculation are available as \`mask\_refine\` and \`mask\_fsc\`, respectively. Typically, the mask used for FSC calculation is tighter than the mask used for refinement. Half maps are available as \`map\_half\_A\` and \`map\_half\_B\`. ### Masks In versions of CryoSPARC before v5, there is a single mask output, \`mask\`, which contains the refinement mask. It is the same as the \`mask\_refine\` part of the Refined volume output. Starting with CryoSPARC v5, each mask has its own output, typically \`mask\_refine\` for the mask used during refinement, \`mask\_fsc\` for the mask used to calculate the FSC, and \`mask\_fsc\_auto\` for the autotightened mask used to calculate the final FSC. See \[Dynamic Masking in Refinements (v5.0+)\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-dynamic-masking-in-refinements-v5.0) for more on mask generation in v5. ### Plots \[The plots produced by Homogeneous Refinement are explained in the Common CryoSPARC Plots\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots) guide page. ## Common Problems Refinement is the central process of single particle analysis. As such, it is difficult to provide an exhaustive overview of potential problems arising during refinements. However, a few pathologies are more common than others. ### Map has spikes or shells Spikes of density radiating away from the center of the map or shells of density surrounding the map are both signs of overfitting. Often, this means that there is still a significant amount of “junk” in the particle stack, and more particle curation is necessary. If you’re unfamiliar with techniques for particle curation, they are covered in detail in a \[case study in this guide\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-dktx-bound-trpv1-empiar-10059). ### Map has streaks or blurring along a single viewing direction !\[\](/files/RTYOSMqY7EXdGaMK5ewW) This effect is called anisotropy, and is a telltale sign of orientation bias, also known as preferred orientation. Typically, correcting this issue requires new data, but some cases of anisotropy can be corrected with careful particle picking. Maps with these issues usually have low cFAR scores, which is a measure of map anisotropy. More information about orientation bias is available in the \[Orientation Diagnostics tutorial\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-orientation-diagnostics) and \[a case study on EMPIAR 10096\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-picking-induced-orientation-bias-in-ha-trimer-empiar-10096-and-10097#re-picking-the-ha-dataset). ## Common Next Steps Typically a volume needs to be visually inspected to understand the results of a refinement. In most cases, an improvement of visible features in the 3D map and/or a reduction in noise is desirable. {% hint style="info" %} A better GSFSC resolution alone may not be indicative of a truly improved map — visual inspection is an important component of the single particle analysis pipeline. {% endhint %} If noise features are visible (see Common Problems), the input particle stack should be cleaned and a new refinement re-run. At this stage, \[Heterogeneous Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-heterogeneous-refinement) or \[Ab-Initio Reconstruction\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-reconstruction/job-ab-initio-reconstruction) are typically most suited to particle curation (rather than 2D methods). If one region of the map is high quality and others are blurry, a \[Local Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-new-local-refinement-beta) with a focused mask on the blurry region may be useful. Simultaneously, \[3D Classification\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-3d-classification) may be able to separate particles into separate conformations, which can then be refined independently. If the map looks symmetric (or if symmetry was imposed), performing a refinement with symmetry relaxation turned on may reveal some asymmetry hidden in the data. 3D Classification of \[Symmetry Expanded\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-symmetry-expansion) particles with a mask around a single asymmetric unit of the map can also help classify asymmetry. ## Recommended Alternatives \[Non-Uniform Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-non-uniform-refinement-new) may perform better than Homogeneous Refinement when the particles have unstructured density (e.g., a micelle or nanodisc) or large flexible regions. \[Local Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-new-local-refinement-beta) may perform better than Homogeneous Refinement when a particle has multiple domains which move relative to each other, Homogeneous Refinement tends to align the larger of the two. This process is discussed in more detail in the \[TRPV1 case study in this guide\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-dktx-bound-trpv1-empiar-10059#masked-refinement-of-the-ctd), as well as \[a workshop recording\](https://guide.cryosparc.com/processing-data/tutorial-videos#part-2-trpv1-and-a-standard-workflow) working on the same dataset. ## Ewald Sphere Correction The interaction of the electron beam with the sample on the grid is a complex process which is difficult to precisely understand and model. Instead, for nearly all cryo-EM methods, the input particle images are treated as projections of the volume corrupted by the CTF. This is called the \*projection approximation\*, and works well for almost all datasets. However, if a reconstruction of a large particle is going to high resolution, this approximation may degrade the quality of the final map. In these cases, the projection approximation must be corrected by turning on \`Do EWS correction\` (i.e., “Do Ewald sphere correction”). This name is due to the fact that the projection approximation is equivalent to considering a mathematical construction called the \*Ewald sphere\* as if it were flat. In cases where Ewald sphere correction may improve the map, there are actually two possible solutions: the sphere may have a positive curvature and a negative curvature. There is no way of knowing the curvature of the sphere ahead of time — separate reconstructions with positive and negative curvature must be run. If the two reconstructions produce similar GSFSC resolutions, the Ewald sphere is likely negligible for this dataset and can be kept off in future refinements. If one is better than the other, that curvature is correct for all future refinements of this data — in other words, each dataset only needs the Ewald sphere curvature determined one time. For example, consider this reconstruction of an Adeno-Associated Virus from EMPIAR 10202 (Tan et al. 2018), which goes to 1.87 Å without Ewald sphere correction: !\[A comparison of reconstructions of AAV particles with no EWS correction (left), negative curvature correction (center), and positive curvature correction (right). All three reconstructions are sharpened with a B-factor of -50 and had identical settings aside from EWS correction and curvature.\](/files/kyDHslrq5zfITeKFAQyF) A comparison of reconstructions of AAV particles with no EWS correction (left), negative curvature correction (center), and positive curvature correction (right). All three reconstructions are sharpened with a B-factor of -50 and had identical settings aside from EWS correction and curvature. In this case, correcting for the Ewald sphere does improve the reconstruction, both in terms of the GSFSC resolution (improved to 1.74 Å) and visible inspection of the resulting maps. It is also clear that, for this dataset, the Ewald sphere has a negative curvature — the positive curvature reconstruction is worse (GSFSC 1.97 Å) than when we do not correct for the Ewald sphere curvature at all. Compare this result to that of performing Ewald sphere correction on a GPCR from EMPIAR 10673 (Zhang et al. 2020): !\[A comparison of reconstructions of a GPCR with no EWS correction (left), negative curvature correction (center), and positive curvature correction (right). All three reconstructions are sharpened with a B-factor of -50 and had identical settings aside from EWS correction and curvature.\](/files/N42GojVu29ttDxaIEeGN) A comparison of reconstructions of a GPCR with no EWS correction (left), negative curvature correction (center), and positive curvature correction (right). All three reconstructions are sharpened with a B-factor of -50 and had identical settings aside from EWS correction and curvature. The initial, no Ewald sphere correction reconstruction goes to 2.28 Å. Correcting for the Ewald sphere with either curvature produces an essentially identical reconstruction — this particle is simply too small, and this map at too low a resolution, for the sphere’s curvature to have a noticeable effect on the reconstruction. ## References Dang, S. \*et al.\* Structural insight into TRPV5 channel function and modulation. \*Proceedings of the National Academy of Sciences\* \*\*116\*\*, 8869–8878 (2019). Nakane, T. \*et al.\* Single-particle cryo-EM at atomic resolution. \*Nature\* \*\*587\*\*, 152–156 (2020). Tan, Y. Z. \*et al.\* Sub-2 Å Ewald curvature corrected structure of an AAV2 capsid variant. \*Nature Communications\* \*\*9\*\*, 3628 (2018). Zhang, X. \*et al.\* Differential GLP-1R Binding and Activation by Peptide and Non-peptide Agonists. \*Molecular Cell\* \*\*80\*\*, 485-500.e7 (2020). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-homogeneous-refinement.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-orientation-diagnostics.md). # Job: Orientation Diagnostics Description Orientation Diagnostics is a job \*\*(new in CryoSPARC v4.4, updated in v4.5)\*\* that can aid in diagnosing the presence of preferred orientation. It includes and builds upon 3DFSC (Tan et al., 2017) and Fourier Sampling (Baldwin & Lyumkis, 2020). By default, Orientation Diagnostics reports the conical FSC area ratio, or \*\*cFAR\*\* (v4.4+) and \*\*Relative signal\*\* (v4.5+). cFAR values below 0.5 generally indicate the presence of preferred orientation. cFAR accounts for both the viewing direction distribution and the signal content present within each particle by quantifying the variance of directional half-map Fourier correlations across the viewing sphere. Relative signal captures FSC variation as a function of viewing direction. Regions of low relative signal can help identify missing views whose absence has a deleterious effect on map anisotropy. If particles are supplied, the job will also report the Sampling Compensation Factor or \*\*SCF\\\*\*\* (Baldwin & Lyumkis, 2020 — see below for more information about the significance of the star). SCF\\\* values below 0.81 generally indicate the presence of preferred orientation. SCF\\\* characterizes the sampling of Fourier shells by considering the viewing directions of all particles, without accounting for the signal content contained therein. A junk particle is given equal weight to a real particle. We provide a short summary of the two metrics below. Please see the detailed definitions at the end of this job guide or the \[\*\*Orientation Diagnostics tutorial\*\*\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-orientation-diagnostics) for more information. #### cFAR and SCF\\\* at a glance... | | Range | Anisotropy Threshold | A function of… | | --- | --- | --- | --- | | cFAR | \[0, 1\]
higher is better | 0.5 | alignments & particle images | | SCF\* | \[0, 1\]
higher is better | 0.81 | alignments only | | cFAR | SCF\\\* | Conclusion | | ---- | ----- | --------------------------------------------------------------- | | ↑ | ↑ | No orientation bias | | ↑ | ↓ | ? (inconclusive, perhaps a pathological alignment distribution) | | ↓ | ↑ | Anisotropy due to junk or other contaminants | | ↓ | ↓ | Anisotropy due to junk and/or preferred orientation | ## Input \* Volume or Volumes (all classes) \* \*\*New in v4.5:\*\* If Volumes (all classes) are supplied from an upstream classification job, orientation diagnostics will be computed for each class volume. Note that this input is a \[volumes group\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/split-volumes-group) input. \* \\\[Optional\] Particles \* If supplied, Fourier sampling (and its associated metric, SCF\\\*) will be computed along with other per-particle diagnostics \* \\\[Optional\] Mask \* If supplied, half-maps will be masked prior to the computation of conical FSCs {% hint style="info" %} \*\*NOTE:\*\* The cFSC plot produced during the final iteration of all refinement jobs uses the auto-tightened mask (mask\\\_fsc\\\_auto). This mask is a low-level output of all refinements and will be automatically used by Orientation Diagnostics: \* when connecting a refined volume group as an input to Orientation Diagnostics \* when using the 'Build Orientation Diagnostics' quick action To use a custom mask, connect the input mask group. Note that if a custom mask is connected, the results may differ from the output of an upstream refinement. {% endhint %} ## \*\*Common Parameters\*\* \*Conical FSC\* \* \*\*Number of Directions\*\* \* The conical FSC Area Ratio (cFAR) metric is relatively robust to number of conical axis directions. However, reducing this number can speed up the job for volumes with large box sizes. Increasing this number will create denser spherical plots. \*Sampling Compensation Factor\* \* \*\*Symmetry\*\* \* Particle viewing directions will be expanded to account for symmetry. This parameter should be set to the symmetry applied in the upstream refinement. ## Output \* 3DFSC volume \* A volume of radial cFSC curves interpolated at each 3D voxel location ## Common Next Steps \* Particle picking (to find missing views) ## More details about cFAR, SCF\\\*, and Relative Signal ### cFSC Weighted Area-under-Curve (wAuC) Given a conical Fourier shell correlation, $$C\\\_r (\\hat{v})$$, (where $$r$$ is the Fourier radius in wave number and $$\\hat{v}$$ is the conical axis), we define the wAuC as $$\\begin{align}\\text{wAuC}(\\hat{v}) &\\overset{\\Delta}{=} \\sum\\\_r w\\\_r C\\\_r(\\hat{v}) \\ &= \\sum\\\_r 4 \\pi r^2 C\\\_r(\\hat{v}). \\end{align}$$ In this weighted sum, the Fourier radius, $$r$$, ranges from the DC component to the radius at which $$C\\\_r(\\hat{v})$$ first crosses 0.143. Intuitively, for each Fourier radius $$r$$, the correlation between the half maps is multiplied by the surface area of the Fourier shell at $$r$$. The final result is proportional to the ‘mass’ of the cone in units of correlation. ### cFSC Area Ratio (cFAR) We define the cFSC as the ratio of the minimum to the maximum $$\\text{wAuC}(\\hat{v})$$ over $$\\hat{v}$$, or $$\\text{cFAR} \\overset{\\Delta}{=} \\frac{\\min\\\_{\\hat{v}} \\text{wAuC}(\\hat{v})}{\\max\\\_{\\hat{v}} \\text{wAuC}(\\hat{v})}.$$ ### tFSC Area Ratio (tFAR) {% hint style="info" %} New in CryoSPARC v5 {% endhint %} In some cases, especially with small membrane proteins or targets with unimodal or bimodal viewing direction distributions, the cFAR score can indicate that a map is very anisotropic when it is in fact usable. Often, these targets with a pathologically low cFAR score have a good tFAR score (the same area ratio as the cFAR but calculated with \[tFSCs\](#relative-signal-tfsc)). Starting in CryoSPARC v5.0, we began reporting the tFAR score for use in these targets. ### \*Sampling Compensation Factor (SCF\\\*)\* (Baldwin and Lyumkis, 2020) SCF\\\* measures orientation bias in particle viewing directions — it allows CryoSPARC to convert a potentially difficult-to-parse viewing distribution into a single metric. SCF\\\* is computed from the statistics of ‘Fourier sampling’. Given a single particle, Fourier sampling is a binary function that indicates which ‘slab’ (i.e., a stack of slices) of Fourier voxels are affected when the particle is used in back-projection. Summing this sampling over many particles allows one to see if large chunks of Fourier space are poorly sampled or missing completely. The degree to which this occurs is measured by the Sampling Compensation Factor—a single number that quantifies the extent to which anisotropic viewing direction distributions attenuate the global FSC value. Mathematical Definition (expandable section) To compute the SCF, we consider the sampling of Fourier bins at a particular radius, $$R$$. In total there are \\~$$2 \\pi R^2$$ unique bins in Fourier space. For each particle viewing direction $$\\hat{v}$$, we compute its associated sampling via the slab condition (Baldwin and Lyumkis, 2020): $$\\text{Sp}(\\hat{k}, \\hat{v}, R) \\overset{\\Delta}{=} \\boldsymbol{1} ( \\hat{k} \\cdot \\hat{v}\\leq \\frac{1}{2 R} ),$$ where $$\\hat{k}$$ is a unit vector that defines a Fourier voxel on the shell of radius $$R$$, and $$\\boldsymbol{1}(\\cdot)$$ is the indicator function. Intuitively, this function returns 1 for all Fourier voxels that belong to a ring of radius $$R$$, within a plane orthogonal to $$\\hat{v}$$ (and 0 otherwise). Next, we sum $$\\text{Sp}(\\hat{k}, \\hat{v}, R)$$ for all particle viewing directions to produce $$\\text{Sp}$$, a single set of values that indicate the number of times each bin was sampled. The SCF can then be computed as $$\\text{SCF} = \\left( <\\text{Sp}> <\\frac{1}{\\text{Sp}}> \\right)^{-1},$$ where $$\\frac{1}{\\text{SP}}$$ denotes element-wise reciprocals, and $$<\\cdot>$$ is the arithmetic mean. This value is positive and always less than or equal to 1. Higher numbers indicate more uniform sampling distributions. If there are zeros amongst the sampling set, the above will be not defined. To account for this potential we report $$\\text{SCF\\\*} = \\left(\\frac{<\\text{Sp}^\*>}{p} \\left(p <\\frac{1}{\\text{Sp}^\*}> + q \\right) \\right)^{-1},$$ where $$q$$ is the fraction of zero sampling bins, $$p = 1 - q$$, and $$\\text{Sp}^\*$$ are the non-zero sampling values\*.\* Note that if $$q=0$$, then $$\\text{SCF} = \\text{SCF\*}$$. Please refer to (Baldwin and Lyumkis, 2021) for more details. \### Relative Signal (tFSC) The conical sections used to compute cFSCs cannot be easily mapped to viewing directions. To map FSC values to viewing directions, we use a toroidal section. We define a toroidal section to be the volume swept out by a cone, whose axis is orthogonal to the viewing direction, as it spins about the viewing direction. A toroidal section contains the Fourier components that would be populated by a particle with the same viewing direction, dilated to account for some error (or 'wiggle') in the pose estimate. We set the toroidal half-angle such that its volume is approximately equal to a cone of the same half-angle. ![](https://guide.cryosparc.com/files/Mzk7dFZdPhUfOBegSTvK) Conical vs. toroidal sections and their relation to the Fourier Slice Theorem. Using these toroidal sections, we compute a set of FSC curves. This set then allows to define a number, which we call relative signal, for each viewing direction. Relative signal is the wAuC of each curve, normalized with respect to the maximum within the set. The curve with the greatest wAuC in the set has a relative signal of 1.0, while a theoretical curve that is 0 at every frequency would have a relative signal of 0.0. ![](https://guide.cryosparc.com/files/3sVHhLewkiR35bhlTmuU) FSC curves computed within a toroidal section. Each curve is coloured by its relative signal, which we define to be the wAuC of each curve, normalized with respect to the maximum. We visualize each curve as a single number on an azimuth-elevation chart, and a 3D viewing sphere that encompasses a low-pass-filtered 3D volume. Starting in CryoSPARC v5, the tFSC curves are plotted in the "All toroidal fscs" plot. ## Plot Explanations v4.4 only plots #### cFSC wAuC vs. conical axis The wAuC of a cFSC curve is a proxy for directional signal content. If wAuC is relatively constant when the conical axis is varied, then the signal is isotropic in viewing direction. This plot helps illustrate the variation in cFSC wAuC and can aid in diagnosing the ‘structure’ of anisotropy. #### cFSC resolution vs. conical axis This plot is similar to the plot above, but visualizes the 0.143 crossing, rather than wAuC, of each cFSC curve. #### cFSC summaries within azimuth/elevation regions To add back in a coarse notion of directionality to the above plot, we reproduce the same statistics for twelve different regions of axis space. This plot can help identify differences in cFSC variance to further elucidate the source of anisotropy. \### Summary of cFSC curves We summarize all cFSC curves in this plot, visualizing statistics as a function of spatial frequency rather than conical axis. As of v4.5, this plot also displays the cFAR score. ![](https://guide.cryosparc.com/files/4gI5eNYBhqhg34q9ffx3) In blue: statistics over cFSC curves: mean, min, max, +/- one standard deviation plotted against spatial frequency. In green: histogram over 0.143 crossings of the same curves. \*\*Starting in v5.0\*\* Orientation Diagnostics also produces a plot of all cFSC curves, colored by their relative \[wAuC\](#cfsc-weighted-area-under-curve-wauc). This may be helpful for understanding the distribution of cFSC curves in some cases. ![](https://guide.cryosparc.com/files/XbjF4nWRnr2R92T17eWX) \*\*Also starting in v5.0\*\*, raw cFSC curves are also written to a \`csfsc.csv\` file in the job's directory (it cannot be downloaded through the GUI). In this file, the first column \`wave\_number\` gives the wave number for the resolution shell in that row. The resolution of that row is therefore $$ \\mathrm{Resolution} = \\mathrm{\\frac{box\\\_size \\times pixel\\\_size}{wave\\\_number}} $$ All columns other than the first contain cFSC value for a single cone at that row's resolution. The column headers are a 3D vector identifying the cone by the vector pointing from the origin to the center of the cone's base. In general, the exact values of these numbers are not expected to be important, rather just that they uniquely identify a given cone. ### Relative signal by viewing direction Relative signal visualized in a 2D azimuth-elevation chart (left), and in a 3D coloured scatter plot (right) with a low-pass-filtered volume embedded within. Low relative signal (i.e., darker colours) indicates a region with under-represented views. ![](https://guide.cryosparc.com/files/CQexFVLp1T1qxfRgGBVP) Relative signal vs. viewing direction (left: azimuth/elevation, right: 3D coloured scatter plot with structure embedded inside). \*\*Starting in v5.0\*\* Orientation Diagnostics plots all of the tFSC plots. This plot also reports the \[tFAR score\](#tfsc-area-ratio-tfar). ![](https://guide.cryosparc.com/files/Ecya2zLleuKYMKbNOd3O) \### Relative signal in azimuth / elevation regions \*\*New in v4.5.\*\* Relative signal within twelve regions of the viewing sphere, defined by different limits on azimuth and elevation. For each region, we show the projection of the structure from the central viewing direction, as well as the mean relative signal withing that region. Regions with low relative signal represent missing or underrepresented views in the data. ![](https://guide.cryosparc.com/files/XEynwQmYqoKLqGX6QH3w) Relative signal within 12 azimuth/elevation regions. \### 3DFSC volume The 3DFSC volume (Tan et. al, 2017) is another way to summarize cFSC curves by storing them in a volume whose voxels are interpolated from cFSC values at the nearest conical axis. In this plot, we visualize the 3DFSC volume via central slices. ![](https://guide.cryosparc.com/files/1LFEYFCRScMmHArXRitS) Central slices of the 3DFSC volume (Tan et. al, 2017) composed of interpolated cFSC values. \### Fourier Sampling {% hint style="info" %} The Fourier Sampling plot is only generated if particles are connected. {% endhint %} These plots visualize the Fourier sampling accumulated over a random subset of the particle viewing directions (default: 10000) — Fourier sampling is anti-podally symmetric (and hence only has \\~ $$2 \\pi R^2$$ bins) so we visualize only the $$z>0$$ hemisphere, following the original publication (Baldwin, P. R., & Lyumkis, D., 2020). N.B., the elevation / azimuth plot on the left should be visually similar to the Posterior Precision plot as posterior precision measures Fourier sampling modulated by the CTF. ![](https://guide.cryosparc.com/files/C9xfauyMeEL68TX5b8MX) The Fourier sampling \\text{Sp} (see equation in mathematical definition) visualized in 3D (right) and via an azimuth / elevation parameterization of the hemisphere (left). \### Particle scale factor vs. viewing direction {% hint style="info" %} The scale factor plot is only generated if particles are connected and their per-particle scales are not all 1.0. {% endhint %} This figure visualizes the average particle scale for a set of viewing directions (uniformly sampled on the viewing sphere). ![](https://guide.cryosparc.com/files/WClh8LvoH2tZ0cTsVOV1) (top) particle scale factors visualized by colour as function of particle viewing direction. (bottom) particle scale factor histogram (reproduced in other refinement jobs). \## References Tan et al. (2017), Addressing preferred specimen orientation in single-particle cryo-EM through tilting. \*Nat Methods\* 14(8), 793-796. Baldwin, P. R., & Lyumkis, D. (2020). Non-uniformity of projection distributions attenuates resolution in Cryo-EM. \*Progress in biophysics and molecular biology\* 150, 160-183. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-orientation-diagnostics.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera.md). # Tutorial: Mask Creation ## What Is a Mask? In many circumstances it is important to specify a region of a 3D volume. Whether this is to indicate a sub-volume to refine in Local Refinement, a volume to subtract for Particle Subtraction, or simply the region in which to calculate the GSFSC, we use a mask to select the volume. A mask is another 3D volume with the same box and pixel size as the volume to which it will be applied. The mask has a value of \`0.0\` outside the region to select and a value of \`1.0\` inside. When the volume is multiplied by the mask, the result will be a box that is empty except for in the region of interest, which will have the same information as the original volume. ![](https://guide.cryosparc.com/files/EnTOQDTpCUcqdksEVuYr) To mask a region of a volume, the original volume is multiplied by a separate volume (the mask) which contains 0.0 outside the region of interest and 1.0 within the region of interest. In \[almost all cases,\](#user-content-fn-1)\[^1\] masks for cryoEM must be “softened” by adding a smooth transition between the \`1.0\` values inside the mask and the \`0.0\` values outside the mask. This softening prevents \[ringing artifacts\](#why-do-masks-need-a-soft-edge) which severely degrade alignment. A softer mask introduces less severe artifacts, but includes more information from outside the region of interest. The \[Volume Tools\](/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools.md) job can be used to create a binarized (\`1.0\` inside and \`0.0\` outside) mask with a soft edge all in one step from an input map or mask. The ideal softness for a given dataset and subvolume typically must be determined empirically, but we recommend a minimum soft padding width of $$5 \\times \\frac{\\mathrm{resolution}}{\\mathrm{apix}}$$ where \*\*resolution\*\* is the GSFSC resolution in Å and \*\*apix\*\* is the pixel size in Å. In some cases, it can be beneficial to add a wider soft edge or even an expansion of the 1.0 values (i.e., making the mask “wider”, called dilation). It is often best to try a few different combinations of dilation and padding and determine which produces the best results. Why do masks need a soft edge? When we view particles, we view them in the spatial domain. However, particles are aligned in the frequency domain. To convert an image to its frequency domain representation we must take the 2D Fourier Transform. This is where the difficulty with hard-edged masks arises. A sharp transition in the spatial domain (the “normal” way of looking at images) becomes an infinitely fluctuating wave in the frequency domain. This wave artifact is known as “ringing". When we align images with ringing artifacts, we run the risk of lining up the artifact from our sharp mask instead of the information from the image itself. !\[A comparison of various wave shapes in the spatial and frequency domains. As the spatial wave gets a softer edge, ringing artifacts in the frequency domain disappear.\](/files/EiAlFVgYACfZ86AO2gBR) The softer the edge is in the spatial domain, the less ringing we observe. However, the softer the mask’s edge, the more of the volume outside our region of interest we include. \## Common Pitfalls ### Mask too tight One very important thing to consider is that masks which are too tight or too high-resolution introduce shared information into both half-maps, which breaks independence between the two half sets and artificially inflates the GSFSC curve. The magnitude of this effect is directly related to how much “ringing” is present in the mask’s frequency domain; thus sharper masks produce more severe artificial correlations. In jobs which produce GSFSC curves, be wary of results in which the Tight FSC curve does not closely follow the Corrected curve. A mismatch between these curves typically indicates an over-tight mask. ![](https://guide.cryosparc.com/files/JQP9M9tOOM1MKHRAMFXD) As a mask gets tighter and tighter, it introduces more and more information to the volume itself. Because the same mask is applied to each half-set, the FSC curve is affected. This results in the tight and corrected FSC curves not following each other --- a telltale signal of a too-tight mask. \### Mask too small One common application of masks is \[Local Refinement\](/processing-data/all-job-types-in-cryosparc/local-refinement/job-new-local-refinement-beta.md). Local Refinement is a powerful technique for improving the quality of smaller sub-volumes of a map. However, when the masked sub-volume is too small, not enough information remains for reliable image alignment. This can result in overfitting, characterized by noise, shells, or “blips”, especially surrounding the edge of the mask. If a small subdomain must be masked out for Local Refinement, we recommend applying a Gaussian prior to reduce overfitting. More information on Gaussian Priors is available on the job page. ## Mask Base Creation {% embed url="" %} A video version of this section of the tutorial. The same three techniques are covered in the article and the video. {% endembed %} The first step in all mask creation workflows is the creation of a mask base. We focus here on the three most common workflows: \* erasing regions of an existing map \* using volume segmentation on an existing map \* creating a mask using a molecular model For all three techniques we will use ChimeraX, a molecular visualization tool developed by the Resource for Biocomputing, Visualization, and Informatics at UCSF. ChimeraX is free for academic, government, nonprofit, and personal use and can be licensed for commercial use. The visuals for mask creation use data from EMPIAR-10073. This dataset was originally collected and processed by \[Nguyen et al\](#references). {% hint style="info" %} \*\*Mask Bases\*\* Throughout this page we make a distinction between mask bases and masks. \*\*Mask bases\*\* are created by a user to specify the region the mask ought to cover. They may have a hard edge and may or may not be binarized. \*\*Masks\*\*, on the other hand, are ready for use in CryoSPARC. They have been binarized, dilated, and padded. Typically a \*\*mask base\*\* is plugged into a Volume Tools job to create a \*\*mask\*\*. {% endhint %} ### Method One: volume segmentation This technique uses watershed segmentation to split a volume into regions. Masks are specified by deleting regions outside the desired volume. This technique remains relatively simple while also allowing for the construction of complex masks and is the method we recommend for most purposes. #### Step 1: Open and blur the input volume Blurring a volume before creating your mask achieves two aims. Practically, it is significantly easier to select a region of interest when high-frequency noise has been attenuated with a blurring operation. Theoretically, it is important that the mask does not introduce high-frequency correlations between the two half maps. Only building blurred masks helps reduce the chance of this happening. 1. Open the volume in ChimeraX. This example uses the results of a non-uniform refinement. For all commands in this example, \*\*the base volume is volume #1\*\*. 2. Blur the volume using a Gaussian filter: \`volume gaussian #1 sDev 2\`. Increasing the value for \`sDev\` will make the map more blurry. This command creates a new, blurred volume. \*\*The blurred base volume is volume #2\*\*. ![](https://lh7-us.googleusercontent.com/ioiz_IVRSTJfYWZsw8MT2BaBgT8iWS-dBZG26ARd5PVMnsd5JZw2ytCih5jDpLOTTHwyMiQJA1ImVZW5R4BBsziT2eE-_vvBkjhiLqV3L0EfKFwX-RSZX-yfAlbwUUVm709utiOoT9aBN0RkBUMMN3I) Comparison of a map before (left) and after (right) application of a Gaussian blur with standard deviation 2. \#### Step 2: Segment the volume In this step, the \[Segger\](#references) tool in ChimeraX splits the blurred volume into several regions. 1. Contour the blurred map until no noise is visible and the region which will ultimately form the mask has the desired topology. 2. Open Segger via clicking Tools > Volume Data > Segment Map ![](https://lh7-us.googleusercontent.com/y6bexV_y_jnVGzpeOZiOaG3T5uFLjXctdZq_cjj5B3FVMseVYPZclvgDBaJBYpSvhCvd2h45RLT2r9_MZ3roaeTsLUdoMaFhgSH9MWmVHvuwrDemh70f7S9KSQBMT4nIVK853or50YL1O-VYwxtj39w) The Segment Map tool can be found in Tools > Volume Data > Segment Map. 3\. Click "Segment" in the Segment Map pane to produce the segmentation. In this case, \*\*the segmentation is model #3\*\*. This segments the map into several regions, each of which is a distinct color. ![](https://lh7-us.googleusercontent.com/fS9krLQcxcZ-7psNeC67R1vtjwYhjS-W1yqAwyHSKgMmbWoS9kbaRTfIqcwoKoSM4dQUYnKkVxCJ_Yf3t-WPu7hhWyJn1trjdWBONmcifLnYrqm9N3XKeOkdR2UsqFK3PL3WNLqCuxlQ576kMUhrnkk) A segmented volume. Each region is a distinct color. \#### Step 3: Hide unwanted regions In this step, we build our mask around the region of interest by hiding regions we wish to exclude from the mask. Because \*\*there is no undo operation\*\* when working with segmented maps, we recommend that users only hide the regions rather than outright deleting them. This also makes it easier to generate a complementary volume for Particle Subtraction, which we will do later on. 1. Open the "Shortcuts Options" dropdown in the Segment Map pane. ![](https://lh7-us.googleusercontent.com/Syf3AyM4mIxtih3OyI-qXUrk4II9qIe1toywPuTudMG9o-U8W64UsTwx_K9ofNZp2xFuxOda1qU4_N2vMtOgqi1xiRHDsVemxMPtnXF1rbixux9uFD5TBiq_ZoTSiWKfwNyXFeaG368uH7A_GwD66G8) The Segment Map pane with the Shortcuts Options panel open. 2\. \*\*Control-click\*\* a region you wish to exclude from the mask to select it. The region should be surrounded by a light-green outline. ![](https://lh7-us.googleusercontent.com/_EOzTnxzFplTmjomhNyVBE-07MOu6jFwT5ZhXBilDz7Fre1DbbxYxQts165MT2eabKN4AbYvAViaXHBI13EHitnOfUiiOmuVZFkGKIK16jk15uqoHEJq1NOKi5QALT4CkoXGZJiY9qMGfja-zhqQN8w) The grey region in the top-left of the map is selected. 3\. Click the Hide button in Shortcuts Options to hide the region. The hidden region remains selected until a new region is clicked, so clicking “Show” will return the region to a visible state. 4. Proceed to hide all regions which are to be excluded from the mask. \* \*\*Control-click-and-drag\*\* selects multiple regions in a box \* \*\*Control-shift-click\*\* adds or removes a region from the current selection \* If a region contains parts of the map you wish to keep and parts you wish to exclude, select only that region and click “Ungroup” in Shortcuts Options. This will break the region into smaller subregions. Repeat the process until you can isolate the regions you wish to exclude. When this process is completed, the segmentation model should have all regions outside your desired mask hidden and all regions in the mask shown. ![](https://lh7-us.googleusercontent.com/zq-mG3LVmAHkKhP3_KaJtR2TFYMhRpjDOhbat3xbajM6lM9oSmPQT7QPtRU-G6yXZ3X9RiFbpDogUDrTGnV4iBj5rfCKPHZLetxzOxTUjK1kdfU_eclCKdvhzC6wobceh78NqrCqJxWueabIm1vbuA4) A mask base segmentation for the tri-snRNP foot domain. \#### Step 4: Create the mask base CryoSPARC cannot accept Segger segmentations, so they must first be converted to \`.mrc\` format. 1. Control-click and drag over the entire segmentation to select all visible regions. 2. In the Segment Map pane, click File > Save \*\*selected\*\* regions to .mrc file. The filename you choose at this stage is not important, as the resulting .mrc file has some issues we will fix in the next step. ![](https://lh7-us.googleusercontent.com/QQJ_2Ev4QNmc5UtMMWt8rjDsvz_i8kv2je4848vSZKpLPTql8_JJDE9vRf3JUwnP7vDK5ArrSc4drSm0PpQJbBFurQCta_GFg4llIIw6wuyDmeVtRGOSLgvtasFqRHy9Hp3Mw19BHfPQNX2JA_Dzaa4) The segmentation can be converted to a .mrc file using File > Save selected regions to .mrc file in the Segger pane. This step generates a new volume, the mask base. In this example, \*\*the mask base is volume #4\*\*. #### Step 5 (optional): Generate the complementary mask base for Particle Subtraction If you are performing Local Refinement, it can be helpful at this stage to create the complementary mask for Particle Subtraction. To do this, we delete all the regions we used to create the mask base, then save another \`.mrc\` file with the remaining regions. 1. With the regions used to create the mask base selected, click “Delete” in the Segment Map pane. 2. In the Segment Map pane, click “All” next to “Show regions:”. This reveals the regions you hid during Step 3. ![](https://lh7-us.googleusercontent.com/gUe5b5EN23seUoFZEi5pggTQuzXt7BcKKXDgHcVKeCzZg5wyzAn6ycIjqMybquGeRNR-1Dfrt0Zt_zSJcSCVQbPXko4dBXEpcKts2O17SDOHN66hdQVIcy6cCSyDpqtEGAGeQbk4BS_6b2omKs4egws) The complementary mask to be used for Particle Subtraction. 3\. Select all regions and save an .mrc file as in Step 4. In this example, \*\*the particle subtraction mask base is volume #5\*\*. #### Step 6: Fix box size When Segger saves the regions to an \`.mrc\` file, it crops the box size to perfectly fit the mask base. This results in a box size that is different from the map’s box size, meaning CryoSPARC will not know where to position the mask. To resolve this problem, we must first resample the mask base onto the original map’s box. {% hint style="info" %} When using commands in ChimeraX, ensure that you are using the correct numbers for your maps, as they may differ from those printed here if you did not take the optional step 5, or your ChimeraX session already had maps or models loaded into it prior to starting this tutorial. {% endhint %} 1. Use the command \`volume resample #4 onGrid #1\` to resample the mask base onto the original map’s box. Note that the resulting maps are positioned in the same region of space and contain the same information, but the box sizes are different. In this case, the volume saved by Segger has a box size of \`96 x 94 x 129\` voxels, while the map and resampled volumes both have box sizes of \`380 x 380 x 380\`. ![](https://lh7-us.googleusercontent.com/1w9x878QutRJ8ySLoOckbErUNRJieidj_cUl18__j4eOJDeWgV5sdTj3A1IrjMc8Be1aFU2vv6erwTmKXff7dE16_qMaHhErpkU7-98197E4lPVSxLi7ldxDX4-TneABaL205tPj4PszUTq23OS8PaI) The resampled volume (cyan) and the volume created by Segger (red) have the same topology but different box sizes. 2\. Save the \*\*resampled volume\*\* (in the case of this example, volume #5 if a particle subtraction volume was not created and volume #6 if a mask subtraction volume was created). This is the mask base, so we recommend using an informative name. 3. \*(If creating a particle subtraction mask)\* repeat the above steps to resample the Particle Subtraction mask base onto the map volume. In this case, the necessary command would be \`volume resample #5 onGrid #1\`. This is the completed mask base for Particle Subtraction. #### Step 7: Upload to CryoSPARC Both of the mask bases are now resampled and ready for import to CryoSPARC, via the Import 3D Volumes job. Once imported, the mask bases can be converted to masks via thresholding, dilation, and padding in the Volume Tools job. ### Method Two: volume eraser For very simple masks, this technique is much faster than Method One. However, it can be susceptible to creating undesired noise and care must be taken when analyzing the resulting masks. #### Step 1: Open and blur the input volume Blurring a volume before creating your mask achieves two aims. Practically, it is significantly easier to select a region of interest when high-frequency noise has been attenuated with a blurring operation. Theoretically, it is important that the mask does not introduce high-frequency correlations between the two half maps. Only building blurred masks helps reduce the chance of this happening. 1. Open the volume in ChimeraX. This example uses the results of a non-uniform refinement. For all commands in this example, \*\*the base volume is volume #1\*\*. 2. Blur the volume using a Gaussian filter: \`volume gaussian #1 sDev 2\`. Increasing the value for \`sDev\` will make the map more blurry. This command creates a new, blurred volume. \*\*The blurred base volume is volume #2\*\*. ![](https://lh7-us.googleusercontent.com/ioiz_IVRSTJfYWZsw8MT2BaBgT8iWS-dBZG26ARd5PVMnsd5JZw2ytCih5jDpLOTTHwyMiQJA1ImVZW5R4BBsziT2eE-_vvBkjhiLqV3L0EfKFwX-RSZX-yfAlbwUUVm709utiOoT9aBN0RkBUMMN3I) Comparison of a map before (left) and after (right) application of a Gaussian blur with standard deviation 2. \#### Step 2: Create copies of the blurred volume Since the volume eraser tool directly modifies the volume it operates on, you must create another copy of the blurred volume if you plan on creating a mask for Particle subtraction. 1. Copy the map with \`volume copy #2\` #### Step 3: Erase the region outside the desired mask In this step, we will erase the regions outside the mask using the Volume Eraser tool. This tool creates a sphere and allows you to erase everything either inside or outside the sphere. There is \*\*no undo function\*\* for this tool, so be careful when erasing volumes. You can create copies of the volume as you go if the process is long and complicated using \`volume copy\` as above. 1. Open the volume eraser tool: Right Mouse ribbon menu > Erase. You should see a sphere appear. Holding down right-click and dragging your mouse moves the sphere. Erasing inside or outside the sphere is accomplished with the buttons in the Map Eraser pane. ![](https://lh7-us.googleusercontent.com/04qv9G8YU2x-K6VRa6oWLvm2rd_dN0GHmbINRIn4pFqyYdpuxIQgzswXKgpRskx0OJKEumZu3f1lrhv4ElSRiwix0vpDh2dJPvsqBPk5TZGjmz_rnszR7IZd9KECHSbdy9SU23ZqkCr4XM-9l7afF7s) The Volume Eraser pane controls the size of the eraser (pink sphere) and allows for erasing (i.e., setting to 0) all values inside or outside the eraser. 2\. Using the sphere, erase all regions outside your desired mask. Perform a close inspection of your final volume, being careful to notice small regions left behind by imprecise eraser placement: ![](https://lh7-us.googleusercontent.com/DvZhYdkLwD4y6-NJuVnmh6ikHttq72Rcm22J3RxV8hfVkQ1DnJ75Iryj28jKJu1bae4U2vWrEnibEM8Xm-BDo3ZEVGSCReVSqQFwmZmi4_9OH7yLIhVDvR94MMTIJ1Wrv2s7rnR2LPNkbd0uXIrWlDA) Small fragments of the map may be left behind by the volume eraser. It is important to closely inspect the map after each eraser operation to remove this "dust". 3\. Save the erased map as your mask base. #### Step 4: (Optional) Create a particle subtraction mask 1. Subtract the erased and blurred map (in this case, #2) from the unerased and blurred map (in this case, #3): \`volume subtract #3 #2\`. If the result has negative values in most voxels and an unexpected and noisy shape, the arguments were likely given in the wrong order. ![](https://lh7-us.googleusercontent.com/JSw8WIy42FpI97ShEJkDFd6Z42KpqmCc8NqObfARaa6LxnzSOts32RQnKyqYuVAB39HdP-Duu9iQEyUj9T5PsJc5VJy7piA27-CjY_60BdY7_nbsjINdWnz8fOjPP_wnTU90n4pQKZ1HFTly1Tfif2Q) Subtracting the local refinement mask base (#3) from the original blurred map (#2) creates the complementary particle subtraction mask (#5). \#### Step 5: Upload the mask bases to CryoSPARC 1. Save the resulting mask bases to \`.mrc\` files and upload the files to CryoSPARC. ### Method Three: molmap ChimeraX can create volumes based on molecular models using a command called molmap. Generating mask bases using this technique is by far the simplest — using a single command, we can create a mask around (in this example) chain U: \`molmap #2/U 16 onGrid #1\`. {% hint style="warning" %} If \`onGrid\` is left out of this command, a seemingly-correct mask base will be generated, but it will be on the wrong grid and so unusable! {% endhint %} In this example, #2 is our molecular model, we generated a mask base with a resolution of 16 angstroms, and #1 is our map from CryoSPARC. Note that even though none of the information in the mask base comes from the map, you still must have it loaded so that the mask base is on the correct grid. {% hint style="info" %} In this command, resolution merely notes the level of detail in the resulting simulated map. It will not affect the quality of refinements using the mask. {% endhint %} Note also that any resolution can be selected. ChimeraX is not simulating any electron microscopy process — it is simply generating a volume using the provided model and the specified resolution. We recommend that masks are never generated with a resolution better than (i.e., never a value lower than) 12 Å. Masks created using molmap are already on the correct grid and can immediately be saved and uploaded to CryoSPARC. ## Converting a mask base to a mask A mask base is converted to a mask by following steps: 1. The mask base is “binarized”. All values greater than a user-selected threshold are set to \`1.0\` and all values below this threshold are set to \`0.0\`. 2. The resulting binary volume is dilated. Additional pixels within a user-selected distance from the volume surface are also set to \`1.0\`. 3. The binary volume has a soft edge added (padding). This edge gradually decreases from \`1.0\` to \`0.0\` and has a user-specified width. All of these steps can be performed simultaneously via a \[Volume Tools\](/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools.md) job, and more information about the necessary parameters are available on that job page. The binarization threshold will be different each time depending on the input mask base, and can be determined with a volume visualization tool like ChimeraX. A threshold should be selected such that there are no floating “specks” of density and the desired topology of the mask is preserved. The amount of dilation required also depends on both the dataset and the sub-volume. Generally adding a few pixels of dilation helps to prevent over-tight masks, and allows for the inclusion of newly-resolved density in the masked volume. Padding is an \[essential component of masking\](#why-do-masks-need-a-soft-edge) in cryoEM to prevent ringing artifacts. We recommend a minimum padding width of $$5 \\times \\frac{\\mathrm{resolution}}{\\mathrm{apix}}$$ where \*\*resolution\*\* is the GSFSC resolution in Å and \*\*apix\*\* is the pixel size in Å, but the optimal result can require significantly larger padding widths. We therefore recommend that users try a variety of mask dilation and padding combinations to find the ideal combination. ## Next steps {% content-ref url="/pages/-MRveaHYieELN9\\\_9\\\_xca" %} \[Job: Local Refinement\](/processing-data/all-job-types-in-cryosparc/local-refinement/job-new-local-refinement-beta.md) {% endcontent-ref %} {% content-ref url="/pages/-MM27iCVzvbU6PO54C2O" %} \[Job: Particle Subtraction\](/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta.md) {% endcontent-ref %} ## References 1. Eric F. Pettersen et al., “UCSF ChimeraX: Structure Visualization for Researchers, Educators, and Developers.,” \*Protein Science : A Publication of the Protein Society\* 30, no. 1 (January 2021): 70–82, . 2. Thi Hoang Duong Nguyen et al., “Cryo-EM Structure of the Yeast U4/U6.U5 Tri-snRNP at 3.7 Å Resolution,” \*Nature\* 530, no. 7590 (February 1, 2016): 298–302, . 3. Grigore Pintilie and Wah Chiu, “Comparison of Segger and Other Methods for Segmentation and Rigid-Body Docking of Molecular Components in Cryo-EM Density Maps.,” \*Biopolymers\* 97, no. 9 (September 2012): 742–60, . \[^1\]: The notable exception to this is 3D Flex Mesh generation, which does not require a soft mask. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots.md). # Tutorial: Common CryoSPARC Plots ## Particle Picking ### NCC vs Power Score ![](https://guide.cryosparc.com/files/Kp0GNxvnw8OnXiEZYs9R) This 2D histogram is presented in Inspect Particle Picks as well as in CryoSPARC Live’s Picking tab. The Normalized Cross Correlation (NCC) is binned along the x-axis, and the local Power Score is binned along the y-axis. The NCC tells us how well how a particle candidate matches the template (used for picking) in terms of its shape; the value is equal to the cross correlation between the template and the patch of the micrograph at each point. It is often helpful to remove picks with low NCC scores. The power score is a measure of pixel intensity at a particular location; the value is equal to the squared amplitude of the signal, after background subtraction. Regions of the micrograph with low power score often correspond to empty patches, or false positive picks. Regions of the micrograph with high power score often correspond to aggregated proteins, nanoparticles, carbon edges, or crystalline ice. Thus, it is often helpful to remove picks with extreme (large or small) power scores relative to the dataset’s distribution. #### Example of using power to remove ice and aggregation ![](https://guide.cryosparc.com/files/8pHmR1Uvbj6G7mLUtPmZ) This is an example of a micrograph’s picks, before and after removing high-powered picks (in this dataset, power score greater than 887). Note how picks from the ice region near the top left are removed, as well as picks from areas of aggregated proteins near the bottom and right side. #### Example of using power to remove low contrast picks ![](https://guide.cryosparc.com/files/FHWiWAkVtodcq5gmHihk) This is an example of a different micrograph’s picks, before and after removing low-powered picks (in this dataset, picks with a power score less than 552 are removed). Note how picks from areas of particularly low contrast, containing many false positive picks, are removed. Note that if micrographs from a \[Micrograph Denoiser\](/processing-data/all-job-types-in-cryosparc/exposure-curation/job-micrograph-denoiser-beta.md) job are used to pick particles, the power and NCC will be different from if a raw micrograph is used. This is because Inspect Particle Picks calculates the scores against the denoised micrographs, if they are present. In either case, the particles will be extracted from the raw micrograph. ![](https://guide.cryosparc.com/files/k3nrxY0ew4629V3Dfmjg) The same particle pick locations are plotted in each heatmap. On the left, the raw micrographs were provided to the Inspect Particle picks job. On the right, the denoised micrographs were provided. \## Basic 2D plots ### Class ESS (Effective Sample Size) Histogram ![](https://guide.cryosparc.com/files/xmuS2VHaPLoBFJ3mfFNL) This histogram shows the distribution of the effective sample size (ESS) of the class posterior distribution across particles. ESS is a measure of the ‘peakedness’ of a probability distribution. A particle with an ESS of $$1$$ confidently belongs to only one class. A particle with an ESS equal to $$K$$, where $$K$$ is the total number of classes, has a uniform probability of $$1/K$$ of belonging to all classes. When many particles have ESS much greater than 1 (as shown in the figure above), the classification routine is uncertain due to duplicate/overlapping classes, overall poor class quality, or incomplete classification. ### Probability of Best Class Histogram ![](https://guide.cryosparc.com/files/NzjDXowbNY6xSwWAt9RE) This histogram shows the distribution of the maximum probability across classes for each particle. A particle with low probability in its best class has significant probability distributed across other classes (i.e., it has high class ESS), meaning that the particle’s classification is uncertain. When most particles have a probability of best class near 1.0, the particle set is confidently classified and classification has converged. ### Class Averages ![](https://guide.cryosparc.com/files/r0SUdHFZfHMqrrFq1pll) This figure displays a grid of class averages with overlaid metrics. The metrics are: (1) the number of particles assigned to the class, (2) the FRC (Fourier Ring Correlation) resolution of the class, and (3) the median class ESS of particles assigned to that class. The resolution reported (metric 2) is the value at which the FRC crosses a threshold of 0.5 for each class. Classes with poor resolution contain many junk particles. Classes with high median particle ESS contain many uncertain particles, indicating that the class may be too similar to other classes, or may contain particles that should belong to several different classes. ## Basic 3D plots ### Real-Space Slices ![](https://guide.cryosparc.com/files/QG1jmbJHc2npYtn3JQ4W) Three real-space slices of a 3D density. These are produced by many refinement jobs within CryoSPARC. Each subplot shows a real-space density slice along one of the coordinate planes: z-y, z-x, and y-x, respectively. The pixel colour is proportional to the scalar density value at each voxel. ### Real-Space Projections ![](https://guide.cryosparc.com/files/rPCm6Z8sv7WxzW36nLUO) Three real-space projections of a 3D density. These appear primarily in Ab-initio Reconstruction’s structure plots. Instead of slicing the density along a plane, the density is summed (i.e. integrated) along the normal to that plane, and the resulting sum is displayed, for the z-y, z-x, and y-x planes respectively. ### Fourier-Space Slices ![](https://guide.cryosparc.com/files/0x58t6rhkLPwUbGmGbN1) These three subplots display coordinate-plane slices of the Fourier volume. The Fourier volume is the 3D grid of complex numbers that result from applying a 3D discrete Fourier transform to the real-space density. Colours correspond to log amplitude (also called the ‘magnitude’ or ‘modulus’) of each Fourier coefficient. Note that although each Fourier component is complex-valued, only the amplitude (and not the phase) is displayed in this plot. ### Guinier Plot ![](https://guide.cryosparc.com/files/2bfMyj5bQ6TpwyeyBCzT) The Guinier plot displays the following: \* In green: the logarithm of the ‘structure factor’ F (i.e., the logarithm of the shell-averaged squared norm of the Fourier coefficients) \* In blue: the straight-line envelope function computed from the ‘B factor’. This envelope function is calculated by fitting a line to the log-structure factor between 10 Angstroms and the 0.143 FSC resolution, and the fitted B-factor is proportional to the slope of this envelope function. Some nuances about how this differs from bfactor estimation in other SPA software can be found on \[our discussion forum\](https://discuss.cryosparc.com/t/estimated-bfactor-differ-by-a-factor-of-two-compared-to-relion/10971/14?u=mmclean). The envelope function models the cumulative effect of all resolution-limiting factors present in the imaging conditions. Estimating the envelope function is useful as it can be used to restore the expected power spectrum through a process called \[Sharpening\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-sharpening-tools). The envelope function itself is given parametrically by a squared-exponential falloff over frequency, with scaling factor $$B$$: $$E(d) = \\exp{\\left(-\\frac{B}{4} \\omega\\\_d^2\\right)}$$ as described in section 4.7 of (Glaeser et al., 2021), and (Rosenthal & Henderson, 2003). ### Orientation Plots The next two plots contain information regarding the distribution of orientations in the dataset. For a more thorough discussion of orientation-related diagnostics, including metrics to diagnose preferred orientation, refer to the \[Orientation Diagnostics job\](/processing-data/all-job-types-in-cryosparc/utilities/job-orientation-diagnostics.md) and \[tutorial\](/processing-data/tutorials-and-case-studies/tutorial-orientation-diagnostics.md). #### Viewing Direction Distribution ![](https://guide.cryosparc.com/files/4cdMBSvNxnZcbIYTLUFb) The viewing direction distribution plot is one of two plots illustrating the diversity of orientations in the dataset. Every particle has an associated viewing direction, which is understood as the direction vector of the integral projection that the 2D particle was generated from, relative to the global orientation of the 3D volume. The set of possible viewing direction vectors can be interpreted as the surface of a unit sphere, or a “globe”. Thus in the viewing direction plot, the x-axis corresponds to azimuth (analogous to longitude) and the y-axis corresponds to elevation (analogous to latitude). The viewing direction plot is a 2D-histogram that shows the number of particles with a viewing direction at a particular elevation/azimuth bin. The viewing direction distribution plot is useful for understanding the diversity of orientations present in the dataset. However, it generally cannot be directly used to infer if the dataset has preferred orientation issues, because the viewing direction distribution doesn’t directly visualize the directions along which the volume is well-sampled. The \[Orientation Diagnostics job\](/processing-data/all-job-types-in-cryosparc/utilities/job-orientation-diagnostics.md) provides a more thorough set of tools for diagnosing orientation issues. #### Posterior Precision Directional Distribution ![](https://guide.cryosparc.com/files/SfA0OsKnSc2Kwaivw0pu) The posterior precision directional distribution plot is another plot illustrating the diversity of orientations in the dataset. If the volume is located at the center of a large circumscribed sphere, the elevation and azimuth angles define the direction of a radial line segment pointing out from the center of the structure. The posterior precision directional distribution plot displays roughly \*how many images contributed to the voxels that lie along this radial line segment.\* Note that this is different from the viewing direction distribution, which shows the axis along which the particle was viewed, i.e. the axis along which the volume was projected to generate the particle. The two plots are related as follows: if the viewing direction plot shows non-zero density at a viewing direction of v, then the posterior precision plot will show nonzero density at the set of all vectors orthogonal to v, i.e. the plane with normal vector v. For a greater understanding of the geometric relation between these two plots, it is useful to gain an understanding of the \[Fourier-slice theorem\](https://en.wikipedia.org/wiki/Projection-slice\_theorem). A related plot of the “Fourier Sampling” displayed in the \[orientation diagnostics job\](/processing-data/all-job-types-in-cryosparc/utilities/job-orientation-diagnostics.md) is very similar to the posterior precision directional distribution plot. The difference between the two is that the posterior precision plot accounts for the loss of information induced by the CTF of the particles, whereas the Fourier Sampling plot displays purely geometric information related to the particles’ alignments. ### Fourier Shell Correlation Plots {% hint style="info" %} CryoSPARC’s automatic FSC generation algorithm changed in v5.0. The plots produced by CryoSPARC v5.0+ will not match all of the images shown here, but the advice given regarding the shape of the FSC curves and what that may mean about the data still holds. {% endhint %} Fourier Shell Correlation (FSC) plots display the correlation coefficient between spherical shells in Fourier space. FSCs are typically calculated after applying a mask to each half-map, which excludes solvent noise and other unwanted signal that exists outside the region covered by the target molecule. CryoSPARC’s calculates the FSC using several different masks and plots a curve for each in the FSC figure. #### Prior to CryoSPARC v5 Prior to CryoSPARC v5, refinements plotted five curves at each iteration: \* No Mask: no mask is applied to the half maps. \* Spherical: A soft spherical mask starting 85% of the way to the box edge and ending at the box edge is used. \* Loose and Tight: Automatically generated masks which follow the contours of the map, each with fixed dilation and soft padding amounts in Angstroms. ![](https://guide.cryosparc.com/files/o71vqg9ywKKMteK1cO9U) Once a refinement converged, the spherical mask FSC was no longer plotted. Instead, an FSC corrected by \[noise-substitution\](#high-resolution-phase-randomization) was added. !\[\](/files/EScAexXkWkQRNgZXqcVP) #### CryoSPARC v5+ Starting with CryoSPARC v5, the loose and tight masks are replaced with a single resolution mask. The lines also have different line styles to aid differentiation between them. !\[\](/files/5BIv5QiyUOrQH6iAGZbx) The final plot shows only the resolution mask, the auto-tightened resolution mask, and the auto-tightened resolution mask corrected by \[noise substitution\](#high-resolution-phase-randomization). !\[\](/files/AwrmXCE6iEFaNrEozqFa) Some jobs (such as \[Homogeneous Reconstruction Only\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-homogeneous-reconstruction-only) and \[Local Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-new-local-refinement-beta)) use the input mask directly for computing FSCs rather than generating a new one. The FSC plots produced by these jobs use a different color and name to highlight this fact. !\[\](/files/u76EYhswK2tmEyRoRsKz) #### \*\*High resolution phase randomization\*\* The “corrected” FSC curve is the FSC curve obtained by following a similar procedure to that outlined by Chen et al. in their 2013 publication, \[\*\*High-resolution noise substitution to measure overfitting and validate resolution in 3D structure determination by single particle electron cryomicroscopy\*\*\](https://www.sciencedirect.com/science/article/pii/S0304399113001472). This procedure is applied to any standard refinement algorithm, such as homogeneous refinement in CryoSPARC. As published by Chen at al, it consists of creating a second set of particles that are identical to the original dataset except with random phases beyond a certain resolution in the particle dataset; this dataset is identical to the first at low and medium resolutions, but does not contain phase information of the signal at high resolutions. Then, this dataset is to be separately refined against a reference, simultaneously with the original particle dataset. This procedure was developed as a way to measure systematic contamination (or “overfitted noise”) that has been induced by application of a mask during FSC calculation; the FSC curves from the “phase-randomized” dataset can be compared quantitatively with the FSC curves from the original refinement. While this procedure robustly detects overfitted noise that builds up over the course of a refinement, it is twice as computationally costly as a standard refinement procedure. Thus, a cheaper approximate version of the procedure has been adopted in CryoSPARC and other SPA softwares. The implemented version of high-resolution phase randomization instead only happens on the \*raw half-maps\* in a refinement, for the purpose of calculating a corrected FSC curve as specified in equation (4) of Chen et al. Specifically, the way in which this corrected FSC curve is computed involves randomizing the phase of the half-maps’s Fourier coefficients beyond a certain frequency, set to be 75% of the frequency at which the tight masked curve crosses the 0.143 threshold. In CryoSPARC, the plotted “corrected” curve is coincident with the standard “Tight” masked FSC curve below this resolution. Above this resolution, the “corrected” curve is given by equation 4 of \[Chen et al. (2013)\](https://www.sciencedirect.com/science/article/pii/S0304399113001472), referred to by the authors as the $$\\text{FSC}\\\_{\\text{true}}$$ curve. At the phase randomization resolution, the curve often has a sharp dip which arises due to the discontinuity in Fourier structure phases. These dips are a common occurrence and are generally regarded as a positive indicator that phase randomization was correctly applied. It is important to note that this modified phase-randomization procedure means that the corrected FSC curve \*does not\* reliably indicate whether overfit noise has built up during the refinement. \*\*The corrected FSC curve can only indicate whether the mask used to compute the FSC (this is the “Tight” mask in any FSC plot) is “too tight” to reliably report resolution\*\*. Devising improved resolution metrics is an important problem facing the overall field of cryo-EM, and a foolproof metric of resolution does not currently exist. In the figure below, three examples of FSC curves along with associated mask tightnesses are shown. The leftmost side shows an example of a corrected FSC curve that indicates a mask with good tightness has been used, with minimal shared overfitting between the half-maps. The middle plot shows an example of the mask being slightly too tight — note how the “corrected” curve drops around 3.8 Å but eventually returns to being coincident with the tight curve. Finally, the rightmost plot shows an example where the tight mask is significantly too tight, made clear by the corrected curve substantially deviating from the uncorrected (tight) curve, and remaining this way indefinitely. ![](https://guide.cryosparc.com/files/JQP9M9tOOM1MKHRAMFXD) This figure displays three examples of FSC curves along with associated mask tightnesses. \#### FSC Curves not dropping to zero Ideally, the FSC curve should drop to 0.0 before the Nyquist limit. When this occurs, the reconstruction resolution is limited by particle image quality and not pixel or image size. If the FSC remains positive all the way to the Nyquist limit, that means the two half maps are positively correlated at the highest frequency represented in the images. There are two reasons this typically happens: particle images which have been downsampled to too small a box size, and duplicate particles. It is common practice to significantly downsample particles early in the processing pipeline. This speeds early steps during which reconstructions are not expected to achieve high resolutions. Eventually, the particle stack becomes clean enough that the resulting reconstruction achieves Nyquist at this downsampled box size. In this situation, the FSC stays very high across the entire frequency range available in the images. ![](https://guide.cryosparc.com/files/fT2tIr1phN7vINIdpPI8) This FSC remains high all the way to Nyquist. This means there is likely still good information which has been cut off by downsampling. In these cases, it is highly likely that re-extracting these particles with a larger box size (i.e., with less downsampling) will improve the resolution of the reconstruction. This is because downsampling the particle images removes high frequency information. However, the high FSC value at Nyquist indicates that this higher-frequency information would likely correlate between the two maps. ![](https://guide.cryosparc.com/files/9PmTgUI3Me8xBTnQ6l5P) The same particles as the previous image, in the same poses, re-extracted to a full box size. Note that the resolution significantly improves without any further alignment, and the FSC reaches zero well before Nyquist. On the other hand, FSC curves for maps with duplicate particles remain positive all the way up to Nyquist, but have a long rightward “tail” as shown in the image below. This can occur when particle picks are too close to each other in the dataset, which may happen when combining particle picks from multiple picking strategies. Particles that are too close may become coincident after being aligned to the reference during a refinement, and if these particles are present in two different half-sets, they will break the independence assumption between half-sets and thus invalidate the reconstructions. ![](https://guide.cryosparc.com/files/4KVkXocRANpQUKKHWurG) A particularly dramatic example of artifactual FSC curves arising from duplicate particles being present in both half-sets. The \[Remove Duplicate Particles\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-remove-duplicate-particles) job may be used to discard particles that are too close to each other, if particle pick locations are available. Note that \[Helical Processing tools\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/helical-reconstruction-beta) in CryoSPARC address this problem differently, by explicitly placing particles with overlapping signal into the same half-set, to preserve the independence between half-sets. #### Sharp bumps in FSC (CTF issues) Systematically incorrect CTF parameters can often manifest as oscillations in a refinement’s FSC. These are characterized as multiple oscillations in the FSC that appear like the curves in the image below. If these are observed in final refinements, it is likely that one or more of the microscope optical parameters are incorrectly specified: important parameters to check are the pixel size, accelerating voltage, and spherical aberration. This phenomenon is discussed in more detail in \[RELION’s documentation\](https://relion.readthedocs.io/en/release-4.0/Reference/PixelSizeIssues.html#cs-and-the-error-in-the-pixel-size). ![](https://guide.cryosparc.com/files/MVtTFWTUr1ayhLIVyxHK) An example of artifactual FSC oscillations owing to an incorrect spherical aberration specified at movie import time. \#### Dip in FSC due to disordered regions For membrane proteins with disordered regions (e.g. micelles or nanodiscs), it is common for there to be a region in the frequency band (approximately between \\~9 Å and \\~5 Å) where the FSC value dips lower than surrounding values. This is due to the stronger presence of disorder in those frequency bands from the lipids forming the micelle or nanodisc, which have no fixed position relative to the protein structure. Generally, this dip is an expected artefact when refining membrane proteins. An example is shown below. ![](https://guide.cryosparc.com/files/qqY45tGSfIJO9ktckhQS) Example of a healthy FSC curve of a membrane protein, with a dip in the frequency band (approximately between ~9 Å and ~5 Å) owing to disorder. \### Noise model The noise model used in a CryoSPARC job is a parameter of the statistical model that governs image formation. Observed images are modelled to be a tomographic projection of the underlying density at some pose, convolved with the point spread function (PSF), and subjected to additive gaussian noise. Physically, this gaussian noise is used to represent the “shot noise” induced during the imaging process in the microscope’s detector. The images, projections, and noise are all represented as two dimensional quantities, and the underlying density is represented as three dimensional. When the image formation model is expressed in Fourier space, gaussian noise is parameterized as having a diagonal covariance, and subject to the further constraint that all noise variance values are constant across pixels belonging to the same frequency band. This is equivalent to the assumption that noise is isotropic over direction, and therefore all noise models are functions of Fourier shell numbers only. This can be best visualized in the advanced noise model plot below, which shows (on the right) a 2D colour-map of the noise model plot in Fourier space; note the values being constant over a given ring. In each frequency ring, the noise variance is estimated via computing the Fourier-space “residual” in each image – this is the squared difference between the noisy raw data, and the CTF-corrupted projection of the signal. The squared residual is averaged across all images, and is further averaged across frequency-band, to produce the noise estimate. A common trend in the noise variance is an increase at high frequencies. An example of this is illustrated in the basic noise model plot, below. This is due to the effect of dose-weighting. While we expect that the noise variance in each individual movie frame is approximately \*\*\*white\*\*\* (that is, approximately constant over different frequencies), the motion-corrected micrograph itself is comprised of a sum of movie frames, and we do not use a uniform weight over frequency or over frame when summing frames to produce a micrograph. This means that the noise variance of the \*\*micrograph\*\* is not expected to remain white, even if the noise variance of each movie frame is white. In all cases, CryoSPARC uses near uniform weights over frame index when averaging at low-frequencies. At high-frequencies, the weights are large for early frame indices, and small for later frame indices, which has the effect of increasing the noise variance at high-frequencies relative to low-frequencies. More information about dose-weighting schemes can be found in our \[Reference-based Motion Correction documentation\](/processing-data/all-job-types-in-cryosparc/motion-correction/job-reference-based-motion-correction-beta.md). ![](https://guide.cryosparc.com/files/VuL7zJrbiEV4qQXfmoRK) Basic noise model plot (produced by many refinement jobs) — This plot shows the current estimated noise variance, \\sigma^2, as a function of wavelength, shown in units of Angstroms (based on the pixel size). ![](https://guide.cryosparc.com/files/Ugb6g2Cw90OLZz4c4DeK) Advanced noise model plot (produced by ab-initio) — this plot is similar to the basic noise plot, but explicitly shows the difference the total noise (sigma) and the empirical error (error), either averaged per shell (left) or as a 2D projection (right). The difference between the two is a result of noise priors and regularizers (cf., Punjani (2016)). Specifically, the error plot is the result of averaging the squared residual across all processed images in the dataset, and the sigma plot is the result of further averaging across frequency-band (this can be thought of averaging across concentric circular bands centered at the plot’s origin). \## References \* R. M. Glaeser, E. Nogales, and W. Chiu, “4.7 B factors and map sharpening,” in Single-particle cryo-em of biological macromolecules, Bristol, UK: IOP Publishing, 2021, pp. 4-59-4–67 \* P. B. Rosenthal and R. Henderson, “Optimal determination of particle orientation, absolute hand, and contrast loss in single-particle electron cryomicroscopy,” Journal of Molecular Biology, vol. 333, no. 4, pp. 721–745, 2003. doi:10.1016/j.jmb.2003.07.013 \* S. Chen \*et al.\*, “High-resolution noise substitution to measure overfitting and validate resolution in 3D structure determination by single particle electron cryomicroscopy,” \*Ultramicroscopy\*, vol. 135, pp. 24–35, 2013. doi:10.1016/j.ultramic.2013.06.004 --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716.md). # Case Study: End-to-end processing of encapsulated ferritin (EMPIAR-10716) ## Introduction The aim of this case study is to demonstrate some advanced tools and processes within CryoSPARC that enable processing of structures with unconventional symmetry present. The dataset this case study will cover is an encapsulin nanocompartment originally collected by \[Jennifer Ross, et al. (2022)\](https://www.science.org/doi/10.1126/sciadv.abj4461), containing four encapsulated ferritin (EncFtn) decamers within. The raw data is publicly available for download as \[EMPIAR-10716\](https://www.ebi.ac.uk/empiar/EMPIAR-10716/). {% hint style="info" %} The main topics of focus covered in this case study include: high-symmetry ab-initio reconstruction, local symmetry, non-point-group symmetry, symmetry expansion, custom geometry operations. The jobs focused on include: \[local refinement\](/processing-data/all-job-types-in-cryosparc/local-refinement.md), \[3D classification\](/processing-data/all-job-types-in-cryosparc/variability/job-3d-classification-beta.md), \[volume alignment tools\](/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools.md), \[align 3D maps\](/processing-data/all-job-types-in-cryosparc/utilities/job-align-3d-maps.md), \[particle subtraction\](/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta.md), \[regroup 3D\](/processing-data/all-job-types-in-cryosparc/variability/job-regroup-3d-classes.md), \[symmetry expansion\](/processing-data/all-job-types-in-cryosparc/utilities/job-symmetry-expansion.md). {% endhint %} Encapsulins are a type of protein that contain internal cargo, and this encapsulin is an icosahedrally-symmetric molecule from the Haliangium ochraceum bacteria. The geometry of the nanocompartment is best illustrated in Figure 1: ![](https://guide.cryosparc.com/files/dzKx9Eh3o7HWesUM4f7e) Figure 1. Schematic diagram of the geometry of encapsulin and encapsulated ferritin in the Haliangium ochraceum bacteria. Due to the complicated geometry of this nanocompartment, processing it can be challenging. The four EncFtn decamers are arranged in an approximate tetrahedral shape, which complicates solving this structure due to the mismatch in symmetry of the cargo and the shell. Furthermore, each EncFtn molecule itself has 5-fold dihedral symmetry, meaning there is an additional symmetry mismatch between the tetrahedral arrangement and the internal D5 symmetry of each EncFtn molecule. While refinement of the encapsulin is fairly straightforward, recovering high resolution in the internal EncFtn is difficult without custom steps that take care to respect the geometry of the cargo. This case study walks through the steps we took to handle this geometry in CryoSPARC, achieving a final high-resolution structure of both encapsulin and encapsulated ferritin. The previously published map of encapsulated ferritin from this dataset reached resolutions of 5-6 Å; using the techniques of this case study, we were able to resolve the encapsulated ferritin to a sub-3 Å structure. This case study is divided into two sections, each with subsections covering the major processing tasks: \* Section A: Encapsulin Processing \* A1: Preprocessing and Particle Picking in CryoSPARC Live \* A2: 2D Classification \* A3: Encapsulin 3D Reconstruction \* Section B: Encapsulated Ferritin Processing \* B1: Group Re-alignment on Tetrahedron \* B2: Custom Symmetry Expansion \* B3: Group Re-alignment on Encapsulated Ferritin \* B4: Local Refinement All processing was done in CryoSPARC and CryoSPARC Live v4.5. ## A: Encapsulin Processing This case study begins with processing the dataset, with the goal of reconstructing encapsulin. ### A1: Preprocessing and Particle Picking Preprocessing of exposures consists of import, motion correction, and CTF estimation. These steps can either be completed separately using individual jobs in CryoSPARC, or simultaneously using CryoSPARC Live. The latter can be quicker as it allows processing exposures in a streaming fashion, where one exposure can be imported, motion corrected, and CTF estimated all in sequence (i.e. without waiting for all other exposures to finish each step). We will use CryoSPARC Live to perform import, motion correction, CTF estimation, particle picking and extraction. {% hint style="info" %} If you haven’t used CryoSPARC Live before, you can review this \[Start to Finish Guide\](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide). {% endhint %} #### Download Raw Data (Subset) This dataset comprises 8,109 movies. {% hint style="info" %} Due to the large size of the dataset, we chose to only process a subset of the 8,109 movies. As we will see, that was sufficient for high resolution reconstruction due to the high symmetry present, but further improvements in map quality are possible if the entire dataset is used. {% endhint %} Thus in order to make processing quicker, we will only download the movies in two subdirectories, \`GridSquare\_16285984\` and \`GridSquare\_16286188\` which amounts to 2,815 movies in total. \`\`\`bash cd /path/to/rawdata # navigate to a container directory to hold the raw data wget https://www.ebi.ac.uk/empiar/world\_availability/10716/data/micrographs/GridSquare\_16285984/ . && wget https://www.ebi.ac.uk/empiar/world\_availability/10716/data/micrographs/GridSquare\_16286188/ . \`\`\` #### Set up Live Session \* Create a new CryoSPARC Project, and within this project, create a new Live Session. Under the configuration tab, enter the following configuration information and parameters: | Parameter | Value | | ------------------------------------- | ------ | | Raw pixel size (A) | 0.326 | | Accelerating voltage (kV) | 300 | | Spherical Aberration | 2.7 | | Total exposure dose (e/A^2) | 40.509 | | Save Results in 16-bit floating point | Yes | | Output F-crop factor | 0.5 | | Minimum particle diameter | 160 | | Maximum particle diameter | 230 | | Use circular blob | Yes | | Use ring blob | Yes | | Extraction box size | 800 | | Fourier crop to box size | 512 | \* In the Configuration Tab, create two exposure groups, with the following fields set: | | Exposure Group 1 | Exposure Group 2 | | ------------------------- | ----------------------------------------------------- | --------------------------------------------------- | | Directory to watch | \`.../10716/data/micrographs/GridSquare\_16285984/Data\` | \`…/10716/data/micrographs/GridSquare\_16286188/Data\` | | File name wildcard filter | \`\*fractions.tiff\` | \`\*fractions.tiff\` | | Enable continuous import | True | True | \* Use at least one \*Preprocessing GPU worker(s).\* Set the number of \*Reconstruction GPU workers\* to 1 (note reconstruction tasks i.e. ab-initio and refinement will not be used in Live for this case study). \* Click “Start session” to begin processing. CryoSPARC Live will automatically begin motion correction, CTF estimation, particle picking, and particle extraction. \* In the \[Overview Tab\](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-4.-exclude-poor-quality-exposures-from-downstream-processing), modify the upper CTF fit resolution threshold to 6 Å. \* In the \[Picking Tab\](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#id-6.-fine-tune-particle-picking), adjust the Normalized Cross Correlation (NCC) and Power Score sliders to remove false positive picks. CryoSPARC Live will work through the exposures and process them until extraction is complete for each exposure. You can tell when the exposures have finished processing when the number of processed exposures equals the number of total exposures. We are now done with CryoSPARC Live for this case study. For the remainder of the processing, we will use the standard CryoSPARC interface. \* Navigate to the top dropdown menu, and click “Go to session workspace”; in this workspace we will carry out the rest of the jobs. ![](https://guide.cryosparc.com/files/49HBupWbpcMig31kp6Zn) \### A2: 2D Classification Once we have an exported stack of particles in CryoSPARC, we will use 2D Classification to curate our particle stack and remove false positive particle picks. \* In the session workspace, locate the most recent “Live Particle Export” job. Add these particles to a new \[2D Classification\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/job-2d-classification) job with the following parameters: | \*\*Parameter\*\* | \*\*Value\*\* | | ------------------------------- | --------- | | Number of classes | 80 | | Minimum separation distance (A) | 60 | | Number of GPUs to parallelize | 3 | \* Once the 2D Classification job is complete, use \[quick actions\](https://guide.cryosparc.com/application-guide-v4.0+/creating-and-running-jobs#job-quick-actions) to queue a Select 2D Classes job. Select classes that show high resolution detail in the encapsulin. Note that since the encapsulin is the dominant signal in the images at this stage, the interior cargo will likely remain blurry and ill-defined for most of the classes, even the classes for which encapsulin is well-defined. Reject all classes that have multiple overlapping encapsulin particles, are empty, include ice or carbon edges, or otherwise have junk in them. ![](https://guide.cryosparc.com/files/VanHoLv5GoRw2QvIPd8H) Figure 2. A screenshot of the Select 2D Classes job used to select a subset of particles to continue to 3D reconstruction. \### A3: Encapsulin 3D Reconstruction The figure below illustrates the workflow for subsection A3 of this case study. ![](https://guide.cryosparc.com/files/sjzrb6qy2G9Qzrsh7csC) Figure 3. Flow chart of particle processing for 3D reconstruction of encapsulin \#### Initial Model Generation Now that we have a set of curated particles, we will move onto 3D initial model generation. High-symmetry structures often require special treatment during initial model generation, as the particle images for these types of structures typically look very similar to each other at low and medium resolutions, regardless of the particle pose. This lack of information in the data makes it difficult for algorithms like Ab-Initio Reconstruction to reconstruct the correct structure when using default parameters. Typically, running Ab-Initio Reconstruction in this setting will yield “flattened” density, with all particles assigned to the same viewing direction. There are two options to work around the lack of information in the images: 1. \*\*Enforce symmetry during Ab-Initio Reconstruction\*\*: This will guarantee that a symmetric structure is found. 2. Alternatively, \*\*Disable “Enforce non-negativity\*\*”: This parameter has empirically been found to help discourage ab-initio from producing flattened models. We can see the difference between options 1 and 2 in the final structures found by Ab-initio in each case: ![](https://guide.cryosparc.com/files/71DuSuAHa1o7BDX7KwSH) Figure 4. Symmetry-enforced ab-initio reconstruction. Internal details are lost ![](https://guide.cryosparc.com/files/dSpSuqH96UQBfwA6JLiR) Figure 5. Asymmetric ab-initio reconstruction (with non-negativity off). Internal tetrahedral arrangement of EncFtn is visible. The first option is undesirable because we would like to preserve the internal asymmetric structure within the encapsulin as best as possible. The internal structure \*doesn’t\* follow an icosahedral symmetry like the outer shell does, so enforcing symmetry will prevent any details from being resolved inside of the encapsulin. \* Taking the particles from the Select 2D Classes job, we will build an \[Ab-initio Reconstruction\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-reconstruction/job-ab-initio-reconstruction) job with the following parameters: | Maximum Resolution (Angstroms) | 8 | | ------------------------------- | --- | | Initial Resolution (Angstroms) | 25 | | Center Structures in Real space | Off | | Enforce non-negativity | Off | #### Volume Alignment Tools (Symmetry Alignment) After obtaining an initial model of the encapsulin, we would like to refine it to high-resolution. Since we gave Ab-Initio no symmetry information, the structure is oriented arbitrarily. This is fine for intrinsically asymmetric structures, but for symmetric structures, we must ensure they are aligned to the symmetry axes if we later want to \[enforce symmetry\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-homogeneous-refinement#common-parameters) or enable \[symmetry relaxation\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation). {% hint style="info" %} Alignment of the initial volume to the symmetry axes is also done automatically by the subsequent Homogeneous Refinement job, but we include it explicitly here to get familiar with the Volume Alignment Tools job’s parameters, inputs and outputs. {% endhint %} \* To do this, build a \[Volume Alignment Tools\](/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools.md) job, activate symmetry alignment, and input “I” as the symmetry string. Connect both the volume and particles from Ab-Initio Reconstruction to this job. The output volume should be aligned to the icosahedral symmetry axes. #### Refinement of encapsulin Now that we have an aligned model, we can refine this to high resolution using a \[Homogeneous Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-homogeneous-refinement) job. For this job, we will use \*symmetry relaxation\* to give the refinement the best chance of preserving the asymmetry of the encapsulin contents. \* To do this, build a \[Homogeneous Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-homogeneous-refinement), and set the following parameters: | Symmetry | I | | --------------------------------- | ------------ | | Symmetry Relaxation Method | maximization | | Dynamic mask start resolution (A) | 1 | Setting the symmetry relaxation to “maximization” enables symmetry relaxation. It can also be set to “marginalization”, which uses a \[slightly different method\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-symmetry-relaxation#comparing-refinement-methods) for finding the optimal pose. Setting the dynamic mask start resolution to 1 Å causes the job to use no mask, which is important as dynamic masking can remove lower-contrast asymmetric details that we’d like to preserve, such as the internal contents of the encapsulin. From here, with \\~250k particles, we obtained a C1-refined structure of encapsulin at around 3.0 Å. #### Homogeneous Reconstruction Only Now that we have a high-resolution reference structure, there are many avenues to further improve resolutions using reference-based algorithms for latent variable estimation. In this tutorial, we’ll use Global CTF Refinement to correct for high-order aberrations. We’ll also create a symmetrized version of the encapsulin portion of the reference, and then use this for Particle Subtraction \\\*\\\*to generate particle images with only signal from the internal contents present. This will prepare us for section B of this case study: processing the internal encapsulated ferritin structure. To get a icosahedral (I) symmetric reference for subtracting the encapsulin away, we don’t have to repeat a full refinement. \* Instead, connect the particles from the previous C1 refinement to a \[Homogeneous Reconstruction Only\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-homogeneous-reconstruction-only) job, and run the job with a symmetry of I specified. This will work as we have ensured our initial reference to the homogeneous refinement was symmetry aligned. {% hint style="info" %} Note that despite enforcing symmetry in this Homogeneous Reconstruction \\\*\\\*Only, the particles will retain their C1 alignments — thus the particles will remain suitable for downstream processing of the tetrahedral arrangement of EncFtn, and we won’t lose the effort put into preserving the symmetry-break. This would not be true if we re-ran a refinement with symmetry enforced, as alignments would be re-calculated against the symmetric reference. {% endhint %} Application of symmetry will result in a significant increase in resolution owing to the greater number of asymmetric units contributing to the structure. In our case, the resolution improved from 3.0 Å to 2.5 Å over the C1-refined structure. #### Mask generation using Volume Tools \[The Particle Subtraction job\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta) takes in a set of previously-aligned particle images, the corresponding reference volume to which they’ve been aligned, and a \[mask\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera) covering the region of the volume that we would like \*removed\* from the particle images. We’d like to subtract the encapsulin away, leaving particle images with just the encapsulated ferritin inside. To do this, we need to use Particle Subtraction, and provide it with a mask covering just the encapsulin. \* To obtain this mask, download the volume from the upstream Homogeneous Reconstruction job. Open this volume in UCSF ChimeraX, and select a threshold value that preserves most of the encapsulin while removing all of the internal density. \* Note down this threshold value. ![](https://guide.cryosparc.com/files/6yALGvIFoRZgMoDr3nBJ) Figure 6. The symmetric reconstruction shown at two threshold levels. \* Create a Volume Tools job and connect the volume from the reconstruction only job, with the following parameters: | Type of input volume | map | | --------------------- | -------------- | | Type of output volume | mask | | Threshold | \*chosen value\* | | Dilation radius (pix) | 4 | | Soft padding (pix) | 18 | #### Global CTF Refinement \* Connect the particles and volume from the upstream symmetry-enforced Reconstruction Only to a \[Global CTF Refinement\](/processing-data/all-job-types-in-cryosparc/ctf-refinement/job-global-ctf-refinement.md) job. Connect the mask from the previous Volume Tools job. Activate “Fit anisotropic mag”, to allow for estimation of anisotropic magnification; this will also set the number of iterations to 2. Global CTF Refinement works best when operating on the reference with the largest mass and highest resolution available, and on particles with refined alignments. Global CTF parameters are a function of the microscope (i.e., they have little dependence on which region of the protein is being refined), and so can be optimized with the 2.5 Å symmetry-enforced encapsulin map, which is large and high-quality. #### Homogeneous Reconstruction Only To get a reference generated from CTF-refined particles, we’ll repeat reconstruction using the CTF-refined particles. \* Create a Homogeneous Reconstruction Only job. Connect the particles from the previous Global CTF Refinement along with the mask from the Volume Tools job, set the symmetry to “I”, and launch the job. #### Particle Subtraction \* Connect the particles, volume, and mask from the previous Homogeneous Reconstruction Only job to a Particle Subtraction job. Since this mask only covers the encapsulin, the particles will have encapsulin subtracted away and internal contents preserved. ![](https://guide.cryosparc.com/files/NyRRIo414ph6o5cddkyF) Figure 7. Original vs. subtracted images (lowpass filtered) \## B: Encapsulated Ferritin Processing We now have subtracted particles containing just the four copies of encapsulated ferritin. However, we have a tricky case of symmetry to handle: \* Within each EncFtn, there is D5 (5-fold and 2-fold) symmetry \* The four encapsulated ferritin are arranged in a tetrahedron configuration. However this is not equivalent to a \*tetrahedral point symmetry group,\* because tetrahedral point symmetry has 3-fold symmetry at each vertex, but EncFer is 5-fold symmetric Thus, EncFtn has two different types of symmetry. First, each individual copy of EncFtn has D5 point group symmetry, which may be referred to as a \*local symmetry\*. Second, the overall tetrahedral arrangement imparts a \*non-point-group\*, four-fold symmetry, which must be treated with custom operations to superimpose each of the four EncFtn. Within each of the encapsulin nanocompartments, there are $$4\*5\*2=40$$ asymmetric units available for symmetry-averaging. The remainder of this tutorial focuses on how we can use CryoSPARC to align these asymmetric units, remove broken particles/further curate the particles, and refine to high resolution. The figure below demonstrates the processing chain we will use for refining the encapsulated ferritin. ![](https://guide.cryosparc.com/files/6euLJaylbe2iT0zNyABq) Figure 8. Flow chart of particle processing for 3D reconstruction of encapsulated ferritin. Encapsulated ferritin processing is divided into four sections: Group Re-alignment on Tetrahedron (B1); Custom Symmetry Expansion (B2); Group Re-alignment on Encapsulated Ferritin (B3); Local Refinement (B4) \### Homogeneous Reconstruction & Local Refinement (demonstration) Now that we have subtracted particles, we might first attempt to refine the interior. \* First, we launch a Homogeneous Reconstruction job to generate an initial reference from the subtracted particles. \* Next, using the subtracted particles and reference, launch a Homogeneous Refinement. To prevent the job from using masking altogether, set the “Dynamic mask start resolution” to 1 Å. With our stack of 250k subtracted particles, the Homogeneous Refinement reached a resolution of 10 Å. ![](https://guide.cryosparc.com/files/cFgbV1f3PGbDNxqiZ3wb) Figure 9. 10 Å structure refined from subtracted particles. Next, we attempted to locally refine one of the encapsulated ferritin proteins. By using the \[map eraser\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#method-two-volume-eraser) in UCSF ChimeraX, a mask was generated around one protein, and the structure was locally refined. ![](https://guide.cryosparc.com/files/7XyLT8DYakvgTzSuRGgZ) Figure 10. Locally refined EncFtn viewed at a low threshold ![](https://guide.cryosparc.com/files/Rgv3pSzuueUOB9XIzqtL) Figure 11. Viewed at a high threshold — note the artefacts are the strongest density in the map This Local Refinement stalled at a claimed resolution of \\~ 6.9 Å. The non-protein streaking artefacts visible in the map above are a characteristic sign of overfitting. There are two potential reasons why this Local Refinement was unsuccessful: 1. There are many junk particles in the dataset. This is supported by the fact that one of the 3D classes (from the next step) solved does not show a clear tetrahedral configuration of the four EncFtn instances, rather showing a disordered “ring” of density where three distinct EncFtn would be expected: ![](https://guide.cryosparc.com/files/FXBnb6RZ76BJkSompzFg) Figure 12. One of twenty classes found in section B2. Note the disorganized density along the bottom half of the image. ![](https://guide.cryosparc.com/files/m84EHc6AAe5gTg3hghpr) Figure 13. One of twenty classes found in section B2. Note that four separate densities are present. 2\. Initial particle alignments are not close enough to their optimal values. Local Refinement can only move particle alignments by so much, and the larger the search space we give it to make, the greater potential there is for overfitting. Local Refinement works best when it does not have to search a large range of poses, and when tight gaussian priors are used to limit the drift of alignments. Both factors above make it more difficult for local refinement to solve for a high-resolution structure with minimal artefacts. ### B1: Group Re-alignment on Tetrahedron Instead of refinement, we’ll attempt to perform 3D Classification on the tetrahedral arrangement of EncFtn. Since 3D Classification doesn’t update alignments, we’ll be able to see if there is heterogeneity in the orientations of the “tetrahedra”. To bias the classification as little as possible, we’ll provide a spherical mask covering the inside of the encapsulin. If we used a mask generated from the consensus subtracted particles, we may bias the classification to only find classes of the tetrahedron oriented in the same orientation as the consensus structure. #### Obtain spherical mask (UCSF ChimeraX) Before starting 3D Classification, we’d like to use a mask that excludes the corners of the box, along with any other residual density from the encapsulin. At this stage the best mask to use would be a soft spherical mask centered at the box center, as this biases the classification the least (as opposed to a mask contoured to the consensus density). \*\*Generate a mask base in UCSF ChimeraX\*\* Download the \`map\` from the most recent homogeneous reconstruction job. Open the map in a new ChimeraX session. Navigate to the “Tools” tab on the menu bar, and head to \`Tools > Volume Data > Map Eraser\`. The map eraser should open by default in the center of the volume; if not, adjust the pink sphere’s position to the center of the volume via clicking and dragging. Adjust the size of the map eraser to approximately surround the internal disordered density of the encapsulin; refer to the image below for an example: ![](https://guide.cryosparc.com/files/VGZzt4PZMdnntDaw0UWg) Lower the density threshold all the way to the lowest value in the volume — you should see a large cube of density. This is so that we can erase all density outside of a central sphere, leaving us with a spherical mask base in the center. Use the threshold operation to binarize the cube: \`\`\` vop threshold #1 maximum setMaximum 1 \`\`\` Now click “Erase outside sphere”. You should now have a solid sphere of density: ![](https://guide.cryosparc.com/files/FfHm59wvSF2XgdPtpWQD) Finally, save this map. Click \`File > Save...\` and change the “Map” dropdown to the thresholded volume. Give the file a name, and click save: ![](https://guide.cryosparc.com/files/S3VFUY0RTWGjuooGeCgr) \*\*Import Mask into CryoSPARC\*\* \* In your CryoSPARC Workspace, create an Import 3D Volumes job. Provide the path to the mask base. Change the “Type of volume being imported” to \`mask\`, and run the job. \*\*Volume Tools\*\* (padding) \* Connect the imported mask to a new Volume Tools job. Set the following parameters, to pad the mask with a width of \\~20 pixels: | \*\*Parameter\*\* | \*\*Value\*\* | | --------------------- | --------- | | Type of input volume | mask | | Type of output volume | mask | | Threshold | 0.5 | | Soft padding width | 20 | This job will produce a softly-padded mask as its output. #### Group Re-alignment \* Create a 3D Classification job. Take the particles from the Particle Subtraction job, along with the spherical mask, and connect the mask to the “Solvent mask” input. Use the following parameters for the 3D Classification job: | Number of classes | 20 | | -------------------------------- | ---- | | Filter resolution (Å) | 8 | | O-EM batch size per class | 2000 | | O-EM learning rate init | 0.9 | | O-EM learning rate half-life (%) | 0 | | Force hard classification | On | Looking at the volume series from 3D Classification, we can see that the volumes are not in total alignment; 3D Classification spent most of its capacity in finding similar volumes oriented differently relative to each other. This is not surprising, since we have not updated alignments since the initial C1 refinement of the entire encapsulin/encapsulated ferritin complex. We can place volumes back into register, as well as update particle alignments for each class, by using Align 3D Maps. \* Create an \[Align 3D Maps\](/processing-data/all-job-types-in-cryosparc/utilities/job-align-3d-maps.md) job. Activate the “Update particle alignments” parameter. Connect the All volumes output from the 3D Classification job to the “Maps to align (volumes group)” input. Connect the All particles output to the “Particles (all)” input. Connect the spherical mask to the Reference mask input. Finally, pick one of the 3D classes to serve as the reference map — this can be the best resolved class — and connect it to the Reference map input. The results are shown in the video of the orange (left) volume series below, compared to the un-aligned series from 3D classification in gray (right). ![](https://guide.cryosparc.com/files/cAq0sogOBo1bqRJmBkry) Figure 14. GIF demonstrating the results of aligning the twenty classes found by 3D Classification. The volume series in gray (right) shows the twenty classes directly from 3D Classification. The series in orange (left) shows the 20 classes after running Align 3D classes. ![](https://guide.cryosparc.com/files/vfo3pqCmMtLjLO54Mje0) Figure 15. Static version of Figure 14. All twenty classes are displayed as meshes and are overlaid. Note how the classes directly from 3D classification are significantly out-of-alignment. \#### Homogeneous Reconstruction Only \* Using the spherical mask from the previous 3D Classification, launch a Homogeneous Reconstruction Only job. This will produce a consensus structure after the previous Align 3D Maps job. ![](https://guide.cryosparc.com/files/TAgmQlfWZcBdupd7ymqp) Figure 16. This is the consensus structure after the twenty 3D classes have been aligned, particles have been combined across the classes, and the volume has been reconstructed via the Homogeneous Reconstruction Only job. The next step we must do is “effect” the non-point-group symmetry operators through the custom symmetry expansion step. ### B2: Custom Symmetry Expansion {% hint style="info" %} The next portion of the case study describes how CryoSPARC can be used to handle non-point-group symmetry. In this case, we would like to use CryoSPARC to superimpose the four copies of EncFtn, such that they can be combined into one structure for a final refinement, as \*a priori\* it is expected that each of these four units are indistinguishable and are the same structure. Symmetry-averaging that involves point group symmetries can normally be handled via symmetry expansion or refinement with symmetry enforced. Since this symmetry does not follow a point group, particular steps are required to obtain properly symmetry-averaged structures. {% endhint %} #### Mask generation Using ChimeraX, \[create four mask bases\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#mask-base-creation), each surrounding one of the four EncFtn units. This can be done most quickly using ChimeraX’s \`Segment Map\` tool. Our \[Mask Creation Tutorial\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera#method-one-volume-segmentation) covers this step in much more detail. Using the consensus map, lower the threshold value until exactly 4 disconnected discs are present in the density, one corresponding to each EncFtn, (but not too low that any of the EncFtn split into multiple disconnected blobs). The segmentation option \`Group by connectivity\` works well for this dataset, visible under the \`Segmenting Options\` drop-down menu: ![](https://guide.cryosparc.com/files/k2birk7MwesMDp6N79Js) This then produced four segments, each covering one of the EncFtn units. ![](https://guide.cryosparc.com/files/mlmc2rDHgsme471ebWLA) Figure 17. The result of a segmentation on the volume displayed in Figure 16. Following the remainder of the mask generation tutorial, we are able to generate four mask bases that will subsequently be imported into CryoSPARC. Save each of these mask bases with a format such as \`encftn\_maskbase1.mrc\`, \`encftn\_maskbase2.mrc\`, etc., in a directory. #### 1x Import 3D Volumes \* Build an Import 3D Volumes job. Set the path to a wildcard pointing to all four masks (for example, \`/path/to/directory/encftn\_maskbase\*.mrc\`). Change the type of imported volume to \`mask\`, and hit run. The job will import all four mask bases and present them as outputs. #### 4x Volume Tools The next step is to generate dilated and softly-padded masks from these four mask bases. \* To do this, we will need to run four Volume Tools jobs, one with each of the four masks connected as an input: | Type of input volume | \`mask\` | | --------------------- | ----------------------------------------------------------------------------- | | Type of output volume | \`mask\` | | Threshold | \*value selected in the mask generation section\* | | Dilation radius | anywhere between 2-8, depending on how large masks are desired; here chose 2. | | Soft padding width | At least 16; here chose 16. | ![](https://guide.cryosparc.com/files/SqtA4rqIe6YFjZuBjkiS) Tree view of the four volume tools jobs used to generate masks from the mask bases created in ChimeraX. \#### 4x Volume Alignment Tools The next task is to effect the non-point-group symmetry expansion step. In summary: 1. We use Volume Alignment Tools to shift each of the four masks to the center of the box. 2. Volume Alignment Tools simultaneously adjusts the positions of the volume and particles accordingly, to match each of the four shifted masks. Volume Alignment Tools also \*re-generates the unique identifiers (UIDs)\* of each particle, so that CryoSPARC knows to treat each image containing four copies of EncFtn as \*four separate observations\* of our protein of interest. 3. Align 3D Maps is then used to rotationally align (i.e. superimpose) all four volumes. Create four Volume Alignment Tools jobs. Connect each of the masks from the previous four Volume Tools jobs as the mask inputs. Connect the volume and particles from the homogeneous reconstruction only as the volume and particle inputs to each of the Volume Alignment Tools jobs. Finally, set the following parameters and run the jobs: | Re-center to mask center of mass | On | | -------------------------------- | -- | | Reassign UIDs | On | #### Align 3D Maps Next, we use Align 3D Maps to correct the rotational mis-alignment of the four volumes from the previous Volume Alignment Tools jobs. This step completes the custom symmetry expansion. \* Create an Align 3D Maps Job. \* Pick one of the four EncFtn volumes from one of the previous Volume Alignment Tools jobs; this volume will serve as the reference, and the other three will be aligned to it. This will establish the orientation of the consensus of the subsequent 3D classification. (This orientation is not the final one that will be used for the highest resolution refinement, as when we later incorporate D5 symmetry, we will have to align the consensus to the D5 symmetry axes. For now, refinement is proceeding in C1, and we are ignoring the D5 symmetry until we have a cleaner subset of particles.) \* Connect this reference volume and the accompanying mask to the “Reference Map” and “Reference Mask” inputs, respectively. Leave the “Maps to align (volumes group)” input empty. \* Enable “Update particle alignments” parameter. Connect the four EncFtn volumes from each of the Volume Alignment Tools jobs as connections under the “Maps to align (individual volumes)” group. Finally, connect each of the particle sets corresponding the four EncFtn volumes as individual connections under “Particles (map to align, connection X).” Phew! Now run the job. ![](https://guide.cryosparc.com/files/jWqhRAvJXopMwejQuMWS) Figure 18. Before Align 3D Maps — volumes are centered but not superimposed ![](https://guide.cryosparc.com/files/2kJ1zLIcIOvL40x4jqs2) Figure 19. After Align 3D Maps — volumes are superimposed and have the same rotational alignment \### B3: Group Re-alignment on Encapsulated Ferritin At this stage of processing, we now have all of the EncFtn superimposed. However, the particle stack is quite dirty — there are many junk particles, as evidenced by the previous 3D Classification results. The first step we’ll do is repeat 3D Classification, this time using the expanded particle dataset and a mask covering only one encapsulated ferritin. 3D Classification is preferred over local refinement at this stage for a few reasons. 1. 3D Classification is less sensitive than local refinement to junk. When dirty particle stacks are given to local refinement, a common outcome is artefacts and overfitting. When dirty particle stacks are given to 3D Classification, it is often the case that poor particles can be separated from good particles reasonably well via their class assignments 2. The particle stack is (a) contaminated by lots of junk particles and (b) particles’ alignments are far away from coherently superimposing particles onto one rigid structure, as we will see 3. When alignments are far away from their optimal values, Local Refinement will struggle to align the particles 1. Heterogeneous Refinement and Local Refinement both attempt to estimate alignments on a per-particle basis, which can be problematic when there is still a lot of junk, and when alignments are far off from their optimal values. 2. 3D Classification freezes alignments, thus is not able to use alignments as a free variable to overfit. 3. To overcome the fixed alignments of particles, 3D Classification can be combined with Align 3D Maps to allow for re-alignment of classes on a \*volume\* basis. #### Group Re-alignment (Repeat x2) The next step comprises of 3D Classification followed by Volume Alignment Tools and Align 3D Maps; this step will be repeated \*\*twice\*\* in order to iteratively improve the quality of our classification. \*\*3D Classification\*\* \* Build a \[3D Classification\](/processing-data/all-job-types-in-cryosparc/variability/job-3d-classification-beta.md) job. Connect the \`Particles for map {0,1,2,3}\` inputs all as connections to the input particle group. Leave the initial volumes and focus mask inputs empty. Connect the mask from the reference volume chosen in the previous step to the Solvent mask input. Use the following parameters: | Number of classes | 20 | | -------------------------------- | ---- | | Filter resolution (Å) | 6 | | O-EM batch size per class | 2000 | | O-EM learning rate init | 1 | | O-EM learning rate half-life (%) | 0 | | Force hard classification | On | Despite aligning each of the encapsulated ferritin to a common reference, the volumes have \*significant\* diversity both in their position and contents! ![](https://guide.cryosparc.com/files/u9ySsUIqXlBnt5pYhqri) Figure 20: Movie of the twenty Encapsulated Ferritin classes directly from the 3D Classification. Note the diversity in position of the twenty classes. Repeating Align 3D Maps (described in the next sub-section) will help position these volumes back into register, as best as possible, and set us up best for a final Local Refinement. \*\*Volume Alignment Tools (D5 symmetry alignment)\*\* Before running Align 3D Maps, we will use Volume Alignment Tools to align the structure to the D5 symmetry axes. This is in preparation for the final local refinement we’ll do to high-resolution. \* Create a Volume Alignment Tools job. \* Pick the class from the previous 3D Classification, that shows the strongest 5-fold symmetry. Connect this class to the volume input, and connect its corresponding particles to the particles input. Connect the solvent mask from 3D Classification to the mask input. \* Activate the “Do symmetry alignment” parameter. Set the symmetry string to “D5”. Run the job. \*\*Align 3D Maps\*\* \* Create an Align 3D Classes job. Activate the “Update particle alignments” parameter. Connect the \`All volumes\` group from the 3D classification to the \`Maps to Align (volumes group)\` input group. Connect the \`All Particles\` group from 3D classification to the \`Particles (all)\` input group. Use the volume and mask from the previous Volume Alignment Tools job as the reference map and reference mask. Run the job, and observe that the volumes are much closer to alignment than previously: ![](https://guide.cryosparc.com/files/GlJSgyibN5GZERMCkKIu) Figure 21. Movie of the 20 Encapsulated Ferritin classes after running Align 3D. Note that the twenty classes are in-register. After two iterations of Group Re-alignment, much more detail is beginning to form in the classes. Classes can be visualized by downloading the volume \`series\` from the Align 3D Maps job: ![](https://guide.cryosparc.com/files/LX7mzwwwvRqFSzIAhtZF) Inspect the classes, and note down which classes are of the intact EncFtn structure, show clear 5-fold symmetry, and are not at low resolution. Below is an example of the classification for 20 classes, with classes selected for further refinement highlighted in blue. ![](https://guide.cryosparc.com/files/wXT7muwgvXV7dboLudSO) Figure 22. Selected classes from the most recent 3D Classification. The particles assigned to the blue highlighted classes are carried forward into Local Refinement. \### B4: Local Refinement \* Create a \[Local Refinement\](/processing-data/all-job-types-in-cryosparc/local-refinement.md) job. Connect each particles group corresponding to each of the selected classes from the previous Align 3D Maps job. Connect the volume and mask from the latest upstream Volume Alignment Tools job to the Initial Volume and Static Mask inputs, respectively. Use the following parameters: | Use pose/shift gaussian prior during alignment | On | | ----------------------------------------------- | -- | | Standard deviation (deg) of prior over rotation | 6 | | Standard deviation (A) of prior over shifts | 3 | | Re-center rotations each iteration? | On | | Re-center shifts each iteration? | On | | Symmetry | D5 | | Number of extra final passes | 0 | With selecting a good subset of 3D classes, we retained \\~445k of \\~1,009k particles for this Local Refinement. This subset refined to a resolution of \\~2.8 Å, compared to previous Local Refinement of the encapsulated ferritin reaching only in the 6-9 Å. To help understand these results, it’s helpful to examine what changed from the initial Homogeneous Refinement job on all four EncFtn molecules. Why did it work better now? 1. The particles now had accurate-enough starting orientations. Earlier in the workflow, orientations were very poor, and likely too far away from their optimal values. \* Resolving the internal orientation diversity via group re-alignment was crucial to accomplish this. Group re-alignment allowed us to put particles in register, before the reference was high-resolution enough to allow for per-particle-pose estimation 2. Local Refinement also worked better because there was minimal heterogeneity — the broken/misaligned ferritin classes were removed in this final 3D Classification. 3. Finally, Local Refinement had access to a greater number of particles, due to the custom symmetry expansion and D5 symmetry enforcement. ## Encapsulated Ferritin Density Comparing our density map from this case study to the previously published density map, we can see that accounting for the challenging symmetry of this sample paid off! ![](https://guide.cryosparc.com/files/Qwrl17czBJb6tO7jrvRu) Figure 23. Comparison between the map of encapsulated ferritin uploaded to EMDB from this dataset (EMDB-13608, left) and the density reconstructed in this case study (right). To see if this density map was plausible, we re-refined the protein sequence from the atomic model \[PDB 5N5F\](https://www.rcsb.org/structure/5n5f). The 5N5F atomic model was obtained by \[Didi He, et al. (2019)\](https://pubmed.ncbi.nlm.nih.gov/30837306/) via Phenix refinement into a 2.1 Å map from x-ray diffraction. Though this structure \*is\* \*encapsulated ferritin\* from the same species of bacteria as the cryo-EM map, It wasn’t known to us whether we could expect the conformation of 5N5F to be identical to that of our density map. Possibly because the encapsulated ferritin in the cryo-EM sample was imaged inside of encapsulin (instead of being crystallized). ![](https://guide.cryosparc.com/files/7kPK5Sey09j23PLhr9jx) Figure 24. Comparison between the atomic [model entry 5N5F](https://www.rcsb.org/structure/5n5f) from the PDB (left) with a re-refined model (right) into the density map obtained in this case study. The density from this case study is shown in blue and is overlaid on both models. The above figure shows the density map obtained from this case study, sharpened to a B-factor of -60. This is overlaid on the 5N5F atomic model from the PDB (left column), and the re-refined atomic model (right side). The additional density present near the N-terminus of the AA sequence enabled modelling an extra three residues (GLU5, SER4, and SER3) that were not present in the atomic model from XRD. And that’s a wrap! This case study highlighted how to use CryoSPARC to handle the unique geometry and symmetry of the encapsulated ferritin dataset. Further standard processing workflows that weren’t explored in this case study could further improve results, for example: \* Repeated 3D Classification to remove more junk \* Reference Based Motion Correction \* Local CTF (defocus) refinement \* Process \*all\* movies in the dataset \*\*\* --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-encapsulated-ferritin-empiar-10716.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification.md). # Tutorial: 3D Classification {% hint style="warning" %} Note that the 3D Classification job was substantially improved in CryoSPARC v4.0. This tutorial describes the changes and new behaviour in detail below. {% endhint %} ## Introduction \[3D Classification (BETA)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-3d-classification-beta) is a tool in CryoSPARC v3.3+ that provides a way to distinguish discrete heterogeneous states, or classes, from single particle cryo-EM data. Namely, this job currently implements 3D classification \*without alignment (i.e., realignment of particle orientations or shifts)\* through a hybrid online and batch expectation maximization algorithm. By avoiding the computationally burdensome job of realigning particles and relying on higher-order interpolation rather than zero-padding of volumes, 3D Classification can efficiently separate particles into a large number of classes for further downstream analysis at high speed and without very large GPU memory requirements. Furthermore, unlike Heterogeneous Refinement, this job does not require any 3D maps for initialization. Instead, we provide two different initialization modes that can 'bootstrap' reasonable initializations via back-projection. Finally, we also allow for a (soft) mask input to 'focus' the classification on a specific region of heterogeneity, ignoring variation that may be present elsewhere. \*\*In CryoSPARC v4.0, 3D Classification was updated with several significant modifications to the underlying computational algorithm, accepted inputs, initial parameters, and diagnostic plots. Accordingly, this tutorial has also been modified with new salient considerations, and with analysis of two new representative datasets that demonstrate both the power and the limitations of the updated job. Please see the\*\* \[\*\*job guide page\*\*\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-3d-classification-beta) \*\*for a detailed list of new 3D class features included in v4+.\*\* ![](https://guide.cryosparc.com/files/UFVEvdrGVmX8QcmMdURH) 10 states recovered from (focussed) 3D Classification of the voltage-gated sodium channel (Xu et al., 2019). Density shown at two different thresholds. Data from EMPIAR-10261. \## Usage The 3D Classification job has one primary input requirement: particles with orientations and shifts in the \`alignments3D\` key. A typical pipeline may look as follows: 1. \*Particle extraction, 2D classification, motion correction, CTF computation, etc\* 1. Make sure to remove as many 'junk' particles as possible, though it may also be feasible to use the 3D Classification job itself to identify junk classes and remove associated particles 2. \*(Single- or multi-class) \`Ab-initio reconstruction\`;\* 3. \*(Optional) \`Homogeneous refinement\`\* \*\*\*or\\\*\\\*\\\*\\\* \\\*\\\*\\\*\\\*\\\*\\\*\\\*\\\*\`Non-uniform Refinement\`\*\* to find improved alignments for a final set of particles;\* 4. (\*Optional, \*\*updated in v4.0\*\*\*) \*Mask generation:\* 1. Focus mask (\[see our guide on mask generation in Chimera\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement/mask-selection-and-generation-in-ucsf-chimera)); 2. Solvent mask from \`Homogeneous refinement\` or \*\*\*\`Non-uniform Refinement\`\*\*\* 5. \*\*\`3D Classification (BETA)\`\*\* with particles from step 2 or 3, and a focus/solvent mask from step 4. 6. Further refinement of (a subset of) class volumes ## Salient parameters ### \*\*General\*\* \*\*\`Number of classes\`\*\* 2-100+. Unlike Heterogeneous Refinement, this job can feasibly classify large datasets into a large number of classes. \*\*For example, as of CryoSPARC v4.0, we can classify a \\~1.2 million particle dataset (EMPIAR-10077) into 100 classes in approximately 8.5 hours (exclusive of final output checks) on the\*\* \[\*\*CryoSPARC testing hardware\*\*\](https://guide.cryosparc.com/live/performance-metrics#hardware-configurations-used-for-benchmarking)\*\*.\*\* \*\*\`Target resolution\`\*\*2-10Å. This will define the 3D map box size and consequent Nyquist cut-off resolution. The resolution should represent a reasonable size scale at which the heterogeneity is expected, while being as low (i.e., numerically large) as possible to exclude noise. Computation time will also increase as resolution is increased. \*\*Note that in v4.0, each class may also be low-pass filtered below this cut-off, in accordance with its computed FSC curve.\*\* \*\*\`Use FSC to filter each class\`\*\* (default: on). Starting with v4.0, 3D classification now has the (recommended) option to filter each class using its intra-class Fourier Shell Correlation. During online EM, we use a sliding window approach to FSC filtering to avoid computing FSCs with small batch sizes. Accordingly, we apply FSC regularization at every O-EM iteration, but update the per-class FSC curves every 10th iteration (including iteration 0) using a decaying sum of sufficient statistics from the past. During F-EM iterations, FSC curves are re-computed every iteration. \*\*\*Updated in v4.1:\*\*\* \*\*\`Per-particle scale\`\*\*(default: optimal). Starting with v4.1, 3D classification now has the option to optimize per-particle scales with respect to the fixed consensus reconstruction prior to starting classification. By default this is turned on, though upstream scales ('input') or a constant scale of 1.0 ('none') can also be selected. Refer to the \[considerations below \](#effects-of-particle-scale-factors-and-anisotropic-magnification)for more information about these settings. ### \*\*Online Expectation Maximization\*\* \*\*\`Number of O-EM epochs\`,\`O-EM batch size (per class)\` , \`O-EM learning rate init\` :\*\* These three parameters will have coupled effects on the variety and quality of the classes at the end of Online Expectation Maximization (O-EM). For a fixed number of epochs, reducing the batch size will increase the amount of O-EM iterations, the effect of which will also depend on the learning rate. In general, if you observe unexpected class ‘collapse’ during O-EM, we suggest reducing the learning rate in 0.1 increments, and/or reducing the batch size. If the average Class ESS is not near 1 at the of O-EM, we suggest increasing the number of epochs. ### \*\*Full-Batch Expectation Maximization:\*\* \*\*\`Convergence criterion (%)\`:\*\* this parameter determines when to stop F-EM based on the amount of particles that switch classes from the previous iteration to the current one. We find that leaving this at 2% works well in many cases. For more difficult datasets, this may need to be increased to account for some particles that switch persistently because they cannot be classified with high certainty. Alternatively, you can turn on the secondary convergence criterion discussed below. \*\*\`RMS density change convergence check\`:\*\* monitor the root mean square (RMS) of the difference between class volume densities across iterations. For more difficult classification tasks, this number can be quite low despite a relatively high number of class switches (e.g., 5% +). In effect, particles may shuffle around classes, but have no significant effect on the class volumes. To ensure that classes with very few particles don’t have disproportional effect on this number, we computed a weighted average across classes, with weights set based on the relative size of the class. \*\*\*Updated in v4.0:\*\*\* Other parameters such as \*\*\`Initialization mode\`\*\* and \*\*\`Class similarity\`\*\* may also affect classification, but we have not found their impact to be significant in our testing. ## Important considerations #### \*\*Source of\*\* \`alignments3D\` This job can classify any input particles with the \`alignments3D\` key set (e.g., from an ab-initio job, from homogeneous refinement, from an imported EMPIAR particle dataset, etc.), however the quality of the alignments may affect the resulting classes. #### \*\*Solvent and Focus Masks\*\* During the expectation step of both full and online EM, we evaluate the likelihood of each particle under the following class volume $$ V\\\_k \\leftarrow S \\\* (F \\\* V\\\_k + (1-F)\\\*\\bar{V}), $$ where $$S$$ is the solvent mask, $$F$$ is the focus mask, $$\\bar{V}$$ is the consensus volume (computed at the outset and fixed), and $$V\\\_k$$ is the volume associated with class $$k$$. In words: At each iteration, each 3D class volume is masked by the focus mask, and the voxels outside the focus mask are replaced with the consensus reconstruction density voxels (rather than zero). This result is then masked by the solvent mask, and the voxels outside the solvent mask are replaced with zero. If the focus mask is not supplied, we set $$F=1$$and use $$ V\\\_k \\leftarrow S \\\* V\\\_k. $$ If $$V\\\_k$$ contains a micelle, and no focus mask is supplied, the heterogeneity present within the micelle itself will usually dominate the classes and prevent 3D classification from identifying biologically relevant heterogeneity. Whenever possible, we recommend supplying a focus mask. #### \*\*Effects of Particle Scale Factors and Anisotropic Magnification\*\* The class volumes produced by 3D classification may also be affected two other important elements: 1. \*\*Anisotropic magnification\*\*\\ Anisotropic magnification will cause class volumes to look stretched along orthogonal axes (often resulting in a ‘wobbling’ effect when classes are animated sequentially, as below). Although 3D classification does not currently estimate anisotropic magnification within the job, it can use upstream estimates encoded in the particle stack to compensate for this effect. If you suspect anisotropic magnification is affecting your results, you can run a \`Global CTF refinement\` job and reconnect its output to the 3D classification job, ensuring that \*\*\`Correct anisotropic magnification\`\*\* is turned on. ![](https://guide.cryosparc.com/files/umEuSJPM44KANjaD4W55) Anisotropic magnification causes classes to 'wobble'. 2\\. \*\*Particle Scale Factors\*\* 🆕 \*\*As of CryoSPARC 4.1, 3D Classification includes built-in per-particle scale optimization via the parameter\*\* \`Per-particle scale\` \*\*. By default, this parameter is set to\*\* \`optimal\`. \*\*In this mode, optimal per-particle scales are computed with respect to the consensus volume and fixed particle alignments prior to classification. This procedure, combined with some algorithmic changes in v4.1, should largely avoid the convergence behaviour described below.\*\* \*\*Note that in some cases it may still be beneficial to use\*\* \`input\` \*\*scales obtained via a refinement job prior to 3D classification, as these will be optimized simultaneously with alignments and may therefore differ from those computed herein (which use fixed alignments).\*\* \*\*Finally, in some further cases (e.g., in data with significant compositional/discrete heterogeneity), it may also be useful to set this parameter to\*\* \`none\` \*\*. In these cases, per-particle scale optimization may make it more difficult for 3D classification to separate classes with missing/additional density.\*\* Even if two particles contain the same signal, their relative scale (i.e., mean intensity) may differ due to ice thickness and other factors. This can have a significant effect on 3D classification. Similar to anisotropic magnification, we don’t compute these scales within the 3D classification job itself, but the job can use optimal scales from an upstream job to obviate their effect. A common manifestation of unequal particle scales is dramatic ‘reshuffling’ during F-EM iterations. If you observe this, (re-)run a \`Homogeneous refinement\` job with \`Minimize over per-particle scale\` turned on. If per-particle scale factors are indeed an issue, you may observe a multi-modal scale factor histogram (as seen below). The new particle stack from the refinement job can then be input to 3D classification which will account for these scale factors. ![](https://guide.cryosparc.com/files/pQHuoCkLaWh0mlsYlt3M) An example of a class flow diagram from a 3D classification job where particle scales are not equal. We often find that unequal scales cause the F-EM iterations to dramatically ‘shuffle’ the classes. ![](https://guide.cryosparc.com/files/YsEggPSMDGX6qVj3Lf8x) A bimodal distribution of per-particle optimal scales computed with a homogeneous refinement job. Computing these per-particle scales prior to connecting particles to a 3D classification job can significantly affect the resulting 3D classes. \*\*Effective sample size (ESS) and soft class assignments\*\* The ESS is a simple measure of the extent to which a discrete probability distribution is 'dispersed.' In 3D Classification, the \*class\* ESS is evaluated over the posterior of class assignments for each particle. Numerically, the per-particle class ESS is equal to the inverse of the sum of squared class probabilities and ranges from 1 to the number of classes. A value near the number of classes indicates that the class posterior is near a uniform distribution, while a value near 1 represents a \\\`hard' selection of a single class. \*\*\*In v4.0, we now display a histogram of class ESS values as part of the standard suite of diagnostic plots.\*\*\* ![](https://guide.cryosparc.com/files/YNGvNeerzH8vUiKlGly9) Per-particle Class ESS Histogram displayed in 3D Classification (≥v4.0). \*\*Weighted back-projection and further refinement\*\* In 3D Classification, the final output volumes are constructed using a weighted back-projection with weights on each particle defined based on the class posterior. This means that although the output class particle sets are disjoint, each particle may contribute to multiple (or all) volumes. When the dataset-wide mean class ESS is near 1, this effect is minimized. Nevertheless, the volumes themselves are primarily useful as a visualization of each class, and further refinement should be done on the relevant particle sets for a final reconstruction. \*\*In v4.0, 3D Classification includes the option to disable this ‘soft back-projection’ by turning on the parameter named \`Force hard classification\`.\*\* ## Example Results \*\*and Analysis\*\* We present 3D classification results from several publicly available datasets. All volumes are visualized and animated using \[ChimeraX\](https://www.cgl.ucsf.edu/chimerax/). ### \[EMPIAR-10077\](https://www.ebi.ac.uk/empiar/EMPIAR-10077/) \*Ribosome with selenocysteine delivery in E. coli, Fischer et al. (2016)\* This data captures a ribosome complex binding with a ligand. In the original publication, 6 distinct states (see Figure below) were found. ![](https://guide.cryosparc.com/files/GYFob1XDZHWCQUr5IVrM) Figure 1a, Fischer et al. (2016). \#### Inputs \* \*Particles\* \* Part 1: 1.19 million particles from \`Homogeneous Refinement\` \* Part 2: 1.19 million particles from \`Global CTF Refinement\` \* \*Solvent mask\* \* Mask from \`Homogeneous Refinement\` #### \*\*Non-default parameters\*\* \* None #### \*\*Part 1: \*\*\*\*\*Classification without anisotropic magnification correction\*\*\* Without correcting for anisotropic magnification, the classes include biologically-salient conformations but also display a characteristic ‘wobble’ (discussed in important considerations above): ![](https://guide.cryosparc.com/files/w3SbjpicsoP38xXNYSND) Results of a 10 class 3D Classification job without correcting for anisotropic magnification Here, in another 3D classification run, this wobbling is even more pronounced and most noticeable when there is little other conformational change: ![](https://guide.cryosparc.com/files/umEuSJPM44KANjaD4W55) Results of a (different) 10 class 3D Classification job without correcting for anisotropic magnification — here the ‘wobble’ is very evident. \*\*Part 2: \*\*\*\*\*Classification without anisotropic magnification correction\*\*\* To correct for this, we ran a \`Global CTF refinement\` job (with \`Tilt\` , \`Trefoil\`, and \`Anisotropic Mag.\` fits turned on) on the particle stack: ![](https://guide.cryosparc.com/files/lECeHJYzsT4UmuNqfzSw) Then, using the new particle output group, we re-ran 3D classification which produced 10 classes that were no longer stretched in the same way: ![](https://guide.cryosparc.com/files/cVBaaBaClxwU5gnx7VbL) Results of a 10 class 3D Classification job correcting for anisotropic magnification Note that we still see one class (class 2) with significantly more density: ![](https://guide.cryosparc.com/files/xEzUo0qYtDB1sfYZjbVM) This may be indicative of some particle scaling issues. We discuss one way to account for these in the next dataset. ### \[EMPIAR-10697\](https://www.ebi.ac.uk/empiar/EMPIAR-10697/) \*Human RNA polymerase III, Girbig et al. (2021)\* ![](https://guide.cryosparc.com/files/sgV6M0udsh5oxHLwGMd8) Two different conformations of the clamping mechanism of the RNA polymerase, from Fig. 2c, Girbig et al. (2021). \#### Inputs \* \*Particles\* \* Part 1: 166K (polished) particles from \[EMPIAR 10697\](https://www.ebi.ac.uk/empiar/EMPIAR-10697/) \* Part 2: 166K particles from \`Homogeneous Refinement\` (Minimize over per-particle scale on) \* \*Solvent mask\* \* Part 1: mask from \`Homogeneous Reconstruction Only\` job \* Part 2: mask from \`Homogeneous Refinement\` #### \*\*Non-default parameters\*\* \* None #### \*\*Part 1: \*\*\*\*\*Classification with equal particle scales\*\*\* \*\*As of CryoSPARC 4.1, 3D Classification includes built-in per-particle scale optimization via the parameter\*\* \`Per-particle scale\` \*\*. Please see the note above regarding updated convergence behaviour in v4.1.\*\* We ran 3D classification with the default parameters on the imported dataset from EMPIAR 10697 consisting of approximately 166K particles. With these inputs, the job required over 30 F-EM iterations to converge below the standard threshold of 2% class switches. We often observed significant class ‘shuffling’, as you can see below. ![](https://guide.cryosparc.com/files/pQHuoCkLaWh0mlsYlt3M) After 67 total iterations, the job did converge, with the following class distributions: ![](https://guide.cryosparc.com/files/TBQXz0XmOCucLyGxfM16) At first it may seem that we’ve found a number of ‘low population’ classes. However, upon further inspection we see that many of these states are similar: ![](https://guide.cryosparc.com/files/cmofK42unBxSLp6mNWDk) Ten (of ten) classes from a 3D Classification job run on data from EMPIAR 10697, with no modification to particle scales. \*\*Part 2: \*\*\*\*\*Classification with optimized per-particle scales\*\*\* Investigating this further, we see that if we run \`Homogeneous refinement\` on the particles with \*\*\`Minimize over per-particle scale\`\*\* turned on, we see the following scale distribution: ![](https://guide.cryosparc.com/files/YsEggPSMDGX6qVj3Lf8x) This type of distribution is indicative of multiple scale ‘modes’ which will affect 3D Classification. Indeed, if we re-run the job with these scale-optimized particles, the 3D classification job converges within 2 F-EM iterations to the following class distribution: ![](https://guide.cryosparc.com/files/pGWdniLXNLL9TmpYhUjV) This results in four distinct classes, animated below: ![](https://guide.cryosparc.com/files/GwzBEMTIjk35c2TDqH1w) Four states recovered using 3D Classification of human RNA Polymerase III (EMPIAR 10697), after per-particle scale optimization. \### \[EMPIAR 10425\](https://www.ebi.ac.uk/empiar/EMPIAR-10425/) \*A.baumannii MlaBDEF complex bound to AppNHp, Mann et al. (2021)\* ![](https://guide.cryosparc.com/files/31P3JoJvctzTu1yDWlys) The MlaBDEF complex, from Fig 1c, Mann et al. (2021). ![](https://guide.cryosparc.com/files/g0S1cF9p1qFSB6T8dahb) Different MlaB binding configurations, from Supp. Fig. 2, Mann et al. (2021). \#### \*\*Inputs\*\* \* \*Particles\* \* 80K particles from \`Non-uniform refinement\` (after particle picking, 2D class, ab-initio) \* \*Solvent mask\* \* Mask from \`Non-uniform refinement\` \* \*Focus mask\* \* Mask created using ChimeraX (following the CryoSPARC mask generation tutorial) ![](https://guide.cryosparc.com/files/J7Qx5Q7tsZsJ6R3qlkaf) ![](https://guide.cryosparc.com/files/wJIypapGkV45F43dCplX) Focus mask isolating the binding sites of MlaB. \#### \*\*Non-default parameters:\*\* \* Part 1: \* Classes: 5 \* Part 2: \* Classes: 5 \* Force hard classification: True #### \*\*Part 1:\*\* \*\*\*Classification with a focus mask on MlaB binding sites\*\*\* Applying 5-class focussed classification on this dataset results in the following classes (after 6-FEM iterations): ![](https://guide.cryosparc.com/files/l2hMplcgDckoIDKry9RX) ![](https://guide.cryosparc.com/files/C1vMORKlWwte6eqHLFDG) Note the small hump around 2 in the ESS histogram. This indicates that several thousand particles still have significant probability of belonging to two classes — these particles are ‘spread’ about two volumes. When we take a look at the volumes themselves: ![](https://guide.cryosparc.com/files/T5t2FcqchHwyU1PjYn6J) ![](https://guide.cryosparc.com/files/PwCcBkmVECtC25h09X1P) We see that there is no class that contains no MlaB units. ![](https://guide.cryosparc.com/files/qIDJ9eUbqxPkjPmAEQ0Q) Five classes from 3D classification performed on EMPIAR 10425 with a focus on mask on the MlaB binding sites. In this type of case, it might be useful to observe what happens when we turn off weighted backprojection, and instead classify particles using ‘hard classification’. #### \*\*Part 2:\*\* \*\*\*(Hard) Classification with a focus mask on MlaB binding sites\*\*\* With hard classification turned on, we see a very different class distribution: ![](https://guide.cryosparc.com/files/ffIGKTZFerqE401kyl9q) A plurality of the particles are now in class 4, which is a class that may have no MlaB units (though this requires further investigation). ![](https://guide.cryosparc.com/files/NsParYAbA7tCQrfqdoAW) ![](https://guide.cryosparc.com/files/aS1Ci6dlhzh4B8y87vpH) Five classes from 3D classification performed on EMPIAR 10425 with a focus on mask on the MlaB binding sites. In this case, hard classfication is turned on, and we see the potential presence of a ‘no binding’ class. Thus, for data where a significant potion of particles cannot be classified into a single class with certainty (i.e., their class ESS is ≥ 2), turning on hard classification may help uncover classes that would otherwise be ‘smeared’ out by this uncertainty. ## Citations Fischer, Niels, et al. "The pathway to GTPase activation of elongation factor SelB on the ribosome." \*Nature\* 540.7631 (2016): 80-85. Girbig, Mathias, et al. "Cryo-EM structures of human RNA polymerase III in its unbound and transcribing states." \*Nature structural & molecular biology\* 28.2 (2021): 210-219. Mann, Daniel, et al. "Structure and lipid dynamics in the maintenance of lipid asymmetry inner membrane complex of A. baumannii." \*Communications biology\* 4.1 (2021): 1-9. Xu, Hui, et al. "Structural basis of Nav1. 7 inhibition by a gating-modifier spider toxin." \*Cell\* 176.4 (2019): 702-715. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-a-ligand-bound-gpcr-empiar-10853.md). # Case study: End-to-end processing of a ligand-bound GPCR (EMPIAR-10853) ## Introduction In this tutorial we will work step-by-step through the processing pipeline for an active G-protein coupled receptor with a ligand, G-protein and antibody fragment bound, using a dataset originally processed by \[Cao \*et. al\*\](https://doi.org/10.1038/s41586-021-04126-6) and deposited as \[EMDB:24896\](https://emdb-empiar.org/EMD-24896) and \[PDB:7s8l\](https://pdbe.org/7s8l). The raw data are publicly available for download as EMPIAR-10853, and this case study is written so that you can replicate these results yourself. We selected this dataset to provide a processing pipeline for a target with flexibility in the region of ligand-binding, and heterogeneity in the ligand-binding pose. In particular, this case study provides strategies and ideas about how to get a higher quality final particle stack, which we found was essential for downstream local refinement, and focussed classification steps. Image processing was performed using CryoSPARC v4.7. The Mas-related G-protein coupled receptor member X2 (MRGPRX2) receptor is a member of the G-protein coupled receptor (GPCR) protein family that are characterised by their 7 alpha helical transmembrane helices. GPCRs are predominantly found on the cell surface and are important in cell signalling and as therapeutic targets. The MRGPRX2 receptor is largely found on the surface of mast cells and is activated by agonist ligands. Upon activation it binds to Gq (a heterotrimeric G-protein). This in turn activates the mast cell to degranulate and this process plays a role in host defence, inflammatory diseases and pseudo-allergic drug hypersensitivity. In this sample, Gq and a single-chain variable fragment (scFv16) were added to the receptor, along with the agonist peptide ligand Cortistatin-14. We can calculate from the existing pdb 7s8l that the combined complex has a mass of \\~140 kDa and, but for cases where a molecular model is not available, the mass could also be ascertained from protein sequences (for example in \[UniProt\](https://www.uniprot.org/)) or by analysing a purified sample by mass spectrometry. From the structures obtained in \[Cao \*et. al\*\](https://doi.org/10.1038/s41586-021-04126-6), we know that this complex is a non-symmetric membrane protein complex containing a transmembrane receptor domain (TMD) and an intracellular domain formed of the G-protein that extends outside of the membrane, and that the expected ligand binding location is on the cytoplasmic side of the receptor domain. This protein was prepared in a detergent micelle containing lauryl maltose neopentyl glycol, glyco-diosgenin and cholesteryl hemisuccinate. !\[The overall structural features in PDB 7s8l that will be referred to in this case study.\](/files/v9DNbOma6bf132Gjo4Wg) The primary aim of this pipeline is to generate the highest quality map(s) to ascertain the binding pose(s) of the inhibitor peptide, Cortistatin-14. ## \*\*Setting up\*\* Before beginning this tutorial, you should \[create a new project and a workspace\](https://guide.cryosparc.com/application-guide-v4.0+/using-the-cryosparc-interface/projects-workspaces-and-live-sessions#creating-your-first-project) within that project. Download the movies and gain references to a location of your choosing. Our data is downloaded to a directory called rawdata in the project directory using the command: \`\`\`jsx cd /path/to/rawdata wget -m . \`\`\` ## 1. Movie import and preprocessing Now we need to get the data into CryoSPARC. These data were recorded over two days and for each day there is a separate gain reference and a slightly different total dose. \* Import the data as two separate jobs using an \[Import movies\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/import/job-import-movies) job. The \`Movies data path\` needs to match the location of the directory containing the downloaded \`.tif\` files for example \`rawdata/EMPIAR/10853/data/20210311\*.tif\`. The \`Gain reference path\` for the first day needs to be specified for example \`rawdata/EMPIAR/10853/data/CountRef\_20210311\_109\_000.mrc\`. Experimental information such as pixel size, total electron dose / A^2 and voltage are available in the EMDB entry \[24896\](https://www.ebi.ac.uk/emdb/EMD-24896). | Parameter | Setting | | --------------------------------- | ------- | | Flip gain ref & defect file in Y? | Yes | | raw pixel size (A) | 0.91 | | Accelerative Voltage (kV) | 200 | | Spherical Aberration (mm) | 2.7 | | Total exposure dose (e/A^2) | 50.74 | \* Repeat the process for the second day files (20210312) and changing the \`Total exposure dose (e/A^2)\` to 47.87. {% hint style="info" %} In this example the difference in dose is not very substantial between the two days of collection, but we will treat the two subsets of movies separately as an example of how to do this. In cases where movies are collected as a single session, have the same number of frames, and the total doses are very similar (less than \\~10% different), even if there are multiple gain references available, you might prefer to import all of the images together as a single job, using just one of the gain references, and perhaps setting the total dose to an average value, provided that all of the gain references have a similar appearance. {% endhint %} Next we want to correct for beam-induced motion during movie collection and to estimate the defocus values for each micrograph. \* Run \[Patch Motion Correction\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/motion-correction/job-patch-motion-correction), inputting both imported movie sets from above. Use \`save results in 16-bit floating point\`: true and increase \`Number of GPUs to parallelize\` depending on GPU availability. Using float16 format means that the output images will take up half of space than the default 32-bit floating point files. You may expect each job to take several hours on a single GPU. \* Run \[Patch CTF Estimation\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-estimation/job-patch-ctf-estimation) with default settings ## 2. \*\*Excluding poor micrographs from downstream processing\*\* Not all of the movies that are collected will be perfect; they can contain a variety of non-vitreous ice and contamination with ice crystals or other substances, excessive in-movie motion and ice that is too thick or too thin for your sample. We would like to avoid picking particles from poor micrographs because they are often of low value, and can require extra cleaning steps to remove after they are extracted. It is easier to exclude the worst micrographs at the start, by looking at the statistics generated by the Patch Motion and Patch CTF jobs. \* Use a \[Manually Curate Exposures\](https://guide.cryosparc.com/application-guide-v4.0+/interactive-jobs#interactive-job-manually-curate-exposures) job to exclude poor quality images on the basis of their CTF estimated micrograph statistics so they are not used for downstream processing. Input the “Micrographs processed” from Patch CTF, and queue the job. In the job card view go to the pink Interactive tab. {% hint style="info" %} You may notice that many of the images here after import have faintly darker stripes along the left and right sides of the images, (See Figure 1, left). This is likely due to the gain correction. You may prefer to generate a gain reference from the movies using a different software package rather than use the supplied gain reference files, but we found that including this sort of step did not improve the final map quality or resolution. {% endhint %} Select the upper and lower bounds for each parameter to exclude outliers, and browse thumbnails and diagnostics of the images to check the appearance of the excluded and included micrographs. For each parameter, try moving the slider or typing values and click “set threshold” to see how many images have been excluded. We set an upper limit for CTF fit resolution at 3.5 Å to avoid using poor quality images, an upper limit of 70 for Average Intensity to remove outlier images, and an upper bound on Relative Ice Thickness at 1.042 to remove low signal-to-noise images with thick ice. This is a nice dataset with relatively few poor micrographs, and overall we were left with 11,573 micrographs. !\[Figure 1. Example micrograph diagnostics and thresholding for average intensity. Filtering of micrographs on the basis of Intensity with an upper limit of 70. Example diagnostics are shown for an included (blue) micrograph and an excluded (red) micrograph.\](/files/UIZ8hCml7pxBinykAv0e) The above example shows the micrograph, power spectrum, CTF fit and motion for two example movies. The CTF fit and motion for both images look alright, but the thon rings for the red example are weaker, due to interference from the strong vertical lines in the movie. This might originate from a problem with gain correction of the detector for a few images. In other datasets it can also be helpful to set thresholds for average defocus, astigmatism, and full-frame motion distance, but it is a good practice to examine all of the available plots to see where outlier populations arise. ## 3. Micrograph denoising and Junk Detection #### 3A) Micrograph Denoising Particles can be hard to identify by eye due to factors such as thick ice, small particle size, carbon or graphene monolayers and low defocus, and these can make selecting thresholds for particle picks somewhat challenging. To improve the quality of template and blob picks, and make thresholding easier we use the \[Micrograph Denoiser\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/exposure-curation/job-micrograph-denoiser-beta). \* Create a \[Micrograph Denoiser\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/exposure-curation/job-micrograph-denoiser-beta) job, inputting the exposures\\\_accepted and set \`Number of CPUs to Parallelize\` to the number of CPUs available on your node. !\[Figure 2. An example micrograph with lowpass filtering and denoising.\](/files/pGUl8z85D5WQBClJJ8Mf) We can identity the particles much more easily after denoising and so the denoised images are suitable for Blob picking, Template picking, and Inspect Picks job types. #### 3B) Junk detection Micrographs often have different types of unwanted features in them, such as ethane contaminants, ice crystals, crystalline ice and gold or carbon support, which can be tricky to get rid of during classification steps. We can use the \[Micrograph Junk Detector\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/exposure-curation/job-micrograph-junk-detector-beta), released in v4.7, to analyze the micrographs and automatically detect different types of junk. Subsequently, we can reject particle picks that are on or near the junk regions. \* Create a\[ \](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/exposure-curation/job-micrograph-junk-detector-beta)\[Micrograph Junk Detector\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/exposure-curation/job-micrograph-junk-detector-beta)\[ \](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/exposure-curation/job-micrograph-junk-detector-beta)job, inputting the exposures\\\_accepted from the upstream Micrograph Denoiser job. !\[Figure 3. Example micrograph after Junk Detection, whole dataset junk statistics.\](/files/aenuALB6aXdAgBbuBOG2) The Micrograph Junk Detector has masked out in purple regions that contain extrinsic ice junk (ethane contaminants and ice crystals), and from the job statistics we can see that while almost every micrograph contains junk (overwhelmingly extrinsic ice junk), the total area that this junk occupies is relatively low at <4%. Dataset-level junk statistics might be helpful to diagnose patterns in junk appearance that may relate, for example, to sample prep changes. ## \*\*4. Blob particle picking\*\* This is a pretty large dataset, so to speed up generation of 2D templates, we are going to make a subset of 200 micrographs for initial blob picks. \* Run an \[Exposure Sets\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-exposure-sets-tool) Tool job, inputting your exposures from after the Micrograph Junk Detector, and set \`Split batch size\`: 200 and \`Split randomize\` : true {% hint style="info" %} We know this receptor has a G-protein and a scFv bound and has a total mass of 140 kDa. Suitable diameters for picking can be estimated by looking at the “ruler” in the Manually Curate Exposures job, on in ChimeraX from homologous deposited structures or AlphaFold models. The diameter does not need to be precise, and setting the diameter \\~10% larger than the estimated diameter of the particle can be beneficial to account for imperfect estimation, and the effects of defocus in the images. We estimate the longest diameter is around 120 Å based on PDB 7s8l. {% endhint %} Using the micrographs in split\\\_0, run a \[Blob picker\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-picking/job-blob-picker) job with a minimum particle diameter (A) 100, and Maximum particle diameter (A) 130. Not all of the picks made will be of good quality, and to remove egregious junk picks we can use a combination of automatic removal of junk picks by using the junk detection that we ran earlier, as well as manual thresholding of the pick scores. \* Run a second Micrograph Junk Detector job and input the exposures from the first Junk Detector job along with the blob picked particles, setting the \`Rejection distance from junk\`:30. {% hint style="info" %} The distance from detected junk where you want the junk detector to reject particles may depend on the type of junk present, the size of the particle, and the box size you want to extract. For larger particles and larger boxes, a larger value maybe more appropriate. {% endhint %} \* Run an \[Inspect Picks\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-picking/interactive-job-inspect-particle-picks) job and look at the first few micrographs. The overall goal here is to select as many particles as you can without picking empty ice or junk. You should see that there are very few picks on the junk regions as we already rejected those near the detected regions of junk. To remove any remaining junk picks, move the NCC and power score sliders until the lower bound removes empty ice picks, and bring the higher bound down to remove high contrast feature picks without seeing good-looking picks being discarded. Be sure to inspect micrographs of relatively low and high defocus when adjusting the power sliders, so that particles are being picked across the defocus range. You can expect to select \\~200k particles from a 200 micrograph subset. !\[Figure 4. Blob picks after filtering using the Junk detector and Inspect Picks. For the example shown ncc > 0.3 and local power > -5 and < 83.\](/files/PctlrjnWOBL1YeUGCJVz) As we performed Inspect picks with the denoised micrographs you may notice that the contrast is more similar across the range of defocus used than you would typically see with lowpass filtered images. ## 5. Blob pick extraction and 2D classification We expect the MRGPRX2 receptor protein to have a diameter of \\~ 120 Å, and this dataset was recorded with a defocus range extending up to around -2 µm. We want to ensure that we capture most of the delocalised signal caused by imaging with defocus, so we need to make sure the box is large enough. {% hint style="info" %} As the defocus is increased, the higher frequency components from particles are delocalised further out in real space due to the point spread function of the microscope. If too small a box is selected for extraction, some information about your particle is lost, and this may limit the attainable resolution. Conversely using an excessively large a box can lead to a lot of noise in the images, and this can also have a negative effect on the resolution of your reconstruction. As a very rough rule, a box of \\~1.5-2.5 x the diameter of your particle is often appropriate, however very high resolution data or data collected with high defocus may require a larger box. The box size must be an even integer of pixels, and it is best if you choose or downsample to a \[box size that is computationally efficient.\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/extraction/job-extract-from-micrographs#box-sizes-that-allow-for-efficient-processing) {% endhint %} \* Extract the particles using the \[Extract from Micrographs\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/extraction/job-extract-from-micrographs) job with \`Extraction box size (pix)\`: 256 (233 Å which is \\~2 x the particle diameter) and set \`Fourier crop to box size (pix)\`: 64. Select \`Save results in 16-but floating point\`: true to save on the disk space required. We will be using these particles to generate 2D class average templates, so we do not need them to be at high resolution. Therefore we can Fourier crop to a box of 64 and this will speed up the downstream jobs. Expect the number of extracted particles to be lower than the number previously selected, because CryoSPARC rejects particles where any part of the box is outside of the micrograph. Fourier cropping for 2D classification saves on disk space and may speed up caching and early particle cleaning jobs. Now that we have our particles extracted, we want to use 2D class averaging to crudely separate good particles from poor ones. \* Run a \[2D Classification\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/job-2d-classification) job with the extracted particles and the following settings: | Parameter | Setting | | ------------------------------------------- | ------- | | \`Number of 2D classes\` | 100 | | \`Maximum resolution (A)\` | 12 | | \`initial classification uncertainty factor\` | 1 | | \`Circular mask diameter (A)\` | 160 | | \`Circular mask diameter outer (A)\` | 180 | | \`2D zero pad factor\` | 1 | We don’t have a large number of particles, but in order to speed up the classification we set the maximum resolution at a limit where we would expect to be able to discern visually between particles and junk, and between different viewing directions, and we can also reduce the zero padding from 2 to 1. Adding a circular mask can help to keep the class averages in the centre, which is helpful for template picking. The initial classification uncertainty factor influences the diversity of particle views, or junk classes that are identified. A lower value (such as 1 as used here) tends to increase the number of different junk classes, and can be useful when the preparation contains largely one type of particle. In cases where the particle stack contains very little junk, a higher number (such as 5 to 10) may be helpful in separating views. !\[Figure 5. 2D classification of blob picked particles. The major expected class types (side and top views) and example 2D class averages, indicating those suitable for selection and rejection. The detergent micelle is highlighted in green in one class. The class highlighted in yellow appears to show a G-protein and fab that are not bound to the receptor.\](/files/bz2oq284OylPXiKrwMs3) In the class averages shown in Figure 5, most of them contain easily recognisable ovoid density for the detergent micelle, with an asymmetric protein domain protruding out on one side. We also observe some views of the micelle where the protruding domain is not that visible, and so the class averages look like a distorted ellipse. It is important not to exclude these views ( "top views"), as they can be relatively rare amongst GPCR cryo-EM datasets. {% hint style="info" %} While we don’t expect every view to be equally sampled (because of target interactions with the air-water interface or support, or ice thickness restriction) it is still important to have an adequately diverse set of selected particle views in the dataset to produce a reliable 3D volume. {% endhint %} \* Run a \[Select 2D Classes\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/interactive-job-select-2d-classes) job and select the class averages that contain views of the intact target like the examples above. These class averages already look OK and can be used directly as templates for picking without further cleanup. ## 6. Template particle picking, extraction and duplicate removal We can improve the picking quality by supplying 2D templates so that the picks better resemble the expected target. \* Create a \[Template Picker\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-picking/job-template-picker) job and connect the denoised micrographs and the selected templates from Select 2D Classes. Set \`pick on denoised micorgraphs:\` true and \`Particle diameter (A)\`: 170. In our processing the Template Picker yielded \\~14 million particles, but we want to remove the most offensive junk picks before extraction. 1. Run a Micrograph Junk Detector job and input the exposures from the previous Junk detector job along with the template picked particles, setting the \`Rejection distance from junk\`:30. 2. Run an \[Inspect Picks\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-picking/interactive-job-inspect-particle-picks) job and look at the first few micrographs. Move the NCC and power score sliders in the same manner as Section 4 to remove picks that look overly generous. !\[Figure 6. Junk detection and rejection. Example Junk detection and Template-picked particle rejection lose to junk. The accepted particles are shown with a white circle, and the rejected particles are shown with a red circle and partially transparent white fill.\](/files/wNVUnAEcYdbfOzFBVN3Z) \* Extract the particles in the same manner as Section 5. At this stage, we had 11.5M particles. ## 7. Separating junk from good particles with Ab-Initio and Heterogeneous refinements Our aim through any basic single particle cryo-EM processing is to separate junk from good particles, without losing valuable rare views. Sometimes it is better to skip 2D classification and go straight to 3D, to avoid losing rare views at the stage of class average selection. In order to separate the particles by \[Heterogeneous Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-heterogeneous-refinement), we need a good input volume and some junk volumes. We can generate those from the particles that we 2D classified in Section 5, but if you prefer to 2D classify the Template picked particles and use those instead, they should yield approximately the same result. As we already separated some good particles from bad ones after blob picking, we can use those particles to generate low resolution volumes in two separate jobs: 1. Run an \[Ab-Intio\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-reconstruction/job-ab-initio-reconstruction) reconstruction job (1) with \`number of classes\`: 1 and input the “particles selected” from your Select 2D job. 2. Run an \[Ab-Intio\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-reconstruction/job-ab-initio-reconstruction) reconstruction job (2) with \`number of classes\`: 5 and input the “particles excluded” from your Select 2D job. Once the jobs are complete, you can asses the volumes for each class by selecting the Volumes tab within the job, and inspect each volume one by one. You should find the volume from Ab-Initio (1) resembles a GPCR in a micelle, and the 5 from Ab-Initio (2) are less clearly defined volumes that are probably junk. If you find a second nice GPCR volume in Ab-Initio (2), then discard this volume from downstream processing. {% hint style="info" %} The number of junk classes required depends on the diversity of junk that might be present in the sample. Generally we have found 1-5 junk classes sufficient in most cases. {% endhint %} \* Run a \[Heterogeneous Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-heterogeneous-refinement) job (1) inputting all of the extracted particles from Section 6. Input the Ab-Initio models that you obtained above to classify into different classes and set \`Refinement box size (voxels)\`: 64 as this is the box size that the particles were Fourier cropped to, and using the default 128 would lengthen the job runtime. \* As we know which class was the best input volume, we can predict that this same class will be the best output volume, so we can queue up anther Ab-Initio job (3) inputting the particles from the best class from Hetero Refine 1, and setting \`Number of Ab-Initio classes\` : 5 and \`Num particles to use\` : 20,000. We didn’t find that we needed more than 20,000 particles for this job, but if the volume quality is not good you can try with more particles, e.g. 50,000-100,000. \* Inspect the volumes from Ab-Initio (3) to find the best volume(s). \* Run three further sequential Heterogeneous refinement jobs (2-4) using the volumes from Ab-Initio (3), each time taking forward the particles from the best class(es). We found that the percentage of particles found in the best class at each round of Heterogeneous Refinement were 48%, 63%, 92% and 95% and this left us with \\~ 2.3M particles. Generally when Heterogeneous Refinement yields \\~95% of particles in the best class then you are in the realms of diminishing returns and it is usually not worth running further rounds of this job. !\[Figure 7. Volumes from Ab-Initio and Heterogeneous Refinement jobs. Good volumes are coloured in magenta and poor volumes in shades of grey.\](/files/mPxX7l7TNg6uuBeZGU2Y) ## 8. Re-extraction at a larger box size and Non-Uniform Refinement Now that we have cleared out junk particles via Heterogeneous Refinement, we are ready to re-extract with a larger box size for the following reasons: 1. Using a higher pixel sampling (less Fourier cropping) extends the achievable resolution limit (Nyquist frequency) on the particle images. 2. Extracting particles after 3D alignments for example in Heterogeneous refinement means that they will be better centred in the box. 3. A smaller stack of particles (2-3 million compared to the original 14 million) will be read and cached faster for downstream jobs. As we have now aligned the particles we should also take care to remove any duplicates based on their updated shifts before we re-extract. \* Run a \[Remove Duplicate Particles\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-remove-duplicate-particles) job with \`Minimum separation distance (A)\`: 50 and using the particles from the good class(es) at the end of Section 7. \* Re-extract the non-duplicated particles with \`Extraction box size (pix)\`: 320 and \`Fourier crop to box size (pix)\`:256. We expanded the box size in Å to \\~2.5 x the particle diameter to capture more of the delocalised signal, but Fourier cropped back to 256 for computational efficiency. \* Input the particles into a \[Non-Uniform Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-non-uniform-refinement-new) job (Refinement 1) using one of the good GPCR 3D volumes from Heterogeneous Refinement with \`Minimise over per-particle scale\` on, and \`Dynamic mask start resolution (A)\` :1. {% hint style="info" %} The reason why we set the dynamic masking to start at 1 Å (meaning that it will never start, since we don’t expect to achieve a 1 Å resolution) is that we want to disable masking during refinement, and rely on the Non-Uniform regularization to adaptively exclude noise. This method can sometimes yield slightly better map quality with NU refinement. If you wish, you can compare the results with and without this setting to see how it affects the map quality and statistics. {% endhint %} {% hint style="info" %} Per-particle scale is a function that compares the contrast of each aligned particle image with a projection of the refined volume and gives it a score. Effectively this gives a high score to particles with higher contrast that match the reference well, and a low score to those that have lower contrast and do not match the reference well. Minimising over per-particle scale means that the best particles get up-weighted, and the worst ones down-weighted. Often this is beneficial and can slightly improve the map quality and metrics. {% endhint %} \* Examine the output unsharpened and sharpened map in the volume viewer and look at the real space mask, cFSC plot and viewing direction plot. The reported resolution should be around 2.9 Å. \* Run an \[Orientation Diagnostics\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-orientation-diagnostics) job to assess the orientation sampling in more detail by connecting the volume, particles and mask from Refinement 1. Orientation diagnostics by default uses the solvent mask from refinement, however we disabled masking during refinement, so for the mask instead use the \`volume.mask\_fsc\_auto\` volume. {% hint style="info" %} To change the mask taken from a refinement job, \[you can alter the input for the mask volume slot\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/job-builder-tutorial#job-builder-inputs-section). To do this, open the inputs information in the mask field that you already loaded into the Orientation Diagnostics job builder (by clicking the toggle arrow) then go to the Outputs tab of the Non-Uniform Refinement (Refinement 1) job and drag over the slot called \`.volume.mask\_fsc\_auto.F\` into the mask field for the Orientation Diagnostics job. {% endhint %} !\[Figure 8. Non-Uniform refinement maps and diagnostic plots for Refinement 1, along with plots from Orientation Diagnostics.\](/files/4seWKrfmqo9MYoZhq6KX) In Figure 8 we can see that the volume looks good before sharpening, but after sharpening some parts have fragmented density, for example, the bottom of the transmembrane domain. The orientations for this particle are pretty well distributed for a GPCR with a cFAR score of 0.67 and SCF score is 0.826 but they are not uniformly sampled. ## 9. Improving the particle stack using Rebalance Orientations and Subset Particles by Statistic Although the refinement in Section 8 was already pretty good, we aim to investigate and classify the ligand-binding site later on in the pipeline, so we want the particle stack to be as high quality as possible to avoid classification artefacts. Two jobs that can help with this are \[Rebalance orientations\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/job-rebalance-orientations) and \[Subset Particles by Statistic \](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/job-subset-particles-by-statistic)(new in v4.7). First we will filter out some of the slightly excess views, by removing the least helpful particles in those poses. {% hint style="info" %} Rebalancing orientations can be most beneficial on datasets that have strongly over-represented views, but do not have views that are totally absent. In extreme cases, orientation bias can lead to stretched or compressed maps in the direction of low information, as well as resolution anisotropy. {% endhint %} 1. Run a \[Rebalance Orientations\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/job-rebalance-orientations) job setting the \`Intra-bin exclusion criterion\` to \`alignments3D/alpha\`. {% hint style="info" %} \`alignments3D/alpha\` is the internal CryoSPARC software name for the per-particle scale metric described earlier, and so using this setting will mean that particles from over-represented views that have the lowest per-particle scale will be rejected. {% endhint %} We found that \\~300k particles were removed. 2. OPTIONAL: Run a Non-Uniform Refinement (Refinement 2) with these particles with the same settings as used for Refinement 1, and assess the appearance of the per-particle scale histogram (optional) We found the plot showed a bimodal distribution that is overlapping, and this manifests as a shoulder on the main peak (see Figure 9). {% hint style="info" %} When a multimodal distribution is observed in the per-particle scale factors, it can mean that the peak at the low end contains poor quality, poor contrast or poorly matching particles compared to the refined volume. These particles often either do not benefit, or in some cases hinder the overall reconstruction. {% endhint %} We want to remove particles that are of low quality and will not be meaningfully contributing to the reconstruction. To do this: 3. Run a \[Subset Particles by Statistic\](/processing-data/all-job-types-in-cryosparc/import.md) job, selecting \`Subset by\`: Per particle scale. You could use the default subsetting mode that uses gaussian fitting, but we chose to use \`Subsetting mode:\` Split by manual thresholds, \`Number of thresholds:\` 1 and \`Threshold 1\`: 0.9. Finding the optimal threshold that removes the poorest particles but does not negatively affect the map resolution may take some trial and error. Setting a threshold of 0.9 we rejected a further \\~400k leaving us with \\~1.6M particles. 4. Run Non-Uniform Refinement (Refinement 3) using the particles in cluster 1, with the same settings as Refinement 1. 5. Optional: Run a Non-Uniform Refinement (Refinement 3B) using the particles in cluster 0 for comparison with Refinement 3. You should see that the map quality and GSFSC resolution are essentially the same between refinements 3 and 1, but the cFAR has marginally improved. We saw the refinement report an improvement in cFAR from 0.7 to 0.76 calculated without masking. !\[Figure 9. Diagnostic plots and volumes after rebalancing orientation and subsetting particles. A) Diagnostic plots for orientation distributions before and after rebalancing. B) Particle scale distribution before and after clustering. C) Example histogram of per-particle scale values along with the NU refined maps from each particle set.\](/files/MOKXgJ2Jd4uExuh3own6) {% hint style="info" %} The purpose of the steps in Section 9 are to remove poor and unnecessary particles so that the final stack is of higher quality overall. At this stage the effect on the result looks underwhelming, but the quality of the particle stack can have strong effects on downstream jobs like local refinement and 3D classification. We will show a comparison in Section 12 for what happens in local refinement if Sections 3, 9 and 11 of this case study are skipped. {% endhint %} ## 10. Global and Local CTF Refinement Now that we have nominally improved our particle stack, it is worth checking to see if there are microscope aberrations that can be corrected for. We can also go ahead and try Global CTF refinement to fit tilt, tetrafoil, spherical aberration and anisotropic magnification. Correcting for these, where possible, can improve the CTF fit, and therefore the map quality, and as a general rule, higher resolution maps benefit more from these corrections than medium resolution maps. \* Run two \[Global CTF Refinements\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-refinement/job-global-ctf-refinement), inputting the particles from the Local CTF Refinement particles, volume and mask\\\_fsc\\\_auto from Refinement 1. 1. Try using \`Fit Anisotropic Mag.\` : true 2. Try \`Fit Anisotropic Mag.\`, \`Fit Tetrafoil\` and\`Fit Spherical Aberration\` : true \* Run a \[Homogeneous Reconstruction Only\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-homogeneous-reconstruction-only) job for each using the mask\\\_fsc\\\_auto from Non-Uniform Refinement 3. {% hint style="info" %} When testing the helpfulness of Local or Global CTF Refinement in improving map resolution, using homogeneous reconstruction rather than refinement can take a fraction of the time. {% endhint %} !\[Figure 10. Plots from Global CTF Refinement and FSC curves after reconstruction. Tilt & Trefoil, Anisotropic Magnification (Aniso. Mag.) in the X and Y directions and Spherical Aberration from Global CTF Refinement, along with the GSFSC curves from downstream Homogeneous Reconstruction jobs.\](/files/smEXznN60khS8rMTEOxZ) In Figure 10 you can see that both tests improved the resolution from Figure 8, but that including spherical aberration and tetrafoil is even better. The plots for the tilt, trefoil and anisotropic magnification display clear regions of red and blue to fit to, but the spherical aberration is less clear, and by eye it is difficult to know if including the fit will be useful. See the \[Tutorial on CTF Refinement\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement) for a detailed description of how to interpret the plots, but essentially the brighter and more consistent the red and blue areas are in the data, the more reliable the fit its likely to be. These particles are on the smaller side, but as they were recorded on a Talos at 200 kV their signal-to-noise may be sufficient for local (per particle) defocus refinement to account for slight differences in particle height across the grid. \* Run a \[Local CTF Refinemen\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-refinement/job-local-ctf-refinement)t job inputting the particles, volume and volume.mask\\\_fsc\\\_auto mask from Refinement 1. This mask needs to be used because the refinement was run without a solvent mask. Figure 11 shows example outputs from a Local CTF Refinement job, we can see that the spread of defocus change across the particles does not extend beyond 500. The defocus profiles shown have a single obvious minima in the centre flanked by two less favourable local minima. These patterns, where there isn’t just a single dip, can be typical for a small particle. \* Run a \[Homogeneous Reconstruction\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-homogeneous-reconstruction-only) job using the mask\\\_fsc\\\_auto from the Non-Uniform Refinement. We can see in Figure 11 that the resolution was slightly worsened, so we should not include local CTF refinement at this stage. !\[Figure 11. Results from Local CTF Refinement. Example plots for change in defocus, and per particle defocus landscapes from Local CTF Refinement, and the GSFSC curves before and after Local CTF Refinement.\](/files/kNUjGCFKN8vRNFF1dOpZ) \* Using the particles from the Global CTF Refinement including Tetrafoil and Spherical Aberration, run a Non-Uniform Refinement job (Refinement 4) on the particles from the Global CTF Refinement including Spherical Aberration, including the following settings: | Parameter | Setting | | ----------------------------------- | ------- | | \`Minimize over per-particle scale\` | true | | \`Dynamic mask start resolution (A)\` | 1 | | \`Optimize per-goup CTF params\` | true | | \`Fit Tetrafoil\` | true | | \`Fit Spherical Aberration\` | true | You will notice that although we fitted Anisotropic Magnification in the Global CTF, we will not also do so during the Non-Uniform Refinement job. This is for two reasons; first, the anisotropic magnification is not likely to change greatly over the course of processing, whereas the other parameters might change more as the refinement achieves higher resolution, and second, when Aniso. Mag. is refined iteratively during either Homo Refine or NU refine, we have found rare cases where the values become unstable, so it is safer to only correct Anisotropic Magnification during Global CTF Refinement. The resulting map from Refinement 4 should have a higher resolution estimation than Refinement 2 and at this stage we had a map nominally at 2.64 Å with an auto tightened mask. ## 11. Reference Based Motion Correction We have now improved our map resolution with CTF correction, but we can further improve the particle image quality by correcting for dose-dependent sample damage as well as minor particle motions during imaging using \[Reference Based Motion Correction\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/motion-correction/job-reference-based-motion-correction-beta) (RBMC). RBMC requires that the total dose for all exposures is identical within a single job. Since we combined the two movie sets together during Patch Motion Correction, so we now need to split them again. \* Create an \[Exposure Group Utilities\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-refinement/job-exposure-group-utilities) job, inputting the accepted exposures from the Curate Exposures job in Section 2, and set \`Input Selection\`: exposure, \`Action\`: info\\\_only and \`Split Outputs by Exposure Group\` : true. This job will output two subsets of micrographs, one for each of the exposure groups (this is one exposure group per day of collection). We will now run \[Reference Based Motion Correction\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/motion-correction/job-reference-based-motion-correction-beta) in two stages so that we can assess the hyper-parameters and dose weighting before proceeding. 1. Estimate hyper-parameters for motion and dose weights \* Run a \[Reference Based Motion Correction\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/motion-correction/job-reference-based-motion-correction-beta) job (A) using the Non-Uniform refinement inputs from Refinement 4, and one of the micrographs exposure groups split above. \* Change the \[low level input slot\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/job-builder-tutorial#job-builder-inputs-section) for the input mask: drag over the \*\*volume.mask\\\_fsc\\\_auto.F\*\* because the refinement only applied a mask at the stage of FSC autotightening. \* set \`Final processing stage:\`Compute empirical dose weights \* Use more GPUs to accelerate the job if you have the resources \* Run a second RBMC job (B) with the same settings, but with the other micrographs exposure group. Example hyper parameters and empirical dose weights are shown in Figure 12. This dataset has 60 frames per movie. We would expect most of the high frequency information to be found in the first frames, before the sample has undergone radiation damage. In Figure 12 we see that the majority of the weighting (dark red colour) for high frequency information \*is\* in the early frames, but there is also an apparent reappearance of high frequency information in the last frames. {% hint style="info" %} It is important to always check and interpret plots such as the empirical dose-weights. In some cases with small particles and high noise, longer movies or many frames, RBMC’s optimization of priors can lead to a situation where the late frames appear to have strong high frequency signal. If such dose weights are applied during the particle correction stage, the results may be poorer than if only low frequency information is being used from the late frames. In tricky cases like this, we found that it can help to manually strengthen the spatial prior just for the dose-weighting calculation step, and we find that this can often improve the appearance of the empirical dose weights, as well as the reported map resolution after RBMC. {% endhint %} We will try to improve this situation, but if you chose to skip RBMC or continue with correction of the particles using the hyper parameters output from jobs (A) and (B) you may observe over-fitting artefacts later on in the processing pipeline. \* Clone the two RBMC jobs that you just ran (A and B) so that you have jobs (C) and (D), but manually enter in the values for \`Override: acceleration prior strength\` and \`Override: spatial correlation distance\` that were ascertained in jobs (A) and (B) respectively. For the \`Override: spatial prior strength\` set this to be 1/5th of the value that was determined previously, for the example shown above we would use 1.0984e-03 instead of 5.4929e-03. When you inspect the empirical dose weights from the new jobs you should see plots that are more in line with our expectations of radiation damage (see Figure 12C and D). In making the acceleration prior number smaller we are applying a stronger constraint on how much the particles move between frames. Using the updated value (divided by 5) seems to work well for the empirical dose weight estimation, however using this value for the particle correction stage may not yield the best results as very little motion would be corrected, so we will use the improved dose weights, along with the original hyper parameters to get the best of both. !\[Figure 12. Outputs from Reference-based motion correction. A + B) Example Empirical Dose Weights, and priors for the two movie subsets. High frequency information seems to reappear in late frames indicated by a dotted red box. C+ D) Empirical dose weights calculated by using a stronger spatial prior. E + F) Example particle trajectories from the two datasets using the hyper parameters in A + B, and the dose weights from C + D.\](/files/G6KRfqpZJXvTTXDh5PC9) \* Create a new Reference-Based Motion Correction job (E) with the same settings as (C), drag over the hyper parameters from (C) (this will contain both motion priors and dose-weighting) then override the values of the motion hyper parameters with those that you obtained from A) into the \`Override: spatial prior strength\`, \`Override: spatial correlation distance\` and \`Override: acceleration prior strength\`. \* Set \`Save results in 16-bit floating point\` on and motion-correct particles \`Final processing stage:\`motion-correct particles \* Repeat the process to create another RBMC job (F) using the hyper parameters from D, and overriding with the priors from (B). In Figure 12 we can also see the extent of the modelled particle motion for particles from each half of the dataset. \* Take both sets of particles from RBMC (E) and (F) and use them to run a Non-Uniform Refinement job (Refinement 5) with the same settings as in Section 10. The reported resolution and quality of the map should have improved. In Figure 13, we can see that the density for the TMD is poorer than that for the G-protein and scFv16. This is expected because in GPCRs it is typical that the G-protein and receptor domains move somewhat relative to one another, and the G-protein and ScFc16 being more rigid tend to dominate the particle alignment in consensus refinement. Recall that the aim of the processing here is to define the ligand binding pose and interactions, however, the ligand binding site is the poorest part of the map! This misfortune also reminds us of Murphy’s law, but all is not lost, we shall persevere by improving this region using a \[Local Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement) job. At lower contour thresholds where the receptor transmembrane (TMD) density is more visible, you might notice some additional blobs of density around the G-protein. This is not uncommon in single particle analyses of GPCRs. These blobs may represent sharpening artefacts that only become apparent near regions of good ordering, and only visible at a lower contour level than you would normally use to visualize these regions. In our case the only reason we took the contour so low was to see the relatively less well-ordered TMD. !\[Figure 13. Map quality after RBMC. Unsharpened and sharpened maps from Refinement 5 with an inset showing smeared and fragmented density in some transmembrane helices. The expected ligand-binding site is within the red dashed box.\](/files/G95n21SFH1lIyvG0mmA2) ## 12. Local Refinement of the TMD In order to improve the density in the TMD, especially where the ligand binding is expected to occur, we need a custom mask around the area of interest. Different options for mask creation are described \[here\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera), but we will make the mask using the molecular model, and this mask could be re-used if you have have more than one ligand-bound structure of the same protein. {% hint style="info" %} When processing ligand-bound datasets it is quite common that a molecular model of the apo form of the protein has already been built or that a predicted model is available. Sometimes it may also be necessary to do some refinement or model building in order to generate a mask in the precise region that you are interested in for local refinement, and in difficult cases this can be an iterative process. {% endhint %} \* Open the map from Refinement 5 in ChimeraX and also open PDB:7s8l and fit this into the density. \* Create a 20 Å simulated map for chain R (the receptor) and chain A (the ligand) by using the command \`molmap #X/R/A 20\` where X is the model number for the PDB. \* Resample the map on the grid of refinement 3 with the command \`volume resample #Y ongrid #Z\` Where Y is your molmap model number, and Z is the Refinement 5 model number, and note the threshold at which the volume covers all of the TMD density in your Refinement 3 map. \* Save the volume and use the \[Upload Local Files\](https://guide.cryosparc.com/application-guide-v4.0+/upload-local-files) feature to upload it to your CryoSPARC project directory. \* Use an \[Import 3D Volumes\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/import/job-import-3d-volumes) job to import this volume to CryoSPARC \* Use a \[Volume Tools\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools) job to create a custom mask. We need to make this somewhat tight to cover the ligand density but not include too much of the detergent micelle, so set \`Type of output volume\`: mask, set the threshold that you selected in ChimeraX, and add \`Dilation radius (pix)\`: 3 and \`Soft padding width (pix):\`16. The reason we are adding a dilation radius as well as the extending the soft padding is so that the mask covers the density for the ligand as well as the receptor. Inspect the mask to ensure that it is appropriate: an example is shown in Figure 14 as a guide. \* Create a \[Local Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/local-refinement) job and input the volume and particles from Refinement 5 and your newly made mask. We are now refining a small volume at around 40 kDa, and having aligned globally, we want to avoid the poses diverging too much, so we will restrict the rotations and translations at each iteration using the following settings: | Parameter | Setting | | ------------------------------------------------- | ------- | | \`Use pose/shift gaussian priors during alignment\` | true | | \`Re-center rotations each iteration\` | true | | \`Re-center shifts each iteration\` | true | | \`Rotation search extent (deg)\` | 10 | | \`Shift search extent (A)\` | 5 | | \`Initial lowpass resolution (A)\` | 4.5 | {% hint style="info" %} Applying gaussian priors means that movements of particles further away from the input position is softly penalised, encouraging a solution close to the input. For small regions (<50 kDa) particle alignment can be tricky, and over-fitting occurs frequently, so restricting the search extents prevents unrealistic divergence from the input refinement. In some cases reducing the searches may also require additional extra passes, but the map should always be inspected for overfitting artefacts such as radial spikes of unexpected density. {% endhint %} We want to lowpass filter the map, but the default value of 15 in the Local Refinement is probably overly harsh, and we can usually select a value where the GSFSC is still \\~1. In our case that was at \\~ 4 Å but we set \`initial lowpass resolution\`: 4.5 to be a bit more conservative. !\[Figure 14. Local Refinement of the receptor. A) Example mask produced from chain R and A of 7s8l shown as a threshold of 0.95. B) Density for the unsharpened and sharpened map after Local Refinement. The expected ligand-binding site is within the red dashed box. C) Comparison of density in Refinement 5 (purple) and the local refined map (green) coloured by proximity to the fitted model 7s8l chain R. Density near the modelled ligand is in white, and unmodeled density is shown in grey.\](/files/Rkb1bZeTmbYgzUeL3lTp) Although the reported resolution of the Local Refinement is poorer than for the global Refinement 5 (in our case 2.69Å vs 2.41 Å), the density for the TMD is greatly improved as evident by examining the map quality for the transmembrane helices shown in the insets of Figures 14 compared to those in Figure 13 (Refinement 5). ### 12A) Comparison of local refinement map quality with a simpler processing pipeline For comparative purposes, we used the same local refinement settings and mask used above on particles that were processed through a simpler route, skipping Sections 3 and 9 and 11; that is, we did not perform micrograph denoising, junk detection, orientation rebalancing, manual curation on the basis of per-particle scale or reference based motion correction. This pipeline ended up with 3M particles compared to our \\~1.6M in this case study. !\[Figure 15. Comparison of map quality between our case study procedure including sections 3, 9 and 11 (left) and a simpler processing pipeline that omitted these steps (right). Radial over-fitted artefacts are circled in red.\](/files/OZgm6koSd9eqAkfJJJxm) From Figure 15 we can see that if we do not use the more advanced processing steps found in Sections 3, 9 and 11 then the result of local refinement is poorer in quality. The inclusion of poorer quality particles in local refinement is associated with radial plates and streaks of density clearly indicating over-fitting during refinement. {% hint style="info" %} If with this, or your own datasets, even after strict curation of particles you still observe radial density artefacts, this may be exacerbated by refining a small masked volume, or by the presence of a detergent micelle inside the local refinement mask. You can try a different mask, or try further limiting the searches, setting tighter gaussian priors, and limiting the maximum resolution used for alignment. These steps may reduce the reported resolution of the map, but interpretation of an artefact-free map at lower resolution is easier and preferable to one at higher resolution that contains obvious artefacts. {% endhint %} In our good Local Refinement (see Figure 15-4, left), the unsharpened map has sufficient clarity to identify ligand density in the entrance to the TMD on the cytosolic side, however, the ligand density is still relatively poor and the density becomes fragmented after sharpening. A poorer quality of ligand density indicated either partial occupancy, or a mixture of binding poses. To further improve the ligand density we will go on to locally classify this region to see if we can separate empty/full binding sites or alternative binding poses. ## 13. 3D Classification {% hint style="info" %} When working with ligand-bound structures it can often be helpful to have an atomic model of the apo structure pre-built to aid in the identification of ligand density. This model can be used to generate difference maps which can help identify the ligand density, especially if the ligand ends up binding in multiple locations, or in a different location that expected. An Alphafold model or structure of a closely related protein or complex refined into the map might be good enough at this stage, but a model of the same protein is even better. Classification of ligand densities can be rather challenging; unless the ligand contains a heavy metal, if binding is not rigid, or is substoichiometric, then the signal from the ligand will often be relatively small compared to the protein complex, and it is important to avoid the introduction of artefacts by using too small of a mask. Most commonly this will manifest as density in the masked area that is either completely absent, or stronger than the rest of the map. {% endhint %} In order to classify the region that contains the ligand, we need to create a mask that covers just this region. To do this, we can start with chain A (the ligand chain) from 7s8l. \* In ChimeraX load up your Local Refinement map and updated atomic model and fit the model into the map. Follow the same process that we used in Section 11, except use just chain A for the Molmap: \* Save the volume and transfer it to your CryoSPARC directory. \* Use an \[Import 3D Volumes\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/import/job-import-3d-volumes) job to import this volume to CryoSPARC. \* Use a \[Volume Tools\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools) job to create the custom mask, setting an appropriate contour threshold, we used 0.0063, and the \`Type of output volume\` to mask. Our mask relative to the locally refined TMD is shown in Figure 16. \* Create a \[3D Classification\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-3d-classification-beta) job (see \[here\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification-beta) for a detailed tutorial) which will be Classification 1, inputting the volume and particles from the Local Refinement and using the following settings: | | | | ----------------------------- | ----- | | \`O-EM batch size (per class)\` | 20000 | | \`O-EM learning rate init\` | 1 | | \`Class similarity\` | 0 | | \`Filter resolution\` | 3.5 | Setting the class similarity lower than the default of 0.5 means that the density within the mask for each class will be more independent, and increasing the learning rate from the default of 0.4 to 1 allows the volume in each class to be totally replaced at each iteration, allowing for more evolution of the class volumes. The filter resolution is important as this needs to be able to represent the sort of class-to-class differences that you are looking for, but also not to include too higher frequencies that may result in classes with only very minor high frequency differences. It is best to choose the lowest resolution that can still represent the thing that you are interested in, in this case 3.5 Å is a good level to observe ligand density. !\[Figure 16. Local Classification at the ligand site. A)The local refined map with PDB 7s8l fitted in (ligand in white), and the mask used for local class shown at a threshold of \\~0 and 1. B) Classification statistics showing the per-particle class size (ESS) and class distribution at the end of classification.\](/files/CBSAMDpaYnN1Sgd08sNC) In the output you will see some plots indicating statistics from the classification. The example per-particle class size (ESS) plot shown in Figure 16 indicates that most of the images have smeared probability across multiple classes with a mean ESS score of 2.8, which means that on average, each particle might belong to any of 2.8 classes. The class distribution shows all of the classes are equally populated and these two observations together suggest that there either may be continuous heterogeneity (for example flexibility in the ligand binding) or that the signal inside the mask is not sufficient to have high confidence in the classification at the filter resolution. ## 14. Assessment of 3D classes for manual regrouping To inspect the class volumes from 3D classification at higher pixel sampling and resolution, next run a \[Heterogeneous Reconstruction Only\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-heterogeneous-reconstruction-only) job to obtain higher resolution volumes for each of the 10 classes. To visualise the ligand density, we produced difference maps for the 10 classes by subtracting a fitted molmap of the apo receptor PDB structure using ChimeraX. \* Download the volume series (not the sharpened one) \* Open the 10 unsharpened volumes (not as a series) in a new ChimeraX session \* Open 7s8l, fit it into one of the maps and execute the following commands \`\`\`jsx set bgColor white; graphics silhouettes true; fit #11 inmap #1; lighting soft molmap #11 /R/B/C/D/E 3 volume subtract #1 #12; volume subtract #2 #12; volume subtract #3 #12; volume subtract #4 #12; volume subtract #5 #12; volume subtract #6 #12; volume subtract #7 #12; volume subtract #8 #12; volume subtract #9 #12; volume subtract #10 #12 surface dust #13 #14 #15 #16 #17 #18 #19 #20 #21 #22 size 5 volume level 0.07176 \`\`\` These steps should generate maps in which only the unmodeled density is now visible, and small specks and noise have been removed. You can inspect each one in turn and decide which classes look the most promising. You might find using the side view function allows a clearer view by removing detergent density from view. !\[Figure 17. Ligand volumes after 3D classification. A) Density for the difference between the heterogeneous reconstructed volumes and a molmap of the apo PDB model 7s8l. B) simplified representations of the 4 major shapes of ligand density. Coloured boxes indicate the corresponding major ligand density shape that each class was manually assigned to.\](/files/pJ30g5MLMHphNWvuot9P) After inspecting the unmodeled density in the 10 class Heterogeneous reconstruction, we identified 4 broad categories of ligand shape that we refer to as poorly ordered (grey), party-ordered (gold), linear ordered (purple) and circular ordered (green). The manually chosen category for each class has been indicated with the corresponding colour of box behind each volume in Figure 15A. Due to differences in particle stacks, masking or 3D classification initialisation, the volumes you observe may be different to those shown above, but we reproducibly found one or more classes with a circular ordered appearance. \* Run 4 separate Homogeneous Reconstruction Only jobs; one for the particles that match each major ligand shape by grouping together the classes with similar-looking density. ## 15. Mask optimisation for resolution estimation CryoSPARC refinements have in-built masking that automatically tightens the mask at the end of refinement, and this mask is used to calculate the global resolution displayed in the GUI. Map sharpening by applying a global \*B\*-factor is also handled automatically in refinement and reconstruction allowing for the output of automatically sharpened maps. It is always a good idea to inspect the FSC curves and mask, as well as the sharpened map connectivity in case the mask or sharpening \*B\*-factor need to be adjusted. We are going to check those out now in Sections 15 and 16. \* Open up the locally refined map in ChimeraX, and the \*\*mask\\\_fsc\\\_auto\*\*. This mask can be found in the outputs tab of the job in the Refined Volume section (Importantly, this is different from the mask provided in the Outputs Group section, which is the final iteration mask before auto-tightening). Set the map at a contour threshold where you observe the features you are interested in, but not including noise, and set the mask threshold to 1. This mask threshold shows the contour at which the mask was binarised, and on top of this there is a soft edge that extends out to the contour observed at a threshold of \\~0. \* Examine the mask and map together and decide if the mask encompasses the volume of your particle adequately. We found that the auto-tightened mask included some density from the adjacent G-protein (Figure 18, left). \* Examine the FSC for the auto-tightened map and compare the Tight and Corrected curves. As described in the \[mask creation\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera) tutorial the difference between these curves can be used as an indicator of mask tightness. In our example in Figure 18 we do see a sudden drop in the corrected curve, suggesting there is density outside of the mask. !\[Figure 18. Masking for FSC calculation. The auto-tightened FSC mask and manually designed mask shows at a threshold of 1 with their corresponding FSC curves. Red circles show regions of density that are outside of the receptor domain. Yellow shaded boxes indicate where a dip occurs in the corrected curve.\](/files/JWkwHLyVRJ3i3igSCb8a) {% hint style="info" %} A dip in the corrected FSC curve can indicate a mask that is too tight, however this phenomena is common and expected with local refinement, because inevitably there will be some density outside of the masked region of interest that the FSC mask may exclude or cut through {% endhint %} Although this mask is generous enough it also includes density outside of the region of interest, so we will design a new mask that only covers only the receptor and ligand. \* Open up the map from Local Refinement in ChimeraX and also open PDB:7s8l or an updated model and fit this into the density. \* Create a 15 Å simulated map for chain R (the receptor) and chain A (the ligand) by using the command \`molmap #X/R/A\` 15 where X is the model number for the PDB. \* Resample the map on the grid of the local refinement map with the command \`volume resample #Y ongrid #Z\` Where Y is your molmap model number, and Z is the local refinement model number, and note the threshold at which the volume covers all of the TMD density in your local refinement map. \* save the file and transfer and import it to CryoSPARC. We want this mask to be more generous than the one used for Local Refinement, so we will add a dilation radius. \* Create a Volume Tools job and input the newly imported receptor volume. Set the \`Threshold\` to the predetermined value (we used 0.13) and set the \\\`Dilation radius (pix) to a few pixels (we used 6). \* Create \[Validation (FSC)\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-validation-fsc) jobs for each of the homo reconstructed volumes, using the newly made mask and inspect the FSC curves. We found that the reconstructed receptor maps had reported FSC=0.143 resolutions of 2.75-2.83 Å . ## 16. Map sharpening and inspection for model building At the end of the day, the resolution number is less important the map quality, especially in the region of interest and in this case we are most interested in the ligand density and its binding interactions. Taking into account the estimated resolutions, we can now inspect the sharpened maps and look for map feature consistency. At \\~2.8 Å resolution we would expect to be able to see good definition of side chains, some density for the backbone carbonyls, and if it was a soluble region, possibly density for well-resolved water molecules. !\[Figure 19. Optimisation of the sharpening B-factor for model-building. Volumes for a transmembrane helix and the ligand from the reconstructed circular ordered class without sharpening, and with a sharpening B-factor of -101 and -60. B) Difference map density for the homogeneous reconstructions and C) for the deposited maps for emdb 24896.\](/files/RVs3l5WT37M1JUmM70pT) Compare the sharpened and unsharpened density from your Homogeneous Reconstruction jobs. You will find that the auto sharpened map has good side chain density, and better definition of backbone carbonyls than the unsharpened map (features that we would expect to see in this resolution range), however it will likely also has more fragmented density for the ligand. Example densities are shown in Figure 19A. We need to strike a balance here because we are mostly interested in the ligand and its binding interactions, and so we would like to manually adjust the sharpening B-factor ourselves to optimise for this region. \* Create a\[ Sharpening Tools \](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-sharpening-tools)job for each of the homogeneous reconstructed volumes, and add \`B-factor to apply\` that is lower than the value that was previously used. In our case we found that -50 or -60 gave a good balance for the density quality of the receptor and ligand (see Figure 19A). Having appropriately sharpened our maps, we were able to go on to model the ligand poses for three of the maps and refine each using Phenix (ligand models shown in Figure 19B) to aid comparison to the deposited maps and model. ## 17. Interpretations and conclusion Our final maps show density consistent with different binding poses of the ligand Cortistatin-14. By locally refining and classifying, the ligand density is now much clearer in at least two classes than in the deposited map (see Figure 19). Q-scores for the receptor and ligand suggest a fit that is consistent with the reported resolutions. For the Cortistatin-14 peptide to be circularised, a disulphide bond needs to be formed between the two cysteine residues. Using our workflow here, the presence of a circular class and a linear class indicate that the Cortistatin-14 in this sample is partially reduced (i.e. some of the molecules have their cysteines disulphide bonded and some do not). This means that the disulphide bond that usually keeps it circular \*in-vivo\* has been broken by reduction, and we noticed that the reducing agent tris(2-carboxyethyl)phosphine (TCEP) was used upstream during sample preparation. Partial reduction could explain why we have both an ordered-circular (S-S bonded) and ordered-linear (S-S broken) class. {% hint style="info" %} It is always useful to keep in mind the biology and chemistry of the sample that was applied to the grid. You may observe something unexpected in the density and knowing more about your sample can help determine if the feature is likely to be real, or an artefact. {% endhint %} We were unable to build a satisfactory model into the density for the poorly ordered class, and there may be residual heterogeneity in terms of ligand binding poses within the particles used for this map. Examining our three ligand models, we can see that the register in our model is different to that of the original model 7s8l (see Movie 1). We were able to confidently assign the register due to the bulky side chain residues, and in particular the tryptophan. {% embed url="" %} \*\*Movie 1. The final sharpened map and model for the circular ordered class, and comparison to the deposited map and model. The Cortistatin-14 is coloured in orchid, and the receptor in teal.\*\* Density is compared to that of emdb:24869 and the updated model compared to pdb:7s8l. The tryptophan residue is highlighted in gold to show the shift in register between the the models. {% endembed %} Cortistatin-14 binds to the cytoplasmic side of the MRGPRX2 receptor and displays a variety of ligand binding poses. The N-terminus of Cortistain-14 is in the same location in all three cases, and this is likely the important region for binding affinity. The C-terminal portion of the ligand, however, is exposed to the cytoplasmic space and therefore has the capability of forming different arrangements. We wanted to note that if we used the overfitted local refinement in Section 12B for classification using the same settings and mask as used in Section 13, we were unable to find a class with circular ligand density, even though the ligand appeared circular in the parent map. This emphasizes the importance of data quality when performing both local refinement and local classification, and in this case study, using some of the new features recently added in CryoSPARC allowed improved interpretation of the data. You can download our versions of the final maps, half maps and masks from the links below for comparison with your own processing! | | | | | | | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | consensus | \[sharpened map\](https://structura-assets.s3.us-east-1.amazonaws.com/empiar-10853-case-study/EMPIAR-10853\_consenseus\_sharp.mrc) | \[half map A\](https://structura-assets.s3.us-east-1.amazonaws.com/empiar-10853-case-study/EMPIAR-10853\_consensus\_half\_A.mrc) | \[half map B\](https://structura-assets.s3.us-east-1.amazonaws.com/empiar-10853-case-study/EMPIAR-10853\_consensus\_half\_B.mrc) | \[mask\](https://structura-assets.s3.us-east-1.amazonaws.com/empiar-10853-case-study/EMPIAR-10853\_consensus\_mask\_fsc.mrc) | | TMD local refinement | \[sharpened map\](https://structura-assets.s3.us-east-1.amazonaws.com/empiar-10853-case-study/EMPIAR-10853\_TMD\_local\_map\_sharp.mrc) | \[half map A\](https://structura-assets.s3.us-east-1.amazonaws.com/empiar-10853-case-study/EMPIAR-10853\_TMD\_local\_half\_A.mrc) | \[half map B\](https://structura-assets.s3.us-east-1.amazonaws.com/empiar-10853-case-study/EMPIAR-10853\_TMD\_local\_half\_B.mrc) | \[refinement mask\](https://structura-assets.s3.us-east-1.amazonaws.com/empiar-10853-case-study/EMPIAR-10853\_TMD\_local\_mask\_refine.mrc) | | TMD reconstruct circular ligand | \[sharpened map\](https://structura-assets.s3.us-east-1.amazonaws.com/empiar-10853-case-study/EMPIAR-10853\_TMD\_reconstruct\_circular\_ligand\_map\_sharp.mrc) | \[half map A\](https://structura-assets.s3.us-east-1.amazonaws.com/empiar-10853-case-study/EMPIAR-10853\_TMD\_reconstruct\_circular\_ligand\_half\_A.mrc) | \[half map B\](https://structura-assets.s3.us-east-1.amazonaws.com/empiar-10853-case-study/EMPIAR-10853\_TMD\_reconstruct\_circular\_ligand\_half\_B.mrc) | \[FSC mask\](https://structura-assets.s3.us-east-1.amazonaws.com/empiar-10853-case-study/EMPIAR-10853\_TMD\_reconstruct\_circular\_ligand\_mask.mrc) | | TMD reconstruct linear ligand | \[sharpened map\](https://structura-assets.s3.us-east-1.amazonaws.com/empiar-10853-case-study/EMPIAR-10853\_TMD\_reconstruct\_linear\_ligand\_map\_sharp.mrc) | \[half map A\](https://structura-assets.s3.us-east-1.amazonaws.com/empiar-10853-case-study/EMPIAR-10853\_TMD\_reconstruct\_linear\_ligand\_half\_A.mrc) | \[half map B\](https://structura-assets.s3.us-east-1.amazonaws.com/empiar-10853-case-study/EMPIAR-10853\_TMD\_reconstruct\_linear\_ligand\_half\_B.mrc) | \[FSC mask\](https://structura-assets.s3.us-east-1.amazonaws.com/empiar-10853-case-study/EMPIAR-10853\_TMD\_reconstruct\_linear\_ligand\_mask.mrc) | --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-processing-of-a-ligand-bound-gpcr-empiar-10853.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-discrete-heterogeneity-in-a-sample-of-acetogenin-bound-complex-i-empiar-10927.md). # Case Study: Discrete heterogeneity in a sample of Acetogenin-bound complex I (EMPIAR 10927) Processing of particles from EMPIAR 10927 including separation of discrete targets and states on the basis of per-particle scale, Ab-Initio Reconstruction with custom settings, 3D Classification, and clustering by 3D Variability Analysis. ### \*\*Introduction\*\* In this case study we will work through exploratory steps in the processing pipeline for a sample of mouse heart mitochondrial respiratory complex I prepared in the presence of a tight-binding inhibitor called Acetogenin. The results of the original processing were deposited as \[EMDB:13611\](https://emdb-empiar.org/EMD-13611) and \[PDB:7psa\](https://pdbe.org/7psa) and described in \[Grba et al. (2022)\](https://www.jbc.org/article/S0021-9258\\(22\\)00042-4/fulltext). We selected this dataset to illustrate different routes to disentangling discrete heterogeneity from a native sample because of its high contrast, and rich heterogeneity. {% hint style="info" %} There isn’t one “right” way to disentangle the different species, so rather than prescribe the “best” way to do it, we will work through, and compare four different strategies to separate discrete heterogeneity, using a variety of CryoSPARC jobs, and go on to evaluate their performance. {% endhint %} The raw data are publicly available for download as \[EMPIAR-10927\](https://www.ebi.ac.uk/empiar/EMPIAR-10927/). Image processing was performed using CryoSPARC v5.0. !\[Figure 0. Case Study overview\](/files/wYd78nrZW3KwlrtUhBXH) Mitochondrial respiratory complex I is a 1 MDa protein complex that resides in the inner mitochondrial membrane of eukaryotes and works alongside respiratory complexes II, III, IV and complex V, that are together crucial for cellular energy transduction. Complex I has a hydrophilic arm protruding in the mitochondrial matrix, and a membrane arm embedded in the inner mitochondrial membrane. It performs a redox (reduction-oxidation) reaction, taking electrons from NADH, passing them to ubiquinone in the membrane, and pumping protons across the membrane. The gradient of protons is ultimately used to power ATP production by complex V (F₁Fₒ ATP synthase), and the transport of important molecules across the membrane. In the mitochondria, complexes I,III and IV are known to associate together as supercomplexes, but for the purpose of studying complex I in isolation, this sample was solubilised using a detergent (Dodecyl Maltoside) that breaks up the supercomplexes. Mammalian complex I structures have been observed in two distinct states known as the “active” and “deactive” states (alternatively called “open” and “closed”) that we will keep in mind during the processing of this dataset (see Introductory figure). !\[Introductory figure. Simulated maps from PDB 7ak5 and 8om1 are shown along with gaussian smoothed density for the detergent belt. Curved arrows indicate the relative opening and closing of the two arms between the states.\](/files/nZ9OdTJl9q4PrbmOQM1N) ### \*\*Setting up\*\* This is a fairly small dataset with 1283 movies recorded in mrc format with gain correction, and it requires 3.3 TB of disk space. Before beginning this tutorial, you should \[create a new project and a workspace\](https://guide.cryosparc.com/application-guide-v4.0+/using-the-cryosparc-interface/projects-workspaces-and-live-sessions#creating-your-first-project) within that project. Download the 1283 movies to a location of your choosing. For example, our data is downloaded to a directory called \`rawdata\` using the commands: \`\`\` cd /path/to/rawdata wget -m . \`\`\` For particle picking, we will be using TOPAZ (\[Bepler et al., 2019\](https://doi.org/10.1038/s41592-019-0575-8)). TOPAZ will need to be installed separately to use via the CryoSPARC wrapper, and instructions can be found \[here\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/deep-picking/topaz). ### \*\*1. Movie import and pre-processing\*\* \* Import the data using an \[Import Movies\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/import/job-import-movies) job. The \`Movies data path\` should match the location of the directory containing the downloaded \`.mrc\` files, for example:\`ftp.ebi.ac.uk/empiar/world\_availability/10927/data/FoilHole\_\*.mrc\` These movies are already gain corrected so you do not need to add any gain reference. \* Add in the experimental information below that we obtained from the \[EMDB:13611\](https://emdb-empiar.org/EMD-13611) entry. | Parameter | Setting | | ----------------------------- | ------- | | \`Raw pixel size (A)\` | 1.043 | | \`Accelerative Voltage (kV)\` | 300 | | \`Spherical Aberration (mm)\` | 2.7 | | \`Total exposure dose (e/A^2)\` | 50 | \* Run a \[Patch Motion Correction\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/motion-correction/job-patch-motion-correction) job with \`Save results in 16-bit floating point\`:true, so that the output images take up half of the disk space compared to the default 32-bit floating point files (learn more about float16 format \[here\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-float16-support)). The \`Number of GPUs to parallelize\` can be set depending on GPU availability. Parallelizing this job across six GPUs we found each Patch Motion job took 1 hour 44 mins. \* Using the Quick action menu, run a \[Patch CTF Estimation\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-estimation/job-patch-ctf-estimation) job with the default settings. ### \*\*2. Excluding poor micrographs from downstream processing\*\* Movies collected for single-particle cryo-EM can have a variety of different characteristics. Some of these, e.g., a range of defocus values or a range of ice thickness to obtain a range of particle orientations, can be beneficial for image reconstruction. However, movies often also contain unwanted junk and outlier attributes that compromise the quality of their particles, such as excessive in-movie motion and ice that is too thick or too thin for your sample. We often observe junk in the form of non-vitreous ice and contamination with ice crystals, but other features such as the edge of the holey support are also frequently observed. Ideally we want to avoid extracting particles from regions of junk, or from poor quality micrographs, as these images can sometimes be challenging to remove later on in the processing pipeline. The strategy we will use here is to automatically detect regions of junk so that we can remove micrographs with large quantities of junk and poor statistics. \* Using the Quick actions menu, run a \[Micrograph Junk Detector\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/exposure-curation/job-micrograph-junk-detector-beta) job (1). We show example outputs from the mic. Junk Detector job in Figure 1. We can see that most of the micrographs contain the edge of the gold holes (green), and many contain extrinsic ice defects (magenta) such as ethane contaminants or small ice crystals, but these make up only a very small percentage of the imaged area. On the other hand, relatively few images contain intrinsic ice defects (often non-vitreous ice), but this type of junk makes up a higher percentage of the total imaged area in the dataset. !\[Figure 1. Results from the Micrograph Junk Detector job. An example micrograph with identified junk regions, number of micrographs containing each type of junk, and total micrograph area taken up by each junk type are shown.\](/files/AOgUdgHhXYpxaqfmSCI8) We will now inspect the CTF-estimated and junk-detected micrograph statistics so that we can exclude images of poor quality from downstream processing. \* Run a \[Manually Curate Exposures\](https://guide.cryosparc.com/application-guide-v4.0+/interactive-jobs#interactive-job-manually-curate-exposures) job via the Quick actions menu, or via the job builder inputting the “Labelled Micrographs” from Mic. Junk Detector. \* Navigate to the pink Interactive tab \* Set thresholds for outliers on undesirable characteristics \* We chose to select the following thresholds: | Parameter | Threshold | Reason | | ------------------------ | --------- | ------------------------------------------------ | | \`CTF fit resolution\` | max 6 Å | Exclude poorest resolution micrographs | | \`Relative Ice Thickness\` | max 1.119 | Exclude very thick ice with poor signal-to-noise | | \`Junk Area (%)\` | 35 | Exclude micrographs with large amounts of junk | We found that if we set the Junk Area maximum threshold to 35%, this also excluded the majority of micrographs that contained relatively thick ice and intrinsic ice defects. After curation we were left with 1134 micrographs. ### 3. Particle picking and extraction During the original processing pipeline in \[Grba et al. (2022)\](https://www.jbc.org/article/S0021-9258\\(22\\)00042-4/fulltext)\*.,\* particles were picked on the basis of manual picking, followed by template picking, and so picks were focussed on those that resembled complex I from the start. Here, we plan to investigate if there are any other targets present in this sample, and to do so we are going to use a TOPAZ pretrained model to pick particles. {% hint style="info" %} For TOPAZ picking, you don’t strictly need to already know the diameter(s) of your particles - although usually TOPAZ downsamples micrographs according to a supplied diameter, you can instead manually downsample the micrographs as we do here. We downsample to a pixel size where we still expect particle features to be visible. You can additionally calculate a minimum distance between picks. For example, if you want to allow your particles to be at least 100 Å away from each other in the downsampled micrographs then you can use this equation: $$\\dfrac{\\text{interparticle distance (\\AA)}}{\\text{downsample factor} \\times \\text{pixel size (\\AA/pixel)}} = \\text{radius of extracted regions (pixels)}$$ When we add in our pixel size and downsample factor, this gives us a value of 12 that we enter into the job settings. The TOPAZ job does not output example images of downsampled micrographs, but in Figure 2 we show what a micrograph looks like when it is downsampled by a factor of 2,4,8,16 and 32 compared to the original micrograph. We also show an inset of an individual particle, and we can see that at Downsampling factors of 16 and 32, there is significant loss of information about the particle shape, however there isn’t much change in the particle appearance with downsampling of 2 or 4. A higher downsampling factor can facilitate a shorter job runtime, so we made a compromise by selecting a downsampling factor of 8 (a pixel size of 8.34 Å) meaning that in our hands the run only takes \\~13 mins, but the main particle features are still present. Note that for smaller particles or larger pixel sizes you may wish to use less downsampling. {% endhint %} ![](https://guide.cryosparc.com/files/EOTJvheZWDMRVqe5OHer) **Figure 2. Micrograph downsampling with factors of 2 to 32.** An individual particle image inset is shown for each example, and pink bar showing the distance of 100 Å is also shown. \* Create a new TOPAZ Extract job and input the accepted exposures from Curate Exposures. Use the following settings: | Parameter | Setting | Reason | | ----------------------------- | ----------------- | ---------------------------------------------- | | \`Path to TOPAZ executable\` | your\\\_TOPAZ\\\_path | | | \`Select pretrained\` | ResNet16 | | | \`Downsampling mode\` | manual | | | \`Downsampling factor\` | 8 | Downsampling the micrographs speeds up the job | | \`Radius of extracted regions\` | 12 | Definition of inter-particle distance (pixels) | \* Run an Inspect Picks job inputting the micrographs to examine how well TOPAZ picked particles. See if you note any correlations between particle number and micrograph statistics in the scatter plot on the left. We found that the particles in this dataset were picked pretty well by TOPAZ with picks centred on visible particles (examples shown in Figure 3), and very few on empty ice regions; on this basis we accepted all of the picks by clicking “Done picking”. !\[Figure 3. Example TOPAZ picks. Micrographs are representative of thick, and thin ice, showing the pick locations found by TOPAZ in green.\](/files/StfpcbeM4K2OogYmGPzx) {% hint style="info" %} As defocus is increased, higher frequency components from particles become delocalised further out in real space due to the point spread function of the microscope. If too small a box is selected for extraction, some information about your particle is lost, and this may limit the obtainable resolution. Conversely using an excessively large a box can lead to a lot of noise in the images, and this can also have a negative effect on the resolution of your reconstruction. As a very rough rule, a box of \\~1.5-2.5 x the diameter of your particle is often appropriate, however very high resolution data or data collected with high defocus may require a larger box. The box size must be an even integer of pixels, and it is best if you choose or downsample to a \[box size that is computationally efficient.\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/extraction/job-extract-from-micrographs#box-sizes-that-allow-for-efficient-processing) {% endhint %} We know from existing PDB models that the diameter of complex I is around 300 Å, so we will use a box of 450 downsampled to a box of 320. The choice of initial box size here is \\~1.6 x the particle diameter in Å, and then downsampled to a box size that is smaller and more efficient to handle in the software, but not expected to limit the resolution by its Nyquist frequency (in this case Nyquist will be 2.93 Å). \* Extract the particles using the \[Extract from Micrographs\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/extraction/job-extract-from-micrographs) job (1) with a box of 450 pixels (478 Å) and Fourier crop them down to 320 pixels. Fourier cropping makes the images smaller, so that they use up less disk space and some jobs will run faster. This cropping in Fourier space downsamples images in the same way as the \[Downsample Particles\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/extraction/job-downsample-particles) job, so that while the box extent in Å is kept the same, the pixel size is larger and the Nyquist limit (the highest achievable resolution) is lower. Select \`Save results in 16-bit floating point\` to save on the disk space required. We extracted 93,230 particles. ### 4. Initial Refinement and diagnostic 3D Variability Analysis {% hint style="info" %} When processing a repeat target, or re-processing a dataset, it can be faster to use existing 3D volumes to initialise refinements, rather than generating Ab-Initio volumes for each dataset. It can also be tempting to assume that a purified protein contains a homogeneous population. Here, we will see what happens when we do exactly that! As there is already a deposited map for this dataset we will use that, and refine all of the extracted particles. {% endhint %} \* Run an \[Import 3D Volumes\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/import/job-import-3d-volumes) job and use EMDB ID 13611 \* Run a \[Non-Uniform Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-non-uniform-refinement-new) job (NU Refinement (1)) using the extracted particles, and the volume from Import 3D Volumes, and set \`use dynamic refinement mask\` false. For membrane proteins during Non-Uniform Refinement, masking is often not necessary because the regularisation dampens the micelle signal sufficiently. Examine the map and statistics from the refinement. !\[Figure 4. The results from Non-Uniform Refinement 1. Plots shown are global FSC, Conical FSCs, orientation distribution plot, an image of the unsharpened map, and the per-particle scale factors.\](/files/qOFjXMnI6MSwkfhNflur) {% hint style="info" %} Per-particle scale is a function that compares the contrast of each aligned particle image with a projection of the refined volume and gives it a score. Effectively this gives a high score to particles with higher contrast that match the reference well, and a low score to those that have lower contrast and do not match the reference well. Minimising over per-particle scale means that the best particles get up-weighted, and the worst ones down-weighted. Often this is beneficial and can slightly improve the map quality and metrics. In CryoSPARC v5, particles with negative scale (i.e. the inverted image matches the reference better) are automatically rejected. {% endhint %} In Figure 4 we show the results of our NU Refinement (1); the resolution at 3.4 Å is similar to that found in the published processing and the cFAR value of 0.79 and orientation sampling plot indicate that there are good range of different views of complex I present in this dataset. The map looks isotropic and does not show smearing or other visible artefacts. The per-particle scale plot is intriguing though! We observed that \\~3k particles were rejected due to having negative scale values, and the accepted particles formed a bimodal distribution where the high scale peak (expected to be better quality matches to the reference) contains fewer particles than the low scale peak. {% hint style="info" %} Multimodal particle scales often indicate some sort of heterogeneity, and one way to investigate this is to use 3D Variability Analysis. 3DVA can often help to visualise density changes that originate from compositional or conformational heterogeneity, or can show smearing artefacts that indicate the presence of particles with poor alignment, or that do not match to the refined volume. {% endhint %} \* Run 3D Variability Analysis (3DVA (1)) using the particles from NU Refinement (1), and set the Filter resolution to 8 Å. \* From your 3DVA (1) job, use the Quick actions menu to build a \[3D Variability Display\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-3d-variability-display) job (1). Set \`Downsample to box\`: 128 and \`Crop to size after downsample:\` 100. \* Download and examine the three volume series in ChimeraX. ![](https://guide.cryosparc.com/files/MQ9fkJ3MmVZhje0qqTFV) **Movie 1. Volume series from 3DV Display (1).** Pink; component 0; Blue; component 1, Green; component 2. We show examples of the density changes seen in the three components of 3DVA (1) in Movie 1. In 3DVA, the particle poses are fixed according to alignment to the Refinement volume, and the output components are ordered according to the magnitude of density changes. In component 0 (the largest density change), we see density disappearing and reappearing for the hydrophilic domain of complex I, and in components 1 and 2, we see opening and closing, and twisting motions of the hydrophilic and hydrophobic domains. The latter components look similar to the expected motions between the active (closed) and deactive (open) states of the complex, but the first component appears to indicate some sort of compositional heterogeneity - \*appearing\* to indicate the loss of the hydrophilic domain in some particles. ### 5. Strategy 1: Using Per-particle scales to select good particles ![](https://guide.cryosparc.com/files/zDO9UL7PvYm659stvXzJ) To investigate further, we can split the particle set from NU Refinement (1) on the basis of per-particle scale to see what is in both of these peaks. \* Run a \[Subset Particles by Statistic\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/job-subset-particles-by-statistic) job, selecting \`Subset by\`: Per particle scale. Use the default subsetting mode that uses gaussian fitting to two clusters. To help us to understand what each of these particle sets contains (rejected particles, low scale particles and high scale particles) we can run separate 2D classifications. OPTIONAL: \* Create a 2D Classification (1), setting \`Number of 2D classes\`: 25 and drag over the rejected particles from NU Refinement (1). \* Clone 2D Classification (1) and swap the particles for Particles set 0 from Subset particles, this job will be 2D Classification (2). We had 63,943 particles in set 0. \* Clone 2D Classification (1) and swap the particles for Particles set 1 from Subset particles, this job will be 2D Classification (3). We had 26,123 particles in set 1. !\[Figure 5. 2D class averages from 2D Classifications 1, 2 and 3.\](/files/qUOAxdwjOnpSQzDfIEpr) We found that the particles that were rejected during NU Refinement did not form any coherent 2D class averages that resemble protein targets in 2D Classification (1), supporting their rejection from the pipeline. The class averages of low scale particles in 2D Classification (2) looked predominantly like some unidentified proteins embedded within a detergent micelle. Class averages of high scale particles in 2D Classification (3) resembled complex I. This sample therefore appears to contain something else in addition to intact complex I. Now that we have good evidence that the particles in the high-scale peak look like complex I, we can run another NU Refinement job with just those particles to see how the map quality and metric change. {% hint style="info" %} Initial CTF parameter estimation by Patch CTF may not be perfect; there could be variation in the particle depth in the ice affecting its defocus relative to the camera, and there may be uncorrected electron beam artefacts such as small degrees of beam tilt. Typically, when a refined volume reaches around 3.5 Å, it can be worth testing if refining the individual particle defocus values (Local CTF), and / or refining aberrations (Global CTF) improves the resolution and map quality. {% endhint %} \* Clone NU Refinement (1), change the input particles to Particles set 1 from Subset particles and set \`Optimize per-particle defocus:\` True, \`Optimize per-group CTF parameters:\` True and \`Fit anisotropic Mag\`: True. This will be NU Refinement (2). \* Compare the map quality and statistics to NU Refinement (1). !\[Figure 6. The results from Non-Uniform Refinement 2. Plots shown are global FSC, Conical FSCs, orientation distribution plot, an image of the unsharpened map, and the per-particle scale factors.\](/files/ShrhjQ3BOJfg4qm27a4r) We found that the FSC resolution and cFAR improved somewhat after removing the low-scale particles, and noted that the orientation distribution plot looked different. When junk or particles that do not match the reference volume are present during refinement, their alignment orientation can obscure the real orientation sampling of the target molecule. In this case the non-complex I particles made the orientation distribution \*appear\* more uniform, but the directional FSCs and cFAR plot indicate that those particles were not meaningfully contributing to the refined map. As the map is not drastically better than NU Refinement (1) you might wonder what was really gained by taking the time to remove the non-matching particles, but doing so is important to avoid artefacts in downstream processing that rely on poses assigned during refinement (such as Local Refinement, 3D classification and 3D Variability Analysis). In NU Refinement (2) we have achieved a map of similar resolution to the deposited map \[EMDB:13611\](https://emdb-empiar.org/EMD-13611) without the requirement of any manual curation steps. However, to do so, we made an assumption about what was present in the sample by using an existing map as an initial model, rather than running \[Ab-Initio Reconstruction\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-reconstruction/job-ab-initio-reconstruction), and so we only found what we were looking for. {% hint style="info" %} For projects investigating a specific biological question, especially for repeat targets, making assumptions about what is in the mix (intact known target and junk) can accelerate progress towards an answer. For some projects it might be sufficient to stop at this sort of stage, if the region of interest in the map shows an adequate quality of density. Something that is sometimes overlooked is that cryo-EM is a technique that is not limited to hypothesis-driven results, and datasets designed to look at one aspect, might contain information about something else entirely! Although we have already achieved a map that looks reasonable, we are going to open the processing now to explore the dataset without constraining the analysis to the expected target. {% endhint %} ### 6. Strategy 2: Discovering and sorting discrete target heterogeneity by Ab-Initio Reconstruction ![](https://guide.cryosparc.com/files/JZhmXV6ALFxWhG9agwd6) At the beginning of this processing pipeline, we chose to skip the generation of Ab-Initio volumes and tested out the assumption that the particles were all complex I. The refined per-particles scale distribution, and subsequent 2D classifications suggest that the picked particles are heterogeneous, so let’s now take a step back and see if we can identify what other particle types are present. We don’t need to include obvious junk in the Ab-Initio Reconstruction job, so we will identify the poorest classes from 2D Classification (2) and exclude them from the next step. \* From 2D Classification (2) use the Quick actions menu to create a Select 2D Classes job and queue it. In the interactive tab select any classes that look like junk and not at all like protein. We selected 5 of the 25 classes at this stage, and this gave us 55,827 rejected particles. {% hint style="info" %} To speed up Ab-Initio Reconstruction, the images can be downsample first so that the Nyquist frequency is the maximum resolution that you want to use. {% endhint %} \* Create a Downsample Particles job and input the rejected particles from Select 2D, and set the \`Desired approx. pixel size (A)\` to 6 and \`Save results in 16-bit floating point:\` true. This gave us a box size of 80 pixels. \* Create an Ab-Initio Reconstruction job (Ab-Initio (1)), inputting the downsampled particles and use the following settings: | Parameter | Setting | Reason | | --- | --- | --- | | `Number of Ab-Initio classes` | 14 | Using a lot of classes improves chances of finding a greater number of heterogeneous structures | | `Num particles to use` | your total | Specifying this value ensures that the entire particle stack is used | | `Initial Resolution (Angstrom)` | 18 | Some small or membrane target benefit from starting Ab Initio at higher resolution than the default of 35 Å | | `Fourier radius step` | 0.005 | Reducing this value causes the job to take finer steps as the resolution ramps up | | `Initial minibatch size` | 300 | | | `Final minibatch size` | 1000 | | | `Enforce non-negativity` | False | | The number of classes, turning off enforcing non-negativity, and initial resolution values used above were determined empirically from repeat runs with different settings. We took inspiration from \[Kim et al. (2025)\](https://www.biorxiv.org/content/10.1101/2025.09.08.674935v1) for the minibatch size and Fourier radius step. OPTIONAL: As a control experiment, run a second Ab-Initio job (2) with 14 classes but otherwise default settings. We found that Ab-Initio (1) took 8 hrs 10 mins, and Ab-Initio (2) took 1 hr 20 mins. Examine the maps from your Ab-Initio job(s). We show example volumes in Figure 7. Although Ab-Initio (1) took significantly longer to run than Ab-Initio (2), this paid off, because it was able to uncover a wider range of useful volumes. We know that this sample came from a native source of mitochondrial membranes, and was purified without over-expression or affinity tags, so we might expect to see minor contamination of the preparation with other abundant mitochondrial membrane complexes in the particle population. !\[Figure 7. Volumes from Ab-Initio jobs 1 and 2. Ab-Initio Volumes are shown beside PDB models with a semitransparent overlay of a molmap at 20 Å generated using ChimeraX. The PDBs used were complex I; 7psa, complex IV; 2occ (one monomer shown), complex III; 8ibg and complex V; 5ara.\](/files/QKLrwmjGfumq0xmdpRj2) A friendly neighbourhood mitochondrial biologist could probably identify the shape of some of these volumes in this case, but as a general rule, having knowledge about how the sample was prepared, and what impurities were identified by orthogonal techniques such as mass spectrometry can help solve the mystery of unexpected cryo-EM density maps that appear during processing. \* Compare your Ab-Initio volumes to the shape of models for mammalian mitochondrial respiratory complexes II, III, IV and V that are available in the Protein Data Bank. Download PDBs complex I; \[7psa\](https://www.rcsb.org/structure/7PSA), complex IV; \[2occ\](https://www.rcsb.org/structure/2OCC), complex III; \[8ibg\](https://www.rcsb.org/structure/8IBG) and complex V; \[5ara\](https://www.rcsb.org/structure/5ARA). In Ab-Initio job (1) that used custom settings, we were able to identify complex I (with strong or weak density for part of the hydrophilic domain), complex IV monomer, complex IV dimer, complex IV tetramer, complex III dimer and complex V. We were unable to identify any classes that resembled complex II. When we examined the volumes from Ab-Initio job (2) we were only able to identify volumes that looked like complex I and a monomer, and a dimer of complex IV. You may observe multiple classes for complex I and IV. Due to stochasticity in the Ab-Initio job, and particle selections, you might not observe one or more of the volume types shown above. Repeating Ab-Initio Reconstruction will sometimes manifest a different selection of volume types, for example you might see a volume for complex V in Ab-Initio (2) but we consistently found the same volume types on repeat jobs of Ab-Initio (1) with our particle set. ### 7. Strategy 2: Separation and Refinement of different discrete targets by Hetero Refinement We now want to use Heterogeneous Refinement to better separate the different target complexes that we have identified. We can speed up and simplify analyses by selecting only 1 volume for each target type. \* Create a Heterogeneous Refinement job (Hetero Refine (1)). Add one volume each for complex I, complex III, complex IV monomer, complex IV dimer, complex IV tetramer, and complex V along with the remaining junk volumes and connect up the particles from NU Refinement (1). We already ascertained in 2D classification (1) that the rejected particles are not likely to be useful, we will not consider them further. We show example output volumes from Hetero Refine (1) in Figure 8. !\[Figure 8. Volumes from Heterogeneous Refinement. PDB models fitted into each of the protein complex volumes. The PDBs used were complex I; 7psa, complex IV; 2occ (one monomer shown), complex III; 8ibg and complex V; 5ara.\](/files/7Ok8XNYgK3l7EPKk6oq7) We found that among the non-rejected particles from NU Refinement (1), 17% were assigned to junk classes and 83% were assigned to mitochondrial complex volumes. Recall that we did not curate the particles picked by TOPAZ, and relatively few were rejected at the NU Refinement stage, and so this result indicates a good quality of initial particle picking. This sample was prepared in order to study the structure of inhibited complex I, and the original analyses used template picking from the start, which may have missed the other complexes. ### 8. NU Refinement of the individual complexes {% hint style="info" %} It is a good idea to consider if your target particle displays rotational symmetry. You can rotate the volume around and consider if it looks the same after rotation in one or more directions. When a target is symmetric, we can make use of this by applying symmetry during refinement, effectively increasing the signal-to-noise ratio, and often improving the map quality. Care should be taken when selecting symmetry for refinement: applying incorrect symmetry, or applying symmetry to a pseudo-symmetric target can lead to poor map quality, artifactual density or loss of interesting asymmetric map features. {% endhint %} \* Rotate the volumes around and consider if there is symmetry present - we identified what looked like C2 symmetry for complex III dimer, and complex IV tetramer. This means that the maps looked very similar after rotating by 180 degrees. See Figure 9 for examples. \* Examine the output volumes and compare them in ChimeraX to the PDB files that we used above to check the hand of the volumes. This strategy is useful when the map resolution is not high enough to identify the handedness of alpha helices. If you are unsure which hand fits better, you can look at the “average map value” reported in the log. \* For volumes that require a hand flip to match the model: Run a Volume Tools job with that volume as an input and set \`Flip hand:\`True, to give a volume with the correct hand. !\[Figure 9. Manual checks of maps for symmetry and handedness.\](/files/OpCvK5nMldz1LW0SfVwJ) \* Run a Non-Uniform Refinement (3) inputting the correct hand volume and particles for complex I and set \`Use dynamic refinement mask\`: false \* Run a Non-Uniform Refinement (4) inputting the correct hand volume and particles for complex IV monomer and set \`Use dynamic refinement mask\`: false \* Run a Non-Uniform Refinement (5) inputting the correct hand volume and particles for complex IV dimer and set \`Use dynamic refinement mask\`: false \* Run a Non-Uniform Refinement (6) inputting the correct hand volume and particles for complex IV tetramer and set \`Use dynamic refinement mask\`: false and \`Symmetry\`: C2 \* Run a Non-Uniform Refinement (7) inputting the correct hand volume and particles for complex V and set \`Use dynamic refinement mask\`: false \* Run a Non-Uniform Refinement (8) inputting the correct hand volume and particles for complex III and set \`Use dynamic refinement mask\`: false and \`Symmetry\`: C2 {% hint style="info" %} In CryoSPARC v5 you can copy job parameters and paste them to other jobs, so you don’t need to set them individually each time! See the guide page on \[copying and pasting parameters\](https://guide.cryosparc.com/application-guide/creating-and-running-jobs#copy-and-paste-parameters) for more details. {% endhint %} We found that the maps for complex I and complex IV dimer were sufficiently high at \\~3.5 Å that they might benefit from CTF Refinement of per-particle defocus, and per-group beam tilt, trefoil and anisotropic magnification. {% hint style="info" %} When the map resolution approaches the Nyquist frequency that map features become a bit jagged due to under-sampling of the map pixel size. If you have applied Fourier-cropping during particle extraction, you can re-extract the particles less, or no Fourier cropping to improve the map smoothness. It can also be beneficial to re-extract particles after they have been aligned in 3D in order to better centre the extracted region for the particle. {% endhint %} If you have an estimated map resolution of \\~3.5 Å, then: \* Clone the extraction job from Section 3 and swap the particles for those in NU Refinement (3) (Extract from Micrographs (2)) \* Clone the extraction job from Section 3 and swap the particles for those in NU Refinement (5) (Extract from Micrographs (3)) Using these particles re-run NU refinement: \* Clone NU Refinement (3) and set \`Optimize per-particle defocus:\` True, \`Optimize per-group CTF parameters:\` True and \`Fit anisotropic Mag\`: True. This will be NU Refinement (9). \* Clone NU Refinement (3) and copy over the parameters from NU Refinement (9). This will be NU Refinement (10). \* Examine your maps and look at the unsharpened maps, auto-tightened resolution masks, estimated resolution, and cFAR values. We found that NU Refinements (9) and (10) had better global FSC resolution than the counterpart refinements (3) and (5) that did not include CTF Refinement. {% hint style="info" %} In CryoSPARC v5 you can see the near and far extents of the mask relative to the map in the Dashboard or Event Log under Real Space Auto Tight Mask Slices (see examples in Figure 10). This can give an overview feeling for if the mask is adequately covering the density. Regions of weak map density might exist in between the near and far extents, or even outside of the mask entirely, such as in the case of the detergent micelle. It is always a good idea to take a look at the masks used for FSC resolution estimation in ChimeraX while you inspect the map, to satisfy yourself that the mask adequately encapsulates your map density. {% endhint %} !\[Figure 10. NU Refinement of mitochondrial complexes. Sliced views of the auto tightened FSC mask relative to the refined volume, and cFSC plots for each of NU Refinements 9,4,10,6,7 and 8.\](/files/G88Cg5pRG0807ku2AGQr) Our NU Refinement (9) of complex I looks very similar to that in NU Refinement (2), with a good cFAR indicating a wide range of orientations being sampled. The dimer of complex IV also has good orientation sampling, however the other refinements that contain fewer particles have more limited orientations present and therefore caution might be prudent when considering the estimated map resolutions. ### 9. Ideas for comparing different classification strategies for discrete targets Which classification strategy worked better for complex I? NU Refinement followed by subsetting the particles on the basis of scale, or Ab-Initio and Hetero Refine? We can check out the following to make a judgement on which one we like the best: 1. compare the unsharpened map quality, especially in the region of interest 2. compare the map statistics such as GSFSC and cFAR 3. find the uncommon particles (i.e. the ones only assigned to complex I in one or other of the strategies) and investigate if they are complex I or not We could not discern any meaningful differences between the map quality and statistics of NU Refinements (2) and (8). To look at the uncommon particles: OPTIONAL: \* Run a Particle Sets Tool job (1) with the particles from NU Refinement (2) in particles (A) and the particles from NU Refinement (9) in particles (B) with the Action set to Intersect. In our case we found 23,362 particles common to both refinements, and 4,505 that were uncommon (2,323 in \\\[A minus B\], and 2,182 in \\\[B minus A\]). We expect to see some uncommon particles in any experimental datasets with different classification strategies or repeat runs. This is partly due to variations in signal-to-noise in the particle images that affect the reliability of particle alignment. This affects the particle scale values, and class assignment in classifications such as Heterogeneous Refinement. We can examine what the uncommon particles look like by running 2D Classifications. OPTIONAL: \* Create a 2D Classification (4), setting \`Number of 2D classes\`: 25 and drag over the \\\[A minus B\] particles from Particle Sets Tool (1) \* Create a 2D Classification (5), setting \`Number of 2D classes\`: 25 and drag over the \\\[B minus A\] particles from Particle Sets Tool (1) We can also run a control 2D Classification to look at the class average quality of similar number of the common particles OPTIONAL: \* Run a Particle Sets Tool job with the \\\[Intersection (keeping A data)\] particles from NU Refinement (2) in particles (A) and the particles from NU Refinement (8) in particles (B) with the Action set to Split, and a value similar to the number in 2D Classifications (4) and (5). OPTIONAL: \* Create a 2D Classification (6), setting \`Number of 2D classes\`: 25 and drag over the \\\[A minus B\] particles from Particle Sets Tool !\[Figure 11. 2D class averages from 2D Classifications 4,5, and 6. One class that resembles a complex IV dimer is circled in yellow.\](/files/y0OgHrFqqrMHGmaTEgzw) In Figure 11 we show 2D class averages from our 2D Classifications (4),(5) and (6). In 2D Classification (4) we noted a class that looked like a complex IV dimer, but this was not seen in 2D Classifications (5) or (6), in which most of the classes are identifiably complex I, despite the very low particle number (\\~2,200 particles). Overall, our initial experiment of NU Refining against an existing map, and removing junk and other species on the basis of per-particle scale worked pretty well and took a relatively short time, but the resulting particle stack from the strategy using Ab-Initio and Hetero Refine looks slightly cleaner. Either Strategy 1, or Strategy 2 might give enough map detail to answer the question at hand, allowing processing to stop here, but for the curious, we have not yet exhaustively examined the heterogeneity present in the dataset. ### 10. Investigating residual heterogeneity using 3D Variability Analysis We have so far refined 6 maps in Section 8, but how can we be sure that each of the 6 particle sets used are homogeneous? At any point during processing when you want to look for remaining substantial heterogeneity (discrete or continuous), \[3D Variability Analysis\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-3d-variability?q=3dva) can be very useful. As we have 6 targets to consider, we will run 3DVA with just 1 mode to look for the most substantial density change amongst each refined particle set. For each one, we want to select a filter resolution that is lower than the FSC resolution of the input map. \* Create a 3D Variability Analysis job (2) inputting the map and resolution mask from NU Refinement (9) (complex I), set \`Number of modes to solve\`: 1, and \`Filter resolution\`: 8 \* Create a 3D Variability Analysis job (3) inputting the map and resolution mask from NU Refinement (4) (complex IV monomer), set \`Number of modes to solve\`: 1, and \`Filter resolution\`: 12 \* Create a 3D Variability Analysis job (4) inputting the map and resolution mask from NU Refinement (10) (complex IV dimer), set \`Number of modes to solve\`: 1, and \`Filter resolution\`: 8 \* Create a 3D Variability Analysis job (5) inputting the map and resolution mask from NU Refinement (6) (complex IV tetramer), set \`Number of modes to solve\`: 1, and \`Filter resolution\`: 12 \* Create a 3D Variability Analysis job (6) inputting the map and resolution mask from NU Refinement (7) (complex III), set \`Number of modes to solve\`: 1, and \`Filter resolution\`: 12 \* Create a 3D Variability Analysis job (7) inputting the map and resolution mask from NU Refinement (8) (complex V), set \`Number of modes to solve\`: 1, and \`Filter resolution\`: 12 \* For each of the 3D Var jobs use the Quick actions menu to make 3D Variability Display (3DV Display) jobs (2-7), and set the \`Filter resolution\` to match that used in the corresponding 3D Var job. To make the files smaller, for faster download also set \`Downsample to box size:\` 128. \* Examine the movies in ChimeraX and take a look at the shape of the 3DVA mode histogram. We show examples of these in Movie 2 and Figure 12. ![](https://guide.cryosparc.com/files/Hof8YushctkRmkWSsRAQ) **Movie 2: 3DVA volumes from 3DV Display jobs 2-7.** !\[Figure 12. Histograms of component 0 from 3DVA jobs 2-7.\](/files/qG6l8QXkbA2V3hDC9eSb) We found that the motion of complex I and the complex IV tetramer looked interesting. The motion in complex III, as well as complex IV monomer and dimer look like variations largely in the micelle region, that are expected even within a small fairly homogeneous particle stack, and the motion seen in complex V might indicate that there is a subpopulation that has damage to part of the complex. We noticed that the particle distribution across the component was bimodal in the case of complex I, and trimodal in the case of complex IV tetramer. We find that a multimodal distribution in 3DVA often indicates that there is discrete heterogeneity in the particle stack, whereas a single peak is more often indicative of continuous heterogeneity. We will not consider further the changes in complex V due to the low resolution of the map, but we will now go on to look at separating the discrete states that might be present in the complex I and complex IV tetramer. We will do this via three different strategies, then compare the results from each: \* Section 11: Strategy 3 - 3DVA Cluster mode \* Section 12: Strategy 4 - 3D Classification \* Section 13: Strategy 2 revisited - Ab-Initio and Hetero Refine ### 11. Strategy 3: Sub-classification by 3DVA Cluster mode ![](https://guide.cryosparc.com/files/namB6DrItxLAO58tQ0e7) In 3DVA (2), for complex I, we saw a bimodal distribution (see Figure 12), indicating possibly two discrete classes, we can therefore use 3D Variability Display in Cluster mode in order to obtain particle sets that correspond to the two peaks seen. \* Using the Quick actions menu from 3DVA (2) create a 3DV Display job (8) with the following settings: | Parameter | Threshold | Reason | | --------------------------------- | --------- | -------------------------------------------------- | | \`Output mode\` | cluster | Output clustered particle sets | | \`Number of frames/clusters\` | 2 | The number of peak observed | | \`Downsample to box size\` | 128 | Reduce box size by Fourier cropping | | \`Crop to size (after downsample)\` | 100 | Further reduce the box size by real-space cropping | In 3DVA (5), for complex IV tetramer, we saw a trimodal distribution (see Figure 12), indicating possibly three discrete classes, we can therefore use 3DV Display in Cluster mode in order to obtain particle sets that correspond to the three peaks seen. \* Using the Quick actions menu from 3DVA (5) create a 3DV Display job (9) with the following settings: | Parameter | Threshold | Reason | | --------------------------- | --------- | ----------------------------------- | | \`Output mode\` | cluster | Output clustered particle sets | | \`Number of frames/clusters\` | 3 | The number of peak observed | | \`Downsample to box size\` | 128 | Reduce box size by Fourier cropping | Note, that for the complex IV tetramer, we do not crop the box in real space, because the volume for the tetramer takes up more of the box, and we don’t want to cut it off by cropping too much. \* Examine the output volumes and decide if you think that they might represent different states. Sometimes it might be ambiguous, because the input particle poses came from a job with them all mixed together. We can run fresh NU Refinement jobs to estimate the poses again from scratch using each of the particle stacks and volumes to see if they produce volumes with distinct features. \* Clone NU Refinement (9) and swap out the volume and particles from 3DV Display (8) cluster 0, this is NU Refinement (11) \* Clone NU Refinement (9) and swap out the volume and particles from 3DV Display (8) cluster 1, this is NU Refinement (12) \* Clone NU Refinement (5) and swap out the volume and particles from 3DV Display (9) cluster 0, this is NU Refinement (13) \* Clone NU Refinement (5) and swap out the volume and particles from 3DV Display (9) cluster 1, this is NU Refinement (14) \* Clone NU Refinement (5) and swap out the volume and particles from 3DV Display (9) cluster 2, this is NU Refinement (15) \* Examine the output volumes and refinement statistics. !\[Figure 13. Separation of complex I and complex IV tetramer conformations using 3D Variability in cluster mode.\](/files/9UpXqhLROOKQX6fbhQLO) For complex I, we found the map was poorer in NU Refinement (11) (4,588 particles, cFAR 0.72) and some regions of the map had relatively poor density. On the other hand, the map from NU Refinement (12) had similar map statistics and global quality as NU Refinement (9). We noticed global changes in the maps between NU Refinement (11) and (12) similar to the motion seen in Movie 2 where the two arms are more open in the smaller class, and are more closed in the larger class (see Figure 13). In addition we find that there is a region of density that has good definition in NU Refinement (12) but is absent in NU Refinement (11) (indicated in the inset image, circled). Both the global open/closed conformation, and ordering/disordering of the circled region (part of subunit NDUFA9) are documented characteristics of the active and deactive states, so we can tentatively assign NU Refinement (11) to the deactive state, and NU Refinement (12) to the active state. For the complex IV tetramer, we found that NU Refinement (15) was of very poor quality, but NU Refinements (14) and (13) produced low resolution maps showing the oxidase monomers in two different packing conformations, affecting the shape of the detergent micelle. One micelle is more of a parallelogram, and the other is more oval and we will refer to them as conformation A and conformation B, respectively. Although we only started with \\~5,000 particles we were still able to separate two unique states with 3DVA by looking for very low resolution changes (12 Å). Success here might reflect the resolution range being able to capture changes in micelle shape and relative positions of entire monomers. We have evidence now that there are at least two good classes of complex I, two good classes for the complex IV tetramer, and perhaps some poor quality or junk particles in NU Refinement (15), but we don’t know if 3DVA was the best way to separate the particle sets. ### 12. Strategy 4: Sub-classification by 3D Classification ![](https://guide.cryosparc.com/files/r1w40usGBVYku94Xgznc) As we already have particle poses assigned for complex I and complex IV tetramer in NU Refinements (9) and (6) respectively, we can go ahead and try \[3D Classification\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-3d-classification-beta) to compare the results that we get with the same number of classes as we used for 3DVA clusters. \* Create a 3D Classification job and input the particles from NU Refinement (9), set \`Number of classes\`: 2, \`Filter resolution\`: 8, and \`Use latent mixing coefficients\`: true \* Create a 3D Classification job and input the particles from NU Refinement (6), set \`Number of classes\`: 3, \`Filter resolution\`: 12, and \`Use latent mixing coefficients\`: true For both jobs, we enabled latent mixing coefficients (a new option in CryoSPARC v5), and this can help reduce the likelihood of encountering a great number of similar-looking classes with equal particle counts. For each output class, run NU Refinements (16-17 for complex I and 18-20 for complex IV tetramer), using the same settings as Section 9. \* Examine the maps and refinement statistics {% hint style="info" %} Both 3DVA and 3D classification rely on good input particle poses from an upstream refinement, but if one or more particle populations in the refinement don’t align to the consensus volume very well, then the results of 3DVA and 3D Classification may be suboptimal. It is sometimes therefore worth considering a classification strategy that also allows poses to be simultaneously updated during particle sorting, such as Ab-Initio Refinement and Heterogeneous Refinement. {% endhint %} We will compare the classification strategies from Sections 11, 12 and 13 later on. ### 13. Strategy 2 revisited: Sub-classification by Ab-Initio and Hetero Refine ![](https://guide.cryosparc.com/files/Nt7kjMi74kwnDtzaD8Dv) We have separated complex I into two classes, and complex IV tetramer into 3 classes using 3DV cluster mode, and 3D Classification and will now separate particles using Ab-Initio and Hetero Refine. This is the same strategy that we used to initially discover and separate the 6 different targets that we have found within this dataset in Sections 6 and 7. \* Run an Ab-Initio Reconstruct job (3) inputting the particles from NU Refinement (9) (complex I), setting \`Number of Ab-Initio classes\`: 2 \* Use the Quick actions menu to queue up a Hetero Refine (2) \* Run an Ab-Initio Reconstruct job (4) inputting the particles from NU Refinement (6) (complex IV tetramer), setting \`Number of Ab-Initio classes\`: 3 \* Use the Quick actions menu to queue up Hetero Refine (3) \* For each output class, run NU Refinements (21-22 for complex I and 23-25 for complex IV tetramer), using the same settings as Section 9. ### 14. Comparing classification strategies for complex I All three of the strategies used in Sections 11-13 might produce maps with sufficient information and features to answer the biological question at hand, but sometimes one method performs noticeably better than the others. It can be tempting to rely on finding the highest FSC resolution, but this does not always indicate the most accurate particle sorting. {% hint style="info" %} The populations assigned to each state are not always the same for different particle sorting methods. In this example, we found that with 3DVA Cluster mode, 3D Classification, and Ab-Initio Reconstruction followed by Hetero Refine, the curated complex I in this sample is 18.6, 16.3 or 21.2% in the deactive state. A range of values is expected for experimental samples, and reflect the inherent noisiness of the images being sorted, stochastic differences, and the way that they are being analysed. {% endhint %} \* Identify which three sub-classified NU Refinements have a smaller population and match the deactive state and examine these maps for the following steps. OPTIONAL: \* Run a Particle Sets Tool job (5) with the particles from NU Refinement of particles from 3DVA in input:particles (A) and the particles from NU Refinement of particles from 3D Classification in input: particles (B) with the Action set to Intersect. \* Run a Particle Sets Tool job (6) with the particles from NU Refinement of particles from 3DVA in input:particles (A) and the particles from NU Refinement of particles from Hetero Refine (2) in input: particles(B) with the Action set to Intersect. \* Run a Particle Sets Tool job (7) with the particles from NU Refinement of particles from 3D Classification in input: particles (A) and the particles from NU Refinement of particles from Hetero Refine (2) in input particles: (B) with the Action set to Intersect. We show the results of the intersected (common) particles in the table below. | Classification strategy | 3D Classification | Ab Initio & Hetero Refine | | ----------------------- | ----------------- | ------------------------- | | 3DVA cluster | 3,848 | 3,784 | | 3D Classification | x | 3,497 | We found that the particle class assignment using 3DVA cluster mode, and 3D Classification were very similar. Classifying by Ab-Initio Reconstruct and Hetero Refine led to different class assignment for more particles, but to see if all methods are equivalent or if there is a clear winner, you can examine more closely the map features that are specific for one or more of the states you are hoping to isolate. In this case, we know that the deactive state has a relatively open configuration between the hydrophobic and hydrophilic arms. We can align the three maps from NU Refinement that best match the deactive state and compare their global conformations. In Movie 3 we show a volume morph between our NU Refined volumes after classification by 3D Classification, 3DVA cluster mode, and Ab-Initio reconstruct followed by Hetero Refine. We find that the map that has the most “closed” conformation is after 3D Classification, the map after 3DVA cluster mode is intermediate, and the volume refined after Ab-Initio Reconstruct followed by Hetero Refine is subtly more “open”, so for this specific target and conformational change, we might decide therefore that Ab-Initio Reconstruct followed by Hetero Refine is a good way to separate these particles. ![](https://guide.cryosparc.com/files/HQiTem5rkTRDBMD2DeIl) **Movie 3. Volume morph between the deactive NU Refined states.** Particle sets separated by 3D Classification (purple), 3DVA cluster mode (orange) and Ab-Initio Reconstruct and Hetero Refine (green). We also went on to compare the map statistics and quality between the various refinements run on the complex I active state in Figure 14, along with the unsharpened map generated from the deposited half maps in emdb:13611. We made a new autotightened FSC mask using CryoSPARC to accompany the deposited map in order to allow a more direct comparison of the map statistics of the deposited and experimental maps produced here. We also chose to compare unsharpened maps, to avoid introducing another variable per map, in the form of different sharpening B-factors. For this reason, the map features shown in Figure 14 might not quite match those expected from a sharpened map at the reported resolution. !\[Figure 14. Map statistics and unsharpened map density for complex I active state maps emdb:13611, and NU Refinements 1,2,3,9,12,16 and 22. Example density is shown for a representative transmembrane helix in subunit ND3. \\\*Value was calculated using a new mask generated in CryoSPARC.\](/files/InHmAEqQOeaNn6JaGSyy) The best map quality was observed in NU Refinement jobs (9), (12), (16) and (22), and the FSC resolution and cFAR values were slightly improved by removing the deactive particles in classification by strategies 2,3, or 4. ### 15. How global refinement settings can influence map appearance The map quality of NU Refinement (1) in Figure 14 is not dramatically different to that in the later refinements of complex I, even though \\~75% of the particles present weren’t even the same target molecule - how can that be? {% hint style="info" %} During global refinement jobs in CryoSPARC, there are a few ways that particles, or map regions can be handled differently, depending on the settings. \* Particle scale minimization - after alignment to the reference volume, particles with relatively poor matching contrast to the volume projection in that pose are down weighted, and vice-versa. \* Pose marginalisation - when a particle has poor contrast, or does not match the reference volume well, it’s “best” pose can be uncertain. To help with this, instead of being assigned a single pose, the contributions of particles to the reconstructed map can be spread (marginalized) across a range of the best poses for each particle. \* Masking - Applying a mask can prevent density from neighbouring molecules, or low resolution regions, from interfering with particle alignment. \* Non-Uniform regularization - Regions of the map that are relatively disordered, such as a detergent micelle, can lead to overfitting of the map, but Non-Uniform regularization allows these regions to be blurred out and focus the refinement pose assignment on the well-ordered regions of the map. {% endhint %} We ran a set of additional jobs on the initial extracted particles (and the active state particles from Hetero Refine (2)) to investigate the effects of the 4 settings listed above. Note that Per-particle scale minimization is enabled by default, and the Non-Uniform Refinement job is the same as the Heterogeneous Refinement, except that it has both Adaptive Marginalization (pose marginalization), and Non-Uniform Regularization turned on as a default. Optional: Run NU Refinement jobs, altering the following settings, to see how the map appearance changes: In Figure 15 we show the map appearance and conical FSCs for A) the initial particle stack that contains all of the complexes (93,230 particles), and B) the complex I active state particles from Hetero Refine (2) (19,090 particles). !\[Figure 15. Example density and cFSCs for refinements of A) particles from the initial extraction, and B) the active complex I class from Hetero Refine 2. Base refinement refers to homogeneous refinement without per-particle scale minimisation or dynamic masking, and the other refinements have NU regularization, particle scale minimization, Adaptive Marginalization and/or Dynamic masking applied, as indicated.\](/files/OsJTwchdkshNKUObyp1s) First, we look at the refinements that were run in the Case Study - NU Refinements (1) and (21) that were run with Non-Uniform Regularization, Adaptive Marginalization, and Per-particle scale minimization, they both have similar side chain definition and cFAR values, but when we strip those settings away and run a base refinement (i.e. homogeneous refinement without particle scale minimisation or dynamic masking) the maps between the particle sets look quite different! The base refinement before classification is poor, with a low cFAR (0.36) and very little side chain definition. On the other hand, the base refinement from the classified active complex I has a better cFAR at 0.60, and better side chain definition in the map. Separately enabling Non-Uniform Regularization, Per-particle scaling, Dynamic masking, or Adaptive Marginalization improves the cFAR and map quality for both particle sets. For the active complex I particles (Figure 15 B), and both Non-Uniform Regularisation and Dynamic masking have similar results, and they are also similar to NU Refinement (21), leading us to consider that the major issue with the base refinement may be that the micelle density or density from adjacent particles are interfering with good alignment of particles to the target protein region. Note that combining Dynamic masking with Non-Uniform Regularization usually does not have any additive benefit as both strategies mitigate a similar problem with different approaches, and so we did not use Dynamic Masking in NU Refinements (1) or (21). For the initial extracted particles (Figure 15 A) we see that Non-Uniform Regularization provides the most noticeable improvement in map quality and cFAR compared to the base refinement (0.56 vs 0.36). Unlike the active complex I particles, the map quality from Non-Uniform Regularization alone is not as good as NU Refinement (1) which also includes Adaptive Marginalization and Per-particle scale minimization. This result indicates that in addition to low resolution regions interfering with particle alignment, there are particles in the stack that are challenging to align to the refinement volume, and that the refinement is better when the contribution of those particles to reconstruction is reduced. {% hint style="info" %} Non-Uniform Regularization, Adaptive Marginalization and Per-particle scale minimization can powerfully improve map quality and map metrics, and are usually recommended for membrane proteins especially where the stack contains particles with relatively poor contrast (such as from thick ice or low defocus). As we have seen in this case study, these settings can also mitigate the effects of non-matching or junk particles. This might provide a useable map, but makes heterogeneity less obvious, and downstream jobs such as 3D Classification might produce unexpected, or unreliable results due to the presence of a large number of poorly aligned and non-matching particles. For this reason it is important to investigate the particle scale plot from global refinement, and even if you are working with a membrane protein, it can sometimes be a good idea to run a test homogeneous refinement for comparison. {% endhint %} ### 16. Comparing classification strategies for complex IV tetramer \* Identify which of the sub-classified NU Refinements match the complex IV conformations A, B and poor quality map shown in Figure 13 and examine and compare the map quality for each. We found the map quality, FSC resolution and cFAR values to be similar for the conformation A and B maps from all three classification strategies, and we were not able to choose one strategy that looked better than the others. We looked at the proportion of the complex IV tetramer particles that got assigned to each of the three classes from the three different strategies. | Classification strategy | Particles in conf A | Particles in conf B | Particles in poor quality class | | --------------------------------------- | ------------------- | ------------------- | ------------------------------- | | 3DVA cluster | 1,283 (27%) | 1,131 (24%) | 2,306 (49%) | | 3D Classification | 1,946 (41%) | 1,702 (36%) | 1,072 (23%) | | Ab-Initio Reconstruct and Hetero Refine | 1,916 (41%) | 2,377 (50%) | 423 (9%) | We noticed that the proportion of particles assigned to conformation A, B and the poor resolution class varied quite substantially, with Ab-Initio Reconstruct followed by Hetero Refine assigning a much smaller poor quality class. The maps from any of these three strategies could be equally useful, however some caution may be prudent in the interpretation of the populations of class assignment in this case. ### Interpretations and Conclusions EMPIAR 10927 contains a high degree of discrete heterogeneity and at least 8 distinct targets/conformations. Among these were complex IV in dimer and tetramer conformations. We noted that the density for monomers of complex IV were arranged antiparallel, and this is unexpected from a function perspective, as it would enable the active sites to reside on both sides of the mitochondrial inner membrane. It seems likely that these dimer and tetramer conformations may arise during the necessary detergent solubilisation and sample concentration steps during purification. Co-purification of native, or even tag-assisted purifications with other proteins is common, and understanding the origin and purification process can accelerate interpretation of the cryo-EM data. \*\*A rapid strategy of refining against an existing map, and selecting the high-scale peak worked for a hypothesis-driven classification of particles\*\* {% hint style="info" %} In NU Refinement (1), despite no manual particle curation steps, we already achieved a similar resolution and map quality, and better cFAR than the deposited map, with NU Refinement (2) showing slightly better reported resolution and map features. These maps would be adequate for identifying the inhibitor binding location and took less than 3 hours of processing time to achieve for this small dataset. It is possible that the improved cFAR reflects the reduced influence of human choice during our processing, which we find can be a source of unintentional bias, especially at the 2D classification stage. An automated pipeline using the strategy of refining all picks with an existing high quality map, followed by splitting the particles on the basis of per-particle scale could easily be built for other targets with pre-set thresholds for exposure curation. Importantly, this simple approach did not uncover the rich heterogeneity present in the dataset. {% endhint %} \*\*A longer strategy of many class Ab-Initio and Hetero Refine uncovered unexpected target heterogeneity\*\* {% hint style="info" %} In Section 7 we found that the particle stack used in NU Refinement (3) (after Ab-Initio Reconstruct and Hetero Refine) was slightly cleaner, and the side chain features in NU Refinement (9) were slightly improved despite apparently similar reported resolution and cFAR. Not only was the complex I map better with this strategy, but it also uncovered 5 other discrete species that were present in the sample. {% endhint %} \*\*Evaluation of different classification sorting strategies\*\* {% hint style="info" %} The particle sorting route that you choose for your dataset will be highly dependent on your aims, and the nature of the sample. Here, we provided ideas about ways to compare different strategies, and ultimately, a good aim might be for the best per-class map quality, and an idea of how reproducible the class assignment is. {% endhint %} ### \*\*References\*\* \[Dan Grba \*et al.\* (2022) \*\*Cryo-electron microscopy reveals how acetogenins inhibit mitochondrial respiratory complex I\*\* \*J Bio Chem\* \*\*298(3)\*\*:101602\](https://www.jbc.org/article/S0021-9258\\(22\\)00042-4/fulltext) \[Tristan Bepler \*et al\*. (2019) \*\*Positive-unlabeled convolutional neural networks for particle picking in cryo-electron micrographs\*\*. \*Nat Methods\* \*\*16,\*\* 1153–1160\](https://www.nature.com/articles/s41592-019-0575-8) \[Kookjoo Kim \*et al\*. (2025) \*\*High-resolution ab initio reconstruction enables cryo-EM structure determination of small particles\*\* \*bioRxiv\* 2025.09.08.674935\](https://www.biorxiv.org/content/10.1101/2025.09.08.674935v1) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-discrete-heterogeneity-in-a-sample-of-acetogenin-bound-complex-i-empiar-10927.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/job-2d-classification.md). # Job: 2D Classification ## At a Glance ![](https://guide.cryosparc.com/files/Nxkcr9he4NF7cV5cinzm) Rapidly classify particle images based on their in-plane rotation. ## Description Single particle cryo-EM images are essentially 2D projections of 3D objects. The ultimate goal of single-particle analysis is reconstruction of one or more 3D volumes; however, the calculations necessary for these reconstructions are expensive, relatively slow, and sensitive to noise and outliers in the data. Thus, it can be beneficial to perform an initial “clean-up” step in 2D to quickly discard particle images which are obviously junk or the wrong particle, and also to quickly visualize the contents of a dataset before proceeding further with processing. 2D Classification groups particles into a specified number of classes. Because the technique is using only two dimensions, the particles are only rotated and translated in-plane (i.e., only rotated and translated as you could with a flat, printed version of the image). The average of all particles in a class (called a \*class average\*) typically has a significantly better signal-to-noise ratio than any single particle image. As such, it is much easier to identify and discard a bad class average than it is a single particle image. 2D Classification is fast and useful in many cases. However, it also has important limitations and caveats to consider. We encourage all users to read the \[Common Problems\](#common-problems) and \[Recommended Alternatives\](#recommended-alternatives) sections and keep them in mind while preparing to run or analyze a 2D Classification job. The \[Common Problems\](#common-problems) section discusses the two most common failure modes of 2D Classification: streaky or noisy classes and classes with too many or too few junk classes. It also provides some suggested parameter tweaks to improve results when these issues are observed. The \[Recommended Alternatives\](#recommended-alternatives) section provides an in-depth explanation of why, in some cases, 3D particle curation workflows are recommended over 2D classification. {% hint style="info" %} 2D Classification is one of many jobs in cryo-EM which are based on the general principle of Expectation Maximization. For a detailed explanation of Expectation Maximization and how it works in cryo-EM, please see the dedicated guide page. {% endhint %} ## Inputs ### Particles In CryoSPARC, particle coordinates and extracted particle images are both "particle" type outputs. For 2D Classification, the particles must have been extracted (i.e., they must have a “blob” field). If you encounter an error message when you launch a 2D Classification job that reads > AssertionError: Non-optional inputs from the following input groups and their slots are not connected: particles.blob. Please connect all required inputs. you have connected particle \*locations\* only, and must extract them (using \[Extract from Micrographs\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/extraction/job-extract-from-micrographs)) before performing 2D Classification. 2D Classification also requires images to have CTF estimates, which will be included when particles are extracted from micrographs with CTF estimates (e.g., from \[Patch CTF Estimation\](/processing-data/all-job-types-in-cryosparc/ctf-estimation/job-patch-ctf-estimation.md)). ## Commonly Adjusted Parameters ### Number of 2D Classes This parameter sets the number of classes into which particles will be grouped. In general, as the number of classes increases, the ability to separate images into different viewing directions, or into “good” and “bad” classes improves. With too few classes, “junk” particles may be grouped into a “good” class because there is not enough room for a class that is entirely junk. However, with too many classes the computation can become slow, and there may not be enough signal in the images within each class to successfully sort particles. For the typical dataset with particles numbering in the hundreds of thousands, 50—200 classes is a good starting place. However, this parameter has a significant impact on the results and effectiveness of 2D Classification; as such, some experimentation may be necessary (and is recommended) in order to find the best number of classes for a given dataset. ### Maximum resolution (A) This parameter sets the highest (i.e. finest) spatial frequency used throughout the job, for both alignment and averaging. In most cases, the default setting does not need to be changed. Limiting the resolution available to 2D Classification can reduce overfitting in cases where it is present, but if resolution is too low the algorithm will not be able to align or classify the particles. In general, we recommend using a lower resolution when spiky overfitting artifacts are observed. In general, we do not recommend using a higher resolution for this parameter than the default setting, as higher resolution details should be resolved in 3D rather than 2D. ### Maximum alignment res (A) In some cases, it can be beneficial to use high resolution detail when averaging particle images, but limit the algorithm to use a lower resolution while \*aligning\* particles. This parameter allows setting the maximum resolution for alignment. Setting the maximum alignment resolution using \`Maximum alignment res (A)\` provides the same effect as setting the \`Maximum resolution (A)\` parameter, namely, reducing overfitting by removing high-frequency information. This parameter must be a the same or lower resolution as \`Maximum resolution (A)\`. However, once particles have been aligned at the lower \`Maximum alignment res (A)\` resolution, they are averaged together at the \`Maximum resolution (A)\`. The class averages will therefore be of a higher resolution, but without the risk of overfitting due to noise. ### Minimum alignment res (A) This parameter sets the \*lowest\* frequency which will be included in the alignment step — essentially, the particle images are high-pass filtered to this frequency during alignment. This setting is most useful when there are large, uninteresting parts of the images which degrade alignment (most commonly micelles or other neighboring contaminants). In such cases, setting the \`Minimum alignment res\` between 40—60 Å typically gives the best results. ### Plotting sort method {% hint style="info" %} New in CryoSPARC v5, replacing \`Sort classes by number of particles\`. {% endhint %} ![](https://guide.cryosparc.com/files/jLXmmKPQYFJnRgJ9XeZB) CryoSPARC can plot 2D Classes in one of three ways: \* \*\*similarity\*\* (default starting in v5): Classes are plotted such that a given class looks similar to its neighbors. This makes it easier to detect slight differences between similar-looking classes. \* \*\*size\*\*: In each iteration, the classes are plotted in decreasing size order. The class average in the top left has the most particles; the class average in the bottom right has the fewest. This makes it easy to assess the overall cleanliness of a dataset. \* This mode is equivalent to turning \`Sort classes by number of particles\` on in versions before v5, which was the default. \* \*\*class index:\*\* Classes are always plotted in the same order. The class in the top-left is always class 0; the class in the bottom-right is always class K - 1, where K is the number of classes requested. This plotting order may be useful if you want to use scripting tools after 2D Classification (since it is easy to tell which class average belongs to which class index), or if comparing classes between iterations is important (since classes will stay in the same place in the class average plot). \* This mode is equivalent to turning \`Sort classes by number of particles\` off in versions before v5. Note that because class averages are rotated and shifted in \`similarity\` mode, results may differ slightly between this mode and the other two even with the same random seed. ### Initial classification uncertainty factor 2D Classification starts with randomly generated initial guesses for the 2D class averages. Early in the classification process, these class averages begin to improve, but are still quite far from correct. Therefore, it is important that the 2D classification algorithm account for uncertainty in the class averages; the \`Initial classification uncertainty factor\` parameter controls this effect. This parameter controls how long 2D Classification should remain uncertain about the particles’ class assignments. Increasing the Initial classification uncertainty factor will make the algorithm treat class assignments as uncertain for a larger number of iterations. It can be helpful to increase this parameter when good and bad particles are expected to look very similar. This may avoid the classification getting stuck in a local minimum in which junk particles have been confidently grouped into a good class or vice-versa. See \[Common Problems: Too few or too many junk classes\](#too-few-or-too-many-junk-classes) for more information. ### Circular mask diameter (A) Because the \[CTF delocalizes signal\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/ctf-estimation#the-effects-of-the-ctf-illustrated), it is important to extract particles with a box size that is much larger than the particles themselves. When grids are crowded, a large box size can result in neighboring particles also being present in the image (i.e., crowding). In this case, alignments suffer because the contaminating neighbor is also considered in the alignment. At its worst, this effect can result in several 2D class averages with two or more particles in them, rather than one particle at the center of each average. To avoid the problem of contamination from neighboring particles, the class averages (\*not\* the particle images) are masked with a circular mask at each iteration of 2D classification. Generally, we recommend that this parameter is set slightly larger than the particle diameter. It is also best to keep \`Re-center 2D classes\` on when using a circular mask, so that the particle is centered in the mask as its alignment improves. ![](https://guide.cryosparc.com/files/rIGMLDzELgE69uVhylez) The inner and outer diameter of this mask’s soft edge are controlled by the \`Circular mask diameter (A)\` and \`Circular mask diameter outer (A)\`, respectively. ### Align filament classes vertically ![](https://guide.cryosparc.com/files/Va0lG0TpVaggYGd8S4HB) Class averages of filaments can be aligned such that the filament is vertical in each 2D class. This allows for estimates of an approximate in-plane rotation for each filament image. Note that this is only approximate, and does not attempt to determine the polarity of the class averages. This parameter is turned on by default for helical targets. It should be left off for non-helical targets, since there is no meaningful sense of “vertical” for non-helical objects. ### Remove duplicate particles ![](https://guide.cryosparc.com/files/IDMDzWt4OJjzLAVdpiID) When particles are picked, the same physical particle may be picked and extracted two or more times at different positions on the particle, resulting in duplicate particles in the dataset. Over the course of 2D Classification, the duplicated particle images will be aligned to the same 2D class, and their center positions will be updated. Ideally, this will result in the center positions of the duplicate particle images moving towards the same point on the physical particle. In general, this is a desirable effect. Particle images should become more centered as 2D Classification progresses. However, the presence of duplicate particles causes issues in downstream processing, and so they must be removed from the particle stack before progressing. This parameter detects and removes duplicate particles at the end of 2D Classification if their picked positions plus the translations of the center position modeled by the job are within the radius specified by \`Minimum separation distance (A)\`. Note that the input particles must have locations for this function of 2D Classification to work — particles imported as a particle stack with no attached micrographs (and therefore no location information) cannot have duplicates removed. ### Force max over poses/shifts By default, 2D Classification creates class averages by averaging together all the particles in a class, each at their best pose. In other words, each particle is forced to use its maximum probability pose during averaging, hence the name "Force max". This parameter is on by default, and turning it off instead causes the job to marginalize over pose — essentially “blurring” each particle image out over all likely poses. For a more thorough explanation of this process, please see the Expectation Maximization page. Turning this parameter off can, in some cases, improve 2D classification results with small or low-contrast particles. However, it can often take more iterations for the class averages to converge. We therefore recommend increasing the \`Number of online-EM iterations\` when this option is turned off. Increasing the \`Batchsize per class\` may also improve results when this option is turned off. {% hint style="info" %} If \`Number of 2D Classes\` is set to 20 or lower, this parameter is automatically turned off. {% endhint %} Turning this parameter off increases the amount of time needed to perform a 2D Classification. For example, classifying 50,000 particles from EMPIAR-10261 into 50 classes took 11 minutes with \`Force max\` turned on, and 61 minutes with \`Force max\` turned off. ### Online-EM iterations The parameters discussed in this section are \`Number of online-EM iterations\`, \`Number of final full iterations\`, and \`Batchsize per class\`. In 2D Classification, the class averages are initialized randomly. These random class averages contain no signal and are far from any true projection of the 3D target. Thus, even providing a small amount of good information will likely result in a significant improvement of the average. At early iterations of classification, it would be inefficient to compare the poor quality random class averages to every single particle. CryoSPARC therefore performs several iterations in which only a subset of the particles are aligned to the class averages. These iterations are called \*online-EM iterations\*, and \`Number of online-EM iterations\` controls the number of online-EM iterations performed by the job. Typically 20 is a good starting place, but if the particles have low signal-to-noise ratios, or if \`Force max over poses/shifts\` is turned off, more iterations will likely be required, for example 40. \`Batchsize per class\` controls the number of particles used in these online-EM iterations. By choosing a number of particles \*per class\* instead of the total number, the amount of information provided to each class is held constant regardless of the number of classes. Again, we find the default of 100 is generally sufficient, but \* if the particles have a low signal-to-noise ratio, \* if \`Force max over poses/shifts\` is turned off, or \* if there is a rare particle type present in the sample it may be beneficial to increase this parameter. A good starting value may be between 200 and 500. Once these online-EM iterations have completed, the class averages are typically very high quality. They therefore need significantly more information to improve, so 2D Classification performs a number of final iterations in which all particles are used. These are the \`Number of final full iterations\`, and one is typically sufficient. ### Enforce non-negativity By default, the 2D class average formation model is allowed to make some pixels in the model negative. Turning this parameter on forces each pixel in the class average to be greater than or equal to zero. This constraint on the 2D class averages can substantially change how the classes look, and can improve results in some cases, but can also create streaking or textural artefacts in some cases. Given this, we generally do not recommend turning this parameter on. ![](https://guide.cryosparc.com/files/jTZ0PjPGoh4q5qaIBa9w) \### Use clamp-solvent to solve 2D classes This parameter is similar to \`Enforce non-negativity\` in that it imposes a contstraint on the 2D class averages. Specifically, when this parameter is turned on, the corners of the class average must be zero, and the class average image must also be smooth. This constraint generally results in classes with a "flattened" background (i.e. solvent) region, and can improve results in some cases, but can also create streaking or textural artefacts in some cases. Given this, we generally do not recommend turning this parameter on. ### Do CTF correction For a typical cryo-EM workflow, particle images are extracted from a micrograph and have not been corrected for the Contrast Transfer Function. If particle images have been premultiplied or do not need CTF correction (e.g., negative stain data), this parameter should be turned off. This parameter turns off CTF correction by setting the following CTF parameters in the particles’ metadata: \* Amplitude contrast is set to \`1.0\` \* Defocus is set to \`0.0\` \* Spherical aberration is set to \`0.0\` These values are set for the particles and retained in the output, so subsequent jobs using the output particles will also not correct for the CTF. If CTF correction needs to be re-activated at a later step, the CTF parameters from a prior job can be connected using the low-level interface, or the particles can be re-extracted from micrographs with \`Force re-extract CTFs from micrographs\` turned on. ### Hard classify for last iteration ![](https://guide.cryosparc.com/files/ZvBL8bB6wZ2gp4gamc1s) During alignment, particles are assigned fractional weights to some number of classes. Then, during backprojection (the step in which the class averages are updated), the particles contribute to each class according to their weight in that class. This allows the updated class averages to reflect uncertainty in the class assignments of an individual particle. In some cases, it can be beneficial to force the final class reconstructions to use only each particle’s “best” class (i.e., the class for which that particle has the highest weight). For instance, if a large number of good particles have a small weight in a junk class, they may make that class appear better than it truly is. Turning this parameter on forces the particles to contribute all of their weight to their single best class. Note that this applies only to the \*final iteration\* which is the iteration in which the output 2D class average images and classifications are produced. ### Do orientation alignment {% hint style="info" %} New in CryoSPARC v5.0 {% endhint %} When \`Do orientation alignment\` is on (default), 2D Classification iterations proceed as follows: 1. For every particle-class pair, optimal alignments are found by searching over all poses and shifts 2. Each aligned particle is given weighted assignments to each class, with more weight given to classes they match better 3. New class averages are generated from the new poses and classifications 4. Repeat However, if particles have existing 3D pose estimates, step 1 can be skipped by turning this parameter off. The particles' 2D poses (i.e., in-plane rotations and translations) used during classification are calculated from their existing 3D poses. This separates the alignment and classification steps, so may improve classification of particles which are hard to align in 2D. Additionally, using shifts from 3D methods may reduce the tendency of class averages to drift off center in crowded grids. It is important to note that noise or junk images may in some cases be \*harder\* to detect using 3D poses because they have been optimally aligned to the reference during the 3D refinement. ![](https://guide.cryosparc.com/files/V0b7OuBKZgBY3JpTTrDs) When \`Do orientation alignment\` is turned off, the \`Re-center 2D classes\` parameter is automatically turned off to avoid class averages drifting way from the center aligned during 3D alignment. Additionally, \[\`Plotting sort method\`\](#plotting-sort-method) is set to \`size\` instead of \`similarity\`. Both of these settings can be changed afterward if desired. ## Outputs ### Particles Particles output from a 2D Classification job have a class membership and a 2D pose (translation and in-plane rotation). This allows for removal of, say, all particles in a particular class using a Select 2D Classes job. ### 2D class averages The class averages output is simply the collection of images for each class — it does not contain any information about the particles themselves. This output is used to help subsequent jobs find the images of the class average. ### Rejected particles If \`Remove duplicate particles\` is turned on, any particles rejected as duplicates are output in this group. ### Plots #### Class Averages At each iteration of the 2D Classification the current 2D class averages are displayed to help assess convergence. In addition, several diagnostic plots are produced. ![](https://guide.cryosparc.com/files/rnCJ46k30RglGY3R9DGr) Data from EMPIAR 10261 (Xu et al. 2019) At each iteration, the current class averages are plotted. The following information is overlaid on the class averages as well: \* The number of particles in each class (top of each class) \* A scale bar to help assess particle size (left side of left-most class, every other row) \* The resolution at which the half-sets’ Fourier Ring Correlation (FRC, 2D equivalent to the FSC) correlate at 0.5 (bottom-left of each class) \* The median \[effective sample size\](#effective-number-of-assigned-classes) for particles in each class (bottom right of each class) #### Noise Model ![](https://guide.cryosparc.com/files/c50wYGoE0w3GqFAHTjok) This plot displays the noise model for the current iteration. Essentially, the noise model measures how reliable the images and class averages are at each resolution shell. \*\*Current sigma\*\* (in blue) is the noise model used in the current iteration while aligning particles. It may be different from the \*\*current noise\*\* (in orange), which is calculated directly from the images and averages, due to annealing parameters. Finally, \*\*initial sigma\*\* (in green) is the noise calculated from 100 random images at the start of 2D Classification. More information about the noise model plot is available in the explanation of \[Common CryoSPARC Plots\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-common-cryosparc-plots#noise-model). The Expectation Maximization page discusses the impact of the noise model on classification and alignment. #### Effective number of assigned classes ![](https://guide.cryosparc.com/files/OBMSMkhyTiKwgWhsvemY) At each iteration, particles are assigned to classes based on the error between that class’s average and the particle image. CryoSPARC \*marginalizes over class\*, meaning that each particle contributes to any number of classes, with more of that particle’s weight going to classes to which it is more similar. This histogram measures the total number of classes the particle is contributing to. If a particle is confidently assigned to only one class, it will have an effective sample size (ESS) of 1.0. If a particle splits its density between three classes in a ratio of 50% / 25% / 25%, its effective sample size will be slightly less than 2.0. A particle which is equally likely to belong to all classes will have an effective sample size equal to the number of classes. Formally, the effective sample size of some image $$I\\\_i$$ is the reciprocal of the sum of that particle’s squared probabilities for each class. $$ \\mathrm{ESS}(I\\\_i) = 1/(\\sum\\\_k p\\\_k^2) $$ where $$p\\\_k$$ is the probability that $$I\\\_i$$ belongs to class $$k$$. Early in the 2D Classification job, class assignments are not confident because the quality of the class averages is very poor. Therefore, most particles have a relatively high effective sample size: ![](https://guide.cryosparc.com/files/ncb7VXp1V1f5nGeSjj35) As classification continues, the 2D class averages improve, which in turn makes class assignments more confident. Particles therefore have a lower effective sample size as classification converges: ![](https://guide.cryosparc.com/files/nVPoFjUYAsOj8eXoSWxo) The quartiles of the effective sample size are also output at the end of each iteration in the Event Log: \`\`\` Effective number of classes per image: min 1.00 | 25-pct 1.00 | median 1.01 | 75-pct 1.23 | max 7.22 \`\`\` Several problems can cause effective sample size to remain high through the end of the 2D Classification job: 1. \*\*Overlapping classes\*\*: if classes look similar, the job assigns particles to both of them. If you see classes which look similar by eye and effective sample size remains high, this is a likely cause. The high effective sample size is most likely not a concern in this case. 2. \*\*Incomplete classification\*\*: if effective sample size remains high and 2D classes still look noisy or blurry, the classification may not have converged yet. 2D Classification should be repeated with an increased number of O-EM iterations to give the job more time to converge. 3. \*\*Poor data quality\*\*: if the input particle stack is overwhelmingly junk picks or has low signal-to-noise ratio, class assignments may never become confident. In this case, it may be beneficial to \* use an \[Inspect Picks\](/processing-data/all-job-types-in-cryosparc/particle-picking/interactive-job-inspect-particle-picks.md) job to reduce the number of obvious empty ice or contaminant particles, \* try the advice in \[Common Problems\](#common-problems), or \* try proceeding directly to \[3D techniques\](#recommended-alternatives). #### Probability of best class ![](https://guide.cryosparc.com/files/3IKUm3SAad5402mtuwNW) These plots provide another way of assessing the confidence of class assignments. Rather than plot the \*number\* of classes each particle is assigned to, these plots show the the probability that each particle has for its \*most likely\* class. For instance, if a particle is assigned to only one class, the probability of its best class would be 1.0. If a particle splits its density between three classes in a ratio of 50%/25%/25%, the probability of its best class will be 0.5. Informally, you could think of this value as how confident the 2D Classification job is that, if you forced it to pick a single class for each particle, it would get the class assignment right. Early in the classification, the class averages are too noisy for confident assignment, so the probability of the best class will be low for most particles: ![](https://guide.cryosparc.com/files/e8uLM1TbKyfIMgobiAlu) As classification proceeds and the class averages improve, particles will be assigned with more confidence and the histogram will move to the right: ![](https://guide.cryosparc.com/files/BNtz78RY9rxebEMZhq7D) The quartiles of the probability of best class are also output at the end of each iteration in the Event Log: \`\`\` Probability of best class per image: min 0.12 | 25-pct 0.90 | median 1.00 | 75-pct 1.00 | max 1.00 \`\`\` If the probability of the best class remains clustered at low probabilities, the potential causes and solutions are similar to those of a constant high effective sample size. That section of this page provides potential troubleshooting steps. ## Common Next Steps A 2D Classification job is almost always followed by using Select 2D Classes to remove particles belonging to classes which appear to be junk or contaminant. Alternately, if a 3D volume for the target already exists, \[Reference Based Auto-Select 2D\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/job-reference-based-auto-select-2d-beta) can be used to pick 2D classes based on their similarity to projections of the reference volume. Once 2D Classification produces classes which are free of obvious contaminants and noise users typically move on to 3D methods, starting with \[\*Ab Initio\* Reconstruction\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-reconstruction/job-ab-initio-reconstruction). Note that the 2D pose estimates are not used in subsequent 3D jobs, so it is not strictly necessary to perform 2D Classification before 3D techniques. ## Common Problems ### Streaky or noisy classes ![](https://guide.cryosparc.com/files/GNPt6DcNGv8sefxkRymF) By far the most common pathology in 2D Classification is streaky or noisy classes which have no clear protein features, like those above. In general, if your 2D Classification results look like those above, changing the following parameters may help: \* Turn \`Force max over poses/shifts\` \*\*off\*\* \* Increase the \`Number of online-EM iterations\`. Increasing the default of 20 up to \*\*40\*\* is a good starting place, but as many as \*\*80\*\* may ultimately be required. Experimentation is usually required to find the right value for a given dataset. This setting helps because when parti \* \*\*Increase\*\* the \`Batchsize per class\`, especially if the particle has low signal to noise ratio. A good starting value is perhaps 200, but a higher value may be required. Changing this setting gives the algorithm more information each time it updates the 2D class averages. This is especially helpful in combination with turning off \`Force max over poses/shifts\`, because it counteracts the “blurring” effect of pose marginalization by increasing the total number of particles each iteration sees. Unfortunately, all of the above parameter changes make the job significantly slower. They are thus not set this way by default. ### Too few or too many junk classes In cryo-EM processing generally, there is no way to algorithmically identify a class average as a catch-all “junk class”. Instead, one hopes that requesting a greater number of classes creates additional classes into which junk particles can segregate. This only works if the junk particles look more like each other than they do the good particles — at the end of the day, particles are always assigned to the class they match the best. This creates a challenge for any cryo-EM processing algorithm. We must be confident enough in our class averages to properly assign particles among them, but must also allow them to change enough that they properly reflect the true data. The classification confidence is tuned by the \`Initial classification uncertainty factor\` (ICUF) parameter. When the ICUF is high, 2D Classification treats the class averages as unreliable during the early iterations. This means that even if a particle aligns well to a given class average, it contributes to other classes as well because all alignments are treated as lower quality. This forces all classes to start to look similar in the early iterations. Conversely, a low ICUF means the job will treat all alignments as higher quality right from the first iteration, meaning that classes which begin different will remain different. This often results in a final set of class averages which look more different from each other, but may have fewer unique views of the same object. Changing the ICUF has two main effects. First, with a lower ICUF, class averages stabilize more quickly, since particles are “blurred” across classes to a lesser degree. Note that this effect is highly dataset-dependent. If the particles are small or have low SNR, their class averages will be treated as high quality when in fact they are very poor, which can prevent convergence entirely. ![](https://guide.cryosparc.com/files/mFUtURFJ4e5tYn4gTiKp) The evolution of a single class in each of four 2D Classification jobs. Each job had the indicated ICUF parameter setting. Second, if the ICUF is increased, all classifications are treated as more uncertain. If a particle has a clear best class assignment but ICUF is high, it will still contribute some information to classes for which it has a poor score, since the algorithm is forced to be uncertain about the good alignment. This, in turn, makes those low-quality class averages better, so in future iterations the particle will actually align better to those classes and the cycle will continue. The end result is that 2D Classification jobs with a higher ICUF tend to have more classes which look like each other, and more classes that look like the "average" particle, rather than rare objects or junk. ![](https://guide.cryosparc.com/files/KY872awNWUr01x1jxxss) Note that this is not necessarily an unalloyed good. If 2D class averages will be used for Template Picking, having more views is good; however, if the class averages will be used to filter bad particles, having more good views from the same particles may mean that junk particles have been distributed among the good classes, which makes them impossible to remove by selecting or excluding classes. ## Recommended Alternatives 2D Classification is fast and scales to very large particle sets well, which makes it an appealing job for particle curation. However, it can be difficult to visually identify good class averages, especially for an unfamiliar target. Moreover, good particles may end up classified into junk classes and vice-versa. This makes particle curation with 2D Classification somewhat risky in the sense that it is possible to inadvertently exclude good particles from downstream processing. Therefore, in general, we recommend that only the most obvious junk classes are removed with 2D Classification, and additional curation to retain the best particles be done in 3D. For example, consider this 2D Classification result using images of a GPCR from EMPIAR 11350 (Akasaka et al. 2022). First, a 2D Classification job is run requesting 200 classes and with \`Force Max over poses/shifts\` \*\*on\*\* (default). This job completes in twenty-five minutes on one GPU and produces four classes which have clear GPCR features and 196 classes which do not. ![](https://guide.cryosparc.com/files/d6YyfJetidzdFYJDeuUB) The selected classes have only 28,953 particles. After \[Ab Initio\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-reconstruction/job-ab-initio-reconstruction)\[ Reconstruction\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-reconstruction/job-ab-initio-reconstruction) and \[Non Uniform Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-non-uniform-refinement-new), the resulting map refines to 6.8 Å and shows evidence of overfitting, orientation bias, and other pathologies: ![](https://guide.cryosparc.com/files/W3QFhPMPwysrtQpHV1jT) Turning off \`Force max\` improves the quality of 2D classes, with thirteen good classes and 187 bad classes: ![](https://guide.cryosparc.com/files/fUu3tMfgtpo2xJnfIim3) However, this 2D classification job took 1,571 minutes (26 hours and 11 minutes) to complete on the same single GPU as the previous job — a sixty-fold slowdown. The resulting map is significantly improved both in terms of resolution (4.25 Å) and visible map quality: ![](https://guide.cryosparc.com/files/7G5Zc6cP5ptoe9gwg0QE) This is a reasonable result. However, rather than using 2D Classification only, we could instead filter good particles from bad using a Heterogeneous Refinement job. If we provide a \[Heterogenous Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-heterogeneous-refinement) job with \* \*all\* particles (i.e., the extracted particles before any 2D classification was performed) \* one good class (e.g., the first \*Ab Initio\* map, or perhaps a similar target solved previously) \* several bad maps (e.g., maps from iteration 0 of an \*Ab Initio\* Reconstruction, or some other noise volume) then good particles tend to sort into the first map, while the others collect junk: ![](https://guide.cryosparc.com/files/zc0KkQZ4OA2XRUhfvD90) The resulting good GPCR map contains 141,216 particles — approximately three times as many as the 2D classification job with Force max off. Additionally, this Heterogeneous Refinement job finishes in ninety-seven minutes on only one GPU — approximately four times slower than the 2D Classification job with Force Max turned off, but producing a significantly improved final result. Performing Non-Uniform Refinement on the good class from the Heterogeneous Refinement job produces the best map by far, reaching 3.4 Å and displaying clearly improved map quality: ![](https://guide.cryosparc.com/files/ibY6dnkSb3qG8bstIXVr) Furthermore, the results of \[Orientation Diagnostics\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-orientation-diagnostics) jobs run on each of the refined maps above reveals one potential reason for this significant improvement: the particles which were selected by 2D classification are missing an entire set of views, yielding a cFAR score of 0.09. cFAR scores below 0.5 generally indicate map anisotropy (i.e., that orientation bias is harming the map quality). ![](https://guide.cryosparc.com/files/0JT7WAc5YslsuUh0dUaL) In contrast, the particles selected via Heterogeneous Refinement have a qualitatively more even distribution of particles (note that the color scale differs between the two plots) and a significantly-improved cFAR score of 0.43. ![](https://guide.cryosparc.com/files/riRoKsqwYPBABePiB2ZQ) It may be, then, that 2D classification struggles to separate a particular view of this GPCR from noise or empty micelles. This results in those particles being excluded from downstream analysis despite the fact that they are actually of high quality. As a final sanity check, a 2D Classification job (with Force max off) on the particles selected by Heterogeneous Refinement shows 56,750 particles in 21 class averages with clear GPCR density, indicating that these particles are indeed good picks which were hidden by the preponderance of junk particles in the initial 2D Classification jobs. ![](https://guide.cryosparc.com/files/U5CEL7ckfn9CFRd5pp3y) The remaining classes are likely a mix of junk particles and good particles which cannot easily be distinguished from junk in 2D methods. In conclusion, for smaller targets and targets for which not all views have clear features, it may be beneficial to skip 2D Classification altogether and perform particle curation using only 3D methods, like \*Ab Initio\* Reconstruction and Heterogeneous Refinement. ## What does Force max over poses/shifts do? {% hint style="info" %} This section assumes some familiarity with the Expectation-Maximization algorithm. If you are not familiar with terms like maximization, marginalization, and pose, please see the Expectation-Maximization page. {% endhint %} ![](https://guide.cryosparc.com/files/XfMJTLquKapr5Y8VL3Jl) Force max over poses/shifts switches from the default mode of maximization over pose (top) to marginalization over pose (bottom). In maximization, only the most likely pose is used to contribute to the class average. The others are discarded entirely. When we marginalize over pose, each pose is weighted by its probability. Then those weighted images are combined, and that combination is added to the class average. {% hint style="info" %} Note that in reality, one pose is almost always much more likely than others, and only poses nearby to it have significant probability. Thus, marginalization typically combines poses separated by a few degrees at most around that likely pose. The large (120°) difference shown here is for illustration purposes only. {% endhint %} \`Force max\` is on by default for computational efficiency — it is typically best to weight particle poses by their probability to account for uncertainty. However, in most cases, the probability peak is very sharp and defined, so there is little difference between the two settings. This is why turning \`Force max\` off mostly improves alignment of small, low-SNR particles. In these cases, the probability is distributed more broadly over the poses, so the averages produced by marginalization are significantly different from those produced by maximization. This is also why turning \`Force max\` off slows 2D Classification jobs; each backprojection takes time, and when marginalizing over pose the job must backproject each particle in a number of poses instead of just one. ## References 1. Kumar, K. \*et al.\* Structure of a Signaling Cannabinoid Receptor 1-G Protein Complex. \*Cell\* \*\*176\*\*, 448-458.e12 (2019). 2. Xu, H. \*et al.\* Structural Basis of Nav1.7 Inhibition by a Gating-Modifier Spider Toxin. \*Cell\* \*\*176\*\*, 702-715.e14 (2019). 3. Akasaka, H. \*et al.\* Structure of the active Gi-coupled human lysophosphatidic acid receptor 1 complexed with a potent agonist. \*Nature Communications\* \*\*13\*\*, 5417 (2022). --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/particle-curation/job-2d-classification.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-processing-of-a-novel-motor-bound-nucleosome-state-empiar-10739-part-2.md). # Case Study: processing of a novel motor-bound nucleosome state (EMPIAR-10739) - part 2 Part 2 of our case study on EMPIAR-10739 (Sections 12-16 shown in the flowchart below) explore the processing of a newly discovered minor population state. If you are following along with the processing steps, you will need to first complete \[Sections 1-8\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-end-to-end-and-exploratory-processing-of-a-motor-bound-nucleosome-empiar-10739) from the main case study. Many of the following processing steps were repeated multiple times, and with minor modifications from the preprint, to ensure reproducibility, and so note that particle numbers and resolutions may vary subtly. Image processing steps in this case study were performed using CryoSPARC v4.7.1. !\[Flowchart showing the CryoSPARC jobs used in each section of this case study.\](/files/LqXEbbbwiav4LNrT4j5D) {% hint style="info" %} We repeated Classifications 1 & 2 four times each to check for reproducibility - we found that Classification 1 consistently produced 1-2 volumes; and Classification 2 consistently produced a single volume; that have ALC1 in the active state that is processed in sections 8-11. These classes contained some variation in particle number, and the total particles carried forward to NU Refine 4A-C were 95.6-115k particles. {% endhint %} ## Section 12: 3D Classification of a new state In 3D Classification 1 we found one class that had strong density for well-ordered ALC1 that is tightly bound to the nucleosome. This was clearly visible at a contour threshold of 2 and which is considered the active state (\[Bacic, Gaullier \*et al\*\](https://elifesciences.org/articles/71420)). At a lower threshold of 1, we found a second interesting-looking class volume formed of 22k particles, that contains a density that appears to have relatively few contact points with the nucleosome, except to the acidic patch of histones H2A and H2B (see Figure 19 A). For the purpose of describing the following processing steps, we will refer to this as a loose binding state of ALC1. {% hint style="info" %} We did not find a similar volume in Classification 2 despite repeated runs. However, considering the biochemistry of the target, we expect this binding conformation to be present on both faces of the nucleosome. We considered that the mask used to classify in Classifications 1 & 2 may not be optimal for finding this new state. {% endhint %} \* Inspect the volumes from Classification 1, and look for a volume that is similar to the ones shown in Figure 19. Note that it is possible to have more than one volume that resembles this state. You only need to run 3D Classifications 1 & 2 once, but we show four examples on Figure 19 for illustrative purpose only, to give a sense of the variation in this volume that you might observe. !\[Figure 19. Mask design for Classification 3. A) Example volumes from 4 replicate runs of classification 1. B) Example difference volumes between the identified loose class and the consensus volume. C) Four examples of Mask 6 created from each replicate shown at a contour threshold of 1 and 0.005 compared to the consensus volume (grey).\](/files/m5YIpnA9jT0SRiHIZw1U) We are interested in classifying the region of additional density that is present in the loose class, but that is not present in the consensus map. We can make a difference map that contains just this density by subtracting one volume from another. This could be achieved using the ChimeraX \*volume subtract\* command, as we did in section 8, but this time, we will instead generate a difference map using CryoSPARC directly. \* Run \[Align 3D Maps\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-align-3d-maps) 1, inputting the loose class map from 3D Classification 1 as the “Reference map” and the Consensus map from 3D Classification 1 as the “Map to align”. The job will then align and subtract the consensus volume from the loose class volume. As well as performing the alignment of maps, the job \[Align 3D Maps\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-align-3d-maps) also outputs a difference map as a lower-level output, and this can be found in the Outputs tab of the job card once the job is complete (see Figure 20). \* Download and inspect the difference map to ascertain a threshold where there are no small blobs of density outside of the main large density. We found a threshold of 0.8-1 to look good. \* Create a Volume Tools job, inputting the Reference Volume from Align 3D Maps 1, and drag over the lower level slot for map\\\_difference. This process exchanges the original volume with the difference map volume. Figure 20 shows how to achieve this using the CryoSPARC job builder. For further information about lower level results please see \[this page on the CryoSPARC guide\](https://guide.cryosparc.com/application-guide-v4.0+/low-level-results-interface). !\[Figure 20. CryoSPARC job building steps to use a difference map.\](/files/9JolOkDD0Bw4tWK3wBzB) \* Use the following setting to generate Mask 6 | Parameter | Setting | Explanation | | --- | --- | --- | | `Type of output volume` | mask | | | `Theshold` | | Threshold where no small blobs appear outside the main density | | `Dilation radius (pix)` | 1 | Extend for a little more coverage | | `Soft padding width (pix)` | 3 | Soft edge to avoid masking artefacts | The input map from Classification 1 is Fourier cropped to a box of 56, which has a pixel size of 6.24 so note that this time we don’t need to lowpass filter the map before we make a mask because it is already Nyquist limited to \\~12.5 Å. We also need to use relatively few pixels for Dilation and Soft padding to cover the region in Å that we want. \* Compare Mask 6 to Mask 2. We already identified particles with ALC1 in the active state from Classifications 1 and 2, so we can exclude those from the set so that downstream from here, we are only considering particles here that have not already been assigned a known class type. \* Run Particle Sets Tool 2, inputting the particles from NU Refine 3 into “Particles (A)” and input all of the class particles that show the tight-bound ALC1 from Classifications 1 & 2 into “Particles (B)”. {% hint style="info" %} For difficult classifications cases, focus masks may need to be iteratively designed as class density improves, until a good classification is observed. We found the mask shape of Mask 6, and the downstream classification result at this stage to be variable in different runs, due inherent random behaviour during classification. {% endhint %} \* Clone 3D Classification 1, exchange the input particles for A\\\_minus\\\_B from Particle Sets Tool 2, and exchange the mask for Mask 6. This is 3D Classification 3. | | | | | --- | --- | --- | | **Parameter** | **Setting** | **Explanation** | | `Number of classes` | 40 | Using more classes can help separate low population classes | | `Filter resolution (A)` | 5 | A resolution that will allow us to see more structural detail | | `Initialization mode` | PCA | We found PCA initial volumes gave us a more reproducible result than using simple mode | | `Class similarity` | 0.1 | When looking for density presence/absence, the classes should not be very similar | | `O-EM batch size` | 300 | Using a smaller batch size means more iterations, and more volume evolution | | `O-EM learning rate` | 0.9 | We want the volumes to evolve fast from the start | | `Number of particles to classify` | 400000 | A subset of the whole stack in order to speed up the job | {% hint style="info" %} We found empirically that 400k particles gave a consistent enough result, and this sped up the job from \\~3hr to \\~1hr and so we included this information, but during a typical exploratory pipeline it may make more sense to use the whole particle set, and evaluate if the results are good enough. {% endhint %} Once the job is complete, inspect the output volumes. You might have one or two volumes that are similar to the loose state that we saw in Classification 1. The resulting map now contains more features than the one from Classification 1, due partly to the higher Filter resolution used in Classification 3. We show example map density in Figure 21B. {% hint style="info" %} Now that we have a better idea what shape the density is for this class, we can design a better mask for classification of all the particles. We tried a few different masks at this stage, and compared the quality of downstream refinements. The final results were fairly similar and so the precise masking at this stage may not be crucial as long as the overall region covered is similar to that shown for Mask 7 in Figure 21. {% endhint %} \* Run Align 3D Maps 2, inputting a loose state from Classification 3 as the ‘Reference map” and the Consensus map from Classification 1 as the “map to align”. \* Download and inspect the difference map to ascertain a threshold to set where there are no small blobs of density outside of the main large density. We found a threshold of 0.15-0.22 to look good. \* Create a Volume Tools job, inputting the Reference Volume from Align 3D Maps 2, and drag over the lower level slot for map\\\_difference, as shown in Figure 20. | Parameter | Setting | Explanation | | --- | --- | --- | | `Type of output volume` | mask | | | `Lowpass Filter (A)` | 15 | Filter to remove high resolution features | | `Theshold` | | Threshold where no small blobs appear outside the main density | | `Dilation radius (pix)` | 2 | Extend for a little more coverage | | `Soft padding width (pix)` | 5 | Soft edge to avoid masking artefacts | This time around, the input map from Classification 3 is Fourier cropped to a box of 144, which has a pixel size of 2.43 so we do want to lowpass filter to remove higher resolution features, and we need to use more pixels for Dilation and Soft padding that we did for Mask 6. This mask will be Mask 7. {% hint style="info" %} Recall that this nucleosome is pseudosymmetric, and so it could have this type of interaction on both faces. We can rotate the mask to allow us to perform classification on the opposite side. We do this rather than symmetry expanding and classifying on one side for two reasons 1) we might want the option to check that the density looks the same on both sides 2) the classifications on each side can be performed in parallel, but the symmetry expanded classification is a single job that takes twice as long, so as long as there are 2 GPUs available, it is more time-efficient to perform one classification on each side. {% endhint %} \* Run a \[Volume Alignment Tools\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/job-volume-alignment-tools) job inputting the volume from NU Refinement 3 and Mask 7. \`Set 3D rotation euler angles (deg or rad)\` 0,0,180, and the output mask is Mask 8. \* Compare Masks 2, 6 and 7 (examples shown in Figure 21A) to see how the shape iteratively changed. !\[Figure 21. Comparison of masks and output volumes from classifications 1, 3 and 4. Mask are shown at a contour threshold of 0.5 relative to the map from Refinement 3.\](/files/QgLdcm81t5isoxvV4OxV) \* Clone 3D Classification 3, exchange the focus mask for Mask 7 and remove the value in \`Number of particles to classify\`. This is 3D Classification 4. \* Clone 3D Classification 4, and exchange the focus mask for Mask 8, and this is 3D Classification 5. \* Inspect the maps from Classifications 4 and 5, and identify the loose state volumes that are similar to the volume in Figure 21B (right, blue). In total you might find two or three classes in total that resemble this state. ## Section 13: Non-Uniform Refinement and 3DVA of the loose ALC1 state Now that we have selected the loose state classes (and their particles), we want to look for finer details of the additional density than we can see in the 3D Classes. To do so we will start with Non-Uniform Refinement. We could separately refine the particles from Classification 4, and Classification 5 to retain the DNA asymmetry, but to help us get better density quality for the additional density, we will instead combine the particles and refine them together with C1 symmetry, allowing the asymmetry from the additional density to dominate particle alignment, at a cost of losing the information about the DNA asymmetry. \* Run a Non-Uniform Refinement (NU Refinement 6) with \`Minimise over per-particle scale\`:true and \`Dynamic mask start resolution (A)\`: 1, input a loose-class volume from 3D Classification 4, and the loose class particles from 3D Classifications 4 & 5. Inspect the output volume from Refinement 6 and compare to the example density shown in Figure 22. By using \[Model Angelo\](https://github.com/3dem/model-angelo), we found the density was sufficiently well-ordered to allow identification of the additional density contacting the acidic patch as a part of ALC1. !\[Figure 22. Map and features from the Refinement 6 map. Left; map at a high contour threshold coloured by proximity to the nucleosome (grey), acidic patch residues (teal) and ALC1 (pink). Right; map at a lower contour threshold shown relative to a gaussian filtered version of the Refinement 3 map in semi-transparent white.\](/files/so8kDvIw1S5g6FiJdplr) {% hint style="info" %} Whenever you observe new, unknown density during data processing, it can be helpful to compare this density to existing PDB models or maps of the target in different states that do not show this density. For example, to aid assessment here, you can compare the maps from Refinements 6 and 3, and/or fit in a nucleosome PDB such as \[7otq\](https://www.rcsb.org/structure/7OTQ). As we are writing this case study after depositing a model, we had the option to fit in PDB \[9tv4\](https://www.rcsb.org/structure/9T4V) to the map from Refinement 6, and colour it according to proximity to the nucleosome (grey), acidic patch residues (teal) and additional density (purple) in Figure 22. We found that the density was pretty good for the core nucleosome, and found a region of well-ordered density binding to the acidic surface patch formed by histone H2A and H2B. Although the map quality might vary somewhat between classification runs, in \[Bridges \*et al.\*\](https://www.biorxiv.org/content/10.1101/2025.11.10.687450v1), we were able to identify that this pink density region originated from ALC1 by providing to \[Model Angelo\](https://www.nature.com/articles/s41586-024-07215-4) a sharpened map and sequences of the known components that were in the preparation (Nucleosomal DNA, histones H3, H4, H2A and H2B, HPF1, PARP1 and ALC1). {% endhint %} The Global FSC resolution in our hands was around 2.8 Å, but the region that we are most interested in (pink density in Figure 22) is largely fragmented, featureless and appears only at relatively low contour threshold. This tells us that there must be residual heterogeneity in this particle stack, and so we will now go on to analyse the particles by 3D Variability Analysis. 3DVA is perfect for this scenario because it should be able to give us clues about the type of heterogeneity that is present, and how we might approach the next processing steps. For 3DVA, only variability within the masked region is considered, and so we want to try and ensure that the mask is large enough! We already have Mask 7 that covers the loosely-bound ALC1, but to be on the safe side we will expand it a bit further. \* Clone the Volume Tools job that made Mask 7, change the \`Dilation radius (pix)\` to 5 to produce Mask 9. \* Run a 3D Variability job (3DVA 2), with a \`Filter Resolution (A)\` of 12 and inputting the particles from Refinement 6, and Mask 9. \* Use the quick action to create a 3D Variability Display job, and set \`Downsample to box size\`: 128, so that the resulting maps are a smaller size to download. \* Download the series and open them in ChimeraX We show an example movie showing the three modes that we found in Movie 2. ![](https://guide.cryosparc.com/files/q7Ui4qlXcghLKml3cFuA) **Movie 2. Modes from 3DVA 2.** We found that there was complex heterogeneity present. The most prominent features were a rotation (see Figure 23A), and the presence and absence of a tube of density that runs along a DNA super-groove that is formed by the alignment of two major, or two minor groves in the nucleosome DNA (see Figure 23B). The most distal ALC1 density appears to also have a high degree of heterogeneity in our mode 3. We ran this procedure a few times to check for consistency and found that replicates generated volumes with tubular density in one, or two super-groove locations. Where it appears in two different location, one is a major super-groove, and the other is a minor super-groove (see Figure 23B). You might only see it in one location overall, you might see it in both locations in a single mode, or, as in the example in Movie 2, you might see it in different super-grooves in different modes. !\[Figure 23. Key features of 3VA volumes. A) Top view of two selected volumes, showing the rotation of ALC1 relative the nucleosome, and a diagram approximating of the extent of rotation. B) Side view of two selected volumes showing the tubular density occupying either major (blue) or minor (pink) super-groove, with these regions indicated relative to a surface representation of PDBN 7enn.\](/files/8uu3Ar4VbSSbm9B4vp6T) {% hint style="info" %} When you observe the appearance and disappearance of density in a mode of 3DVA, this usually indicates the presence of discrete heterogeneity, either conformational, or compositional. This is a clue that we might benefit from classifying into discrete states, for example by using 3D Classification, or 3DVA cluster mod {% endhint %} We noted that the appearance of the tubular density appeared to correlate with the rotation extent of the ALC1, and so it seems possible that the tubular density originated from the same part of ALC1, but that it has at least two different interaction sites with the DNA. ## Section 14. Mask design and sub-classification of the loose ALC1 Having established in Section 13 that we likely have discrete heterogeneity present, we want to go ahead and try 3D Classification. As the motion of the ALC1 rotation, and the groove-selection may or may not be coupled, we could either classify with a focus mask over the ALC1 density closest to the nucleosome (we refer to this region as proximal), or we could use a mask over the region of tubular densities. In exploratory processing, you often need to try a variety of approaches in order to find the best route. Here, we will try classifying on both regions, and see which one gives the best classification! The choice of volumes(s) to use for the next step depends on the appearance of your 3DVA volumes. A) If you obtained volumes from 3DVA 2 that resemble both the pink and blue volumes in Figure 22 \* open them up on ChimeraX and sum the volumes with the following command, where X an Y are the model numbers for these volumes: \`\`\`json vol add #X #Y \`\`\` \* Save, upload and import this volume to CryoSPARC. B) If you only obtained a volume that resembles one or other of the two states in Figure 23 \* open the volume from 3DVA 2 with the best tubular density For the tubular density region, in ChimeraX using either the summed volume or the single frame volume produced above, use the map eraser tool to erase most of the density, leaving just the volume for the tubular region show in Figure 24A and apply a gaussian filter. Save this volume and take a note of the contour threshold that looks suitable for mask binarisation. \* Save, upload and import this volume to CryoSPARC. For the proximal density, use the map eraser on the same input volume as above, and erase everything except the proximal ALC1 density, so that it resembles that shown in Figure 24 B, apply a gaussian filter, save the file, and make a note of the threshold \* Save, upload and import this volume to CryoSPARC. We want to generate a solvent mask, and two focus masks - the solvent mask should be large enough to contain both of the focus masks. \* Create a Volume Tools job inputting either the summed volume (A) from above, or the volume from Split Volume Groups (B) that has the best tubular density and use the following settings to generate Mask 10 | Parameter | Setting | Explanation | | --- | --- | --- | | `Type of output volume` | mask | | | `Theshold` | | Threshold where no small blobs appear outside the main density | | `Dilation radius (pix)` | 5 | Extend for a little more coverage | | `Soft padding width (pix)` | 5 | Soft edge to avoid masking artefacts | \* Create one Volume Tools job for each of the two erased volumes that you imported, and use the same settings as for mask 10, but use \`Dilation radius (pix)\` 4 to make the mask slightly tighter to the volume than the solvent mask. This will produce masks 11 and 12. Examples of summed volume and erased volumes, along with their respective Masks 10-12 are shown in Figure 24. !\[Figure 24. Volumes used to generate masks 10-12, and mask density. Erased density and masks are shown relative to the summed volume (in grey), and masks are shown at a contour threshold of 0.99.\](/files/Y5AzcGi9EroBJyfVgJdo) \* Create a new 3D Classification job using the particles from NU Refinement 6, Mask 10 as the solvent mask, and Mask 11 as the focus mask. Use the following settings for Classification 6: | | | | | --- | --- | --- | | **Parameter** | **Setting** | **Explanation** | | `Number of classes` | 3 | One for each of the tube density locations and one spare | | `Filter resolution (A)` | 7 | A resolution that will allow us to see the features | | `Initialization mode` | PCA | We found PCA initial volumes gave us a more reproducible result than using simple mode | | `O-EM batch size` | 100 | Using a smaller batch size means more iterations, and more volume evolution | | `O-EM learning rate` | 1 | We want the volumes to evolve fast from the start | \* Clone Classification 6, and exchange the focus mask for Mask 12. Run this job as Classification 7. When both jobs are complete, examine the volumes. We show example volumes from Classifications 6 and 7 in Figure 25. We ran Classifications 6-7 as 4 replicates, to see how consistent the output volumes were. We found that classification 6 gave less ambiguous classes than Classification 7, and was able to separate out one class with good density for what looks like a long helix interacting with a super-groove (Figure 25, pink) that we will call loose state A, the other two classes tended to have either no density for this helix, or weak density interacting with the adjacent super-groove (Figure 25, blue) that we will call loose state B. This blue state was less consistent in quality between replicates. Classification 7, on the other hand, produced volumes with ambiguous density for the tubular region, despite separating rotation states of the proximal density region. This results seems to indicate that the two regions we classified, are not entirely coupled in their motions. {% hint style="info" %} If your classification does not contain loose state B, and you are interested in processing it further, you can try i) re-running 3D Classification 6 perhaps with a slightly adjusting the focus mask or ii) repeat Classifications 4-6 to find this class due to stochastic events during classification. {% endhint %} !\[Figure 25. Example density from Classifications 6 & 7. Volumes are coloured according to the state that they represent, with green volumes showing ambiguous density.\](/files/XECPMddsFXRDfwk8ne6Z) Every time that we performed replicates of classification 6, we found loose state A (pink), and we sometimes also found loose state B (blue). We have separated some interesting states, but the classification was tricky and not fully reproducible. When situations like this arise it can be a good idea to try 3D Variability Analysis to see if there is residual heterogeneity present. \* Run a Homogeneous Reconstruct Only job inputting the particles from the class that matches our Loose state A (pink) above and examine the map Recall that the particles at this stage were aligned during NU Refinement 6, so the alignment is dominated by the strong signal from the nucleosome. We found that the reconstructed map looked pretty similar to Refinement 6, with the nucleosome density being well-defined, but most of the ALC1 appearing as fragmented density. This indicates residual heterogeneity in the ALC1 part of the map. ## Section 15. 3DVA and Local Refinement of the loose state As we suspect heterogeneity in our Loose state A particles, 3DVA is often a good first port of call! \* Run a 3D Variability Analysis job, inputting the mask from Classification 6, and particles from the class that matches our Loose state A (pink) above, and set the Filter resolution (A) to 12. \* Use quick actions to build a 3DV Display job in simple mode with the following settings: | | | | | --- | --- | --- | | **Parameter** | **Setting** | **Explanation** | | `Downsample to box size` | 128 | Fourier crop the volumes so the file sizes are smaller | | `Crop to size (after downsample)` | 100 | Crop the box so that it is closer to the particle to make the file size smaller | | `Filter resolution (A)` | 12 | Filter the maps for a smoother appearance | \* Examine the output series using ChimeraX ![](https://guide.cryosparc.com/files/sicCFpchYS7TC1rE5ZlJ) **Movie 3. 3DV Display volumes from modes 2,1 and 0.** We show example movies for the three series in Movie 3 where we can see that even when there is helical density in a fixed location, the rest of the ALC1 undergoes additional conformational changes. We see in our series 0 (pink, right) that the proximal density is still rotating around the nucleosome while the helix is in place, therefore from this movie, and the result from Classification 7, we can postulate that the rotation position of the proximal ALC1 density might not be directly coupled to the helix binding in this location. In addition, in our series 1 (middle, blue) we see the proximal density levering up and down, pivoting at the same contact point as series 1. {% hint style="info" %} Where heterogeneity is complex and multidimensional, it can be challenging to decide the route ahead. When there are plenty of particles, it can be beneficial to run hierarchical classifications, or parallel classifications on different regions and take particle intersections. Particle number can become limiting though leading to refined or reconstructed maps with lower resolution and fewer recognisable features. Where continuous heterogeneity is part of the story, sometimes the best compromise is to locally refine the region of interest, in the knowledge that some parts may remain relatively poorly defined. {% endhint %} We ended up with relatively few particles in our loose state A class, typically at \\~ 20,000 in our replicates so we chose not to further split the particle set. Instead we will locally refine the ALC1 region, taking into account the knowledge that there is a rotating motion at the contact point of the proximal density and nucleosome. \* Examine your loose state A volume from Classification 6 in ChimeraX and use the mark surface (found in the Markers menu) then right click on your volume at the point where the ALC1 makes contact with the nucleosome. If you are not satisfied with the location, you can select the marker and use translate selected models (found within the Right Mouse menu) to then optimise the marker placement (see an example in Figure 26). Use the \`measure center\` command to print the new marker coordinates in the Log \`\`\`json measure center #X \`\`\` Where X is the model number for your marker \* make a note of the three numbers that make up the marker position. This will be the pivot point in Å that we will use during local refinement. An example selected location is shown in Figure 26 A. Local refinement requires careful mask design to contain the region of interest, too small and you might see map artefacts and over-fitting, too large and you might be capturing two regions that move independently. \* As well as the loose state A volume, also load up the consensus volume from Classification 1 and use the volume subtract command to generate a difference volume. \`\`\`json volume subtract #Y #Z \`\`\` Where Y is the model number for the loose state A map, and Z is the model number for the consensus map before classification \* Apply a gaussian filter to smooth the map \`\`\`json vop gaussian #A sdev 2 \`\`\` Where A is the model number for the difference map. If the map still doesn’t appear smooth, a more aggressive filter with sdev of 3 or 4 could also be tried. \* Use the Map Eraser tool in ChimeraX to remove extra blobs of density outside of the ALC1 region. \* Find a threshold where the map looks similar to the one shown in Figure 26, save the volume, and upload and import it to your CryoSPARC workspace. {% hint style="info" %} When generating a mask, Volume Tools applies any specified low pass filter before thresholding, but the optimal threshold may change depending on map filtering. One way to navigate this is to first run Volume Tools just setting the low pass filter the inspecting the map in the volume viewer to determine a good binarisation threshold. The job can then be cleared and re-run to generate a mask at an appropriate contour level, extension and soft edge. {% endhint %} {% hint style="info" %} We found that if the mask is extended too far by dilation, that parts of the nucleosome get included in the alignment and resolution estimation, leading to poorer density and a resolution estimation that was too high for the ALC1. {% endhint %} \* Make a Volume tools job, inputting this volume, setting the threshold as determined above and use the following settings to generate Mask 13. | Parameter | Setting | Explanation | | --- | --- | --- | | `Type of output volume` | mask | | | `Lowpass Filter (A)` | 10 | Filter to remove high resolution features | | `Theshold` | | Threshold where no small blobs appear outside the main density | | `Dilation radius (pix)` | 3 | Extend for a little more coverage | | `Soft padding width (pix)` | 4 | Soft edge to avoid masking artefacts | !\[Figure 26. Mask design for Classification 8. The consensus volume from Classification 1 is shown in grey with the pivot point selected for classification indicated as a green sphere. Loose class A state from classification is shown in pink, and Mask 13 is shown in orange at a contour threshold of 0.998.\](/files/3gkAFkN3Kg4HFnRAC6Iz) \* Run Local Refinement with Mask 13, the volume and particles from the loose state A from Classification 6, and the following settings: | Parameter | Setting | Explanation | | --- | --- | --- | | `Use pose/shift gaussian prior during alignment` | true | Using priors penalises deviation too far from the input poses | | `Standard deviation (deg) of prior over rotation` | 3 | A strict range | | `Standard deviation (A) of prior over shift` | 2 | A strict range | | `Override fulcrum coordinates (A or pix)` | your values | | | `Re-center rotations each iteration?` | true | | | `Re-center shifts each iteration?` | true | | | `Maximum align resolution (A)` | 5 | To prevent over-fitting of the map | Examine your map and compare to that shown in Figure 27A. We typically obtained an estimated resolution of around 6-7 Å for the ALC1 region and a cFAR of 0.37. Due to the high degree of heterogeneity as found in Movie 3, the features in your final volume might vary sightly from what we obtained here, or in the related EMDB entry emdb:55533. {% hint style="info" %} We were running Classifications 4-6 in replicates to test for reproducibility, so we had the option of testing if combining the particles from all 4 versions of 3D Classification 6, or conversely, using only the particles that were common to them all (found by using Particle Sets Tool in Intersect mode), would give a better refined volume. Out of our 4 replicates we found a total of \\~45k particles in loose state A classes, and 7.5k of those were common to all 4 replicates. When Locally refined using the same settings and mask as for Local Refinement 2, the combined particles gave us a slightly better map and cFAR (Figure 27A), and the common particles gave us a slightly poorer map quality (Figure 27C). For projects where a rare state is being investigated, or where there is relatively poor signal-to-noise, running replicates of classifications, and combining the particles the belong to the interesting state(s) can sometimes yield a slightly better map quality than using a single 3D classification job. Ultimately, the choice about whether or not to pursue this sort of strategy may depend on time constraints, computational resources, and whether the initial result is sufficient for onward analyses. {% endhint %} !\[Figure 27. FSC curves, conical FSC curves and unsharpened maps from Local Refinement 2. A) Combined particles from 4 replicates of classifications 4-6 (45k particles). B) Particles from a single replicate (19k particles) representing the expected result from following the case study steps. C) Common particles from 4 replicates (7.5k particles).\](/files/5ERfqMSbWFInFVvxqDhW) We found that in all cases, the cFAR at this stage was around 0.3-0.4, indicating a possible orientation bias, but we did not observe obvious map streaking that indicates prohibitive map anisotropy. ## Section 16. \*\*Map sharpening and assessment of map quality\*\* At the end of refinement it is worth assessing if the auto-sharpening has applied an appropriate B-factor. We want the sharpening to enhance higher resolution features, but without causing map fragmentation, excessive noise, or creating sharpening artefacts such as unexpected density extending from the map. As cryo-EM maps tend to contain a range of resolutions, picking a single B-factor to sharpen means taking a compromise value where the high-resolution regions are not as sharp as they could be, and the low-resolution regions are not as connected as they could be. \* Examine the unsharpened and sharpened map from Local Refinement 2, and see if you are happy with the level of sharpening applied. We felt that the map was somewhat over-sharpened due to the appearance of spiky noise extending from the protein that are unlikely to be real features at this resolution. If you wish change the map sharpening: \* OPTIONAL: Run a \[Sharpening Tools\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-sharpening-tools) job, inputting the Local Refinement 2 map and mask, and setting \`B-Factor to apply\` to your desired value, we chose -250. We show example maps for unsharpened, automatically-sharpened and manually sharpened maps in Figure 28A. Assessing the resolution of a local refinement can be tricky, as the masked region can be hard to select and there will inevitably be some density outside of the local refinement mask. In order to proceed we want to ensure that all of the meaningful parts of the map (i.e. the whole nucleosome and ALC1) is inside the mask, to avoid the situation where voxels outside are labelled as having a resolution of 0. Mask 10 should be appropriate for this purpose. \* Run a \[Local Resolution Estimation\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-resolution-estimation) job inputting the map from Local Refinement 2, Mask 10. \* Run a \[Local Filtering\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/post-processing/job-local-filtering) job for the above job \* Visualise the local resolution in ChimeraX by using the Surface Colour function. Our Local Refined map has a local resolution range around 5-15 Å for the ALC1. As expected from the residual heterogeneity we observed in 3DVA, the distal portion that show a lot of movement, is a low resolution, at around 10-15 Å, but the proximal density is closer to 5 Å. !\[Figure 28. Sharpening and Local Resolution. A) The unsharpened, auto-sharpened (B-factor -345) and manually sharpened (B-factor -250) maps from Local Refinement 2. B) locally filtered map from Local Refinement 2 coloured by local resolution.\](/files/jf296n45i8Mwyh5cG9tV) Model building was performed in \[Bridges \*et al\*\](https://www.biorxiv.org/content/10.1101/2025.11.10.687450v1). to the original loose state A map that we found via a similar but not identical route. You can find the locally refined map and the model deposited as \[emdb:55533\](https://www.ebi.ac.uk/emdb/EMD-55533) and \[pdb:9t4v\](https://www.rcsb.org/structure/9T4V). After consideration of the binding interactions between ALC1 and the nucleosome, what we referred to as loose state A during processing was determined to be an intermediate binding state of ALC1 where the C-ATPase and linker regions are relatively well defined, but the N-ATPase and macro domains are low resolution and appear highly dynamic. \* Compare your map to \[emdb:55533\](https://www.ebi.ac.uk/emdb/EMD-55533) and \[pdb:9t4v\](https://www.ebi.ac.uk/emdb/EMD-55533pdb%209tv4). Note that if you fit 9t4v into your map the fit may not be perfect, due to the complex heterogeneity present in the dataset leading to slightly different particle sets each time. \*\*OPTIONAL\*\* If you observed and are interested the loose state B that we found in Section 14, or any other new states then you can repeat the steps for Sections 14-16 to investigate and locally refine ALC1 in those states. ### \*\*Conclusions\*\* In part 2 of the case study, we focussed on the processing of a new state found in EMPIAR-10739. This was investigated by: \* Creating a difference map to aid ALC1 mask generation \* Iterative mask design for 3D classification \* 3DVA with a generous solvent mask \* Local Refinement with a custom fulcrum to improve the ALC1 density The discovery of loose binding states of ALC1 was unexpected! Where time and resources allow, it can be worth investigating unexpected volumes that appear during data processing, or revisit old datasets as data processing software evolves, as there could be something valuable there! ### \*\*References\*\* \[Luka Bacic, Guillaume Gaullier \*et al.\* (2021) \*\*Structure and dynamics of the chromatin remodeler ALC1 bound to a PARylated nucleosome\*\* \*eLife\* \*\*10\*\*:e71420\](https://elifesciences.org/articles/71420) \[Hannah Bridges \*et al.\* (2025) \*\*ALC1 Finds a New Foothold on the Nucleosome’s Super-Groove\*\* \*bioRxiv\* 2025.11.10.687450\](https://www.biorxiv.org/content/10.1101/2025.11.10.687450v1) \[Kiarash Jamali \*et al.\* (2024) \*\*Automated model building and protein identification in cryo-EM maps\*\* \*Nature\* \*\*628\*\*, 450–457\](https://www.nature.com/articles/s41586-024-07215-4) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-processing-of-a-novel-motor-bound-nucleosome-state-empiar-10739-part-2.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp.md). # Case Study: Yeast U4/U6.U5 tri-snRNP Overview In this tutorial we will work step-by-step through an ideal use of Local Refinement. Although we will explain the motivation behind our choice of jobs and parameter settings, the main \[Local Refinement guide page\](/processing-data/all-job-types-in-cryosparc/local-refinement/job-new-local-refinement-beta.md) is an excellent resource for explanations of the theoretical and practical meanings of the parameters. The tri-snRNP complex is a core component of the spliceosome. It comprises four main domains: the body, head, arm, and foot. These domains are arranged in a triskelion-like shape, with the head, foot, and body radiating from the center and the arm extending past the distal end of the body. For this tutorial we will use the clean particle set from EMPIAR-10073. This dataset was originally collected and processed by \[Nguyen et al\](#references). Note that at each step your results may not look exactly like those in the guide due to randomness inherent in the alignment algorithms. As long as your map looks similar overall, and you see similar increases in quality in the focused regions, you are on the right track. ## Setting Up Before beginning this tutorial, you should create a new project and a workspace within that project. Download the particle stack to a location of your choosing. Our data is downloaded to a directory called rawdata in the project directory using the command: \`\`\`sh wget ftp://ftp.ebi.ac.uk/empiar/world\_availability/10073/data/\\\*.mrcs \`\`\` Next, download the \[STAR file\](https://s3.wasabisys.com/cryosparc-test-data-dist/shiny\_correctpaths\_cleanedcorruptstacks.star.gz). This file has starting poses for the particle images, skipping the initial volume generation steps. Finally, import the data using an \[Import Particles\](/processing-data/all-job-types-in-cryosparc/import/job-import-particle-stack.md) job. The \`Particle meta path\` should match the location of the STAR file, and the \`Particle data path\` should be the directory containing the downloaded \`.mrcs\` files. ## Global Refinement To ensure that your particles were loaded correctly, plug the \`Imported particles\` output into the \`Particle stacks\` input of a \[Homogeneous Reconstruction Only\](/processing-data/all-job-types-in-cryosparc/3d-refinement/job-homogeneous-reconstruction-only.md) job. This will use the poses from the STAR file and the imported particle images to build a 3D map, without performing any alignment. ![](https://guide.cryosparc.com/files/veduiNToh0UUho1lvTJJ) A reconstruction of the input particle poses. Domains are highlighted with consistent colors throughout this case study. At a low contour, all four domains are visible. However, at a higher contour the arm and head disappear entirely, and the quality of the foot also degrades. What is a contour? Visualization of 3D maps typically relies on “contouring”, where all the points in the map which equal a value are displayed as a surface. The selection of this value in ChimeraX is achieved by dragging the bar in the Volumes pane to the right (increase) or left (decrease). A specific value can also be entered in the map’s text box. Increasing the contour makes that surface adhere to where electron potential is higher — this typically displays better-aligned regions in higher resolution. Decreasing the contour expands the surface to regions where electron potential is lower — this typically shows blurry, poorly aligned regions. ![](https://guide.cryosparc.com/files/jrpCGCMAamzNt56XPTf7) Increasing the contour causes the head and arm to fade into the background. In cases like this, where different regions of the target have dramatically different resolutions, \[Non-Uniform Refinement\](/processing-data/all-job-types-in-cryosparc/3d-refinement/job-non-uniform-refinement-new.md) often performs exceptionally well compared to traditional Homogeneous Refinements. The poses in the STAR file were generated with homogeneous refinement, so the map may improve simply by performing a Non-Uniform Refinement instead. Plug the Homogeneous Reconstruction Only job’s particles and volume into a Non-Uniform Refinement job as inputs. Leave the mask blank to generate a dynamic mask. Leave all settings as default and launch the job. {% hint style="info" %} Non-Uniform Refinement outperforms Homogeneous Refinement in cases like this because, in each iteration, the map is filtered based on its local quality rather than the global quality. For more information on this algorithm and how it is implemented in CryoSPARC, see the \[Non-Uniform Refinement page\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-non-uniform-refinement-new). {% endhint %} ![](https://guide.cryosparc.com/files/670tKi67ObPdqTNgVPle) Non-uniform refinement significanly improves map quality in the body and foot, but does not recover information from the head or arm. The Non-Uniform Refinement significantly improved the map, both as assessed by GSFSC resolution (4.17 Å → 3.55 Å) and by visual inspection, especially in the foot and body regions. However, the head and arm are still not visible at medium or high contours. To summarize: \* We can successfully align the particle to a consensus reconstruction of with a nominal resolution of 3.5 Å \* We can resolve the body and foot domains at a very low resolution, so we know they are present in the particles \* When we increase the contour, the head and arm disappear, indicating they are poorly aligned The reason the head is blurred is that the particles can only align to either the head or the body or the foot — there is no pose which will perfectly align every domain of the particle. Since the body is the largest domain, that region of the particle is preferentially aligned. Relative to the body, the foot moves the least, the arm the most, and the head somewhere in between. This is why the three domains are blurred, and why we can see more of the foot than the head or arm. Local refinement solves this problem by creating a mask around a sub-volume of choice (for instance, the head). Using this mask eliminates the rest of the volume. When the search volume only contains the head, an image’s assigned pose will only improve when the head is well aligned. In other words, aligning the larger body/foot region will result in a poorer score. ![](https://guide.cryosparc.com/files/2ZO2gQSgjKt5yRL0kpSj) In a global refinement, the entire volume is compared with the full particle image. Local refinement only uses a masked sub-volume. In a global alignment, it’s possible that the head would be too small to align on its own, or the masked head-only volume might incorrectly align to the foot at low resolutions. Local refinement solves this problem by incorporating pre-existing knowledge about these particles. We know the approximate pose of the head in all of our images. We use Local Refinement to fine-tune it, while not allowing the head to move so far that it aligns to the wrong domain or to background noise. Let’s proceed to a local refinement of the head domain. The first step in doing so is creating the mask we will use to select only that sub-volume. ## Mask Generation {% hint style="info" %} Mask generation is a complicated skill that is essential for cryoEM image processing. For more information and guidance about making and using masks, see the \[Mask Creation guide page\](/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera.md). {% endhint %} ### Create the mask base To generate our mask around the head, first load the Non-Uniform Refinement result volume into ChimeraX. To smooth the map and attenuate high-frequency noise in the head, apply a Guassian filter to the map (replace #1 with the number for your input map): \`\`\` volume gaussian #1 sDev 2 \`\`\` What is the difference between a Gaussian filter and a lowpass filter? This command applies a Guassian filter, or blur, to map #1. In essence, every point of the map is spread out into a Guassian peak with a standard deviation of (in this case) 2. A Gaussian blur is like a low-pass filter in that it attenuates high-resolution information while preserving low-resolution information, but its shape is different from the Butterworth filter used in most low-pass applications. As such, there is no way to filter to a specific resolution (e.g., 6 Å) using a Guassian filter. However, to make a mask, we only care that high-resolution features and high-frequency noise are removed, not the exact resolution of the final map. For that reason, a Guassian filter is sufficient here. Why should masks be smooth? Beyond the practical benefit of making it easier to generate them, it is actually important for the reliability of your results that your masks are smooth. If your mask contains high-resolution information (that is, if your mask was not blurred), it may induce spurious correlations between half-sets based solely on the mask rather than the volume themselves. This results in an inflated GSFSC resolution estimate. We therefore recommend that you always filter your mask to a resolution well below the expected GSFSC resolution of your alignment before generating them. ![](https://guide.cryosparc.com/files/gTyAqAYcv3hxNeo8PvGE) A comparison between the unblurred (left) and Gaussian filtered (right) maps. Removing the high-frequency noise makes it much easier to see each domain and aids in building masks that are nice and smooth. Next, we segment this map using \[Segger\](https://www.cgl.ucsf.edu/chimerax/docs/user/tools/segment.html), which segments a volume using watershed segmentation. A GUI for the tool is opened with Tools > Volume Data > Segment Map. Click the “Shortcuts Options” dropdown to display buttons which run convenient commands. Select your Guassian-filtered map in the “Segment map” dropdown and click the “Segment” button, leaving all other settings as default. The results should look something like this: ![](https://lh7-us.googleusercontent.com/cjZmuaz5Jq3NObksVhSNqvzUwAbN8hysDaRUMWxTxO-ILoPmsuVwK_tz0gEwLVou23ZiRj7o4A7NV6QL1h9TG0pXwWcgn-5dkjQlg7pjJAM0vjvGBRbV1U45YS6phhJl-jZF6oJNn5XKjtd2iq3aDHk) The map has been segmented into several regions, each with a distinct color. Segger has split the map into (in this case) 61 regions. You can build your mask by selectively hiding these regions, leaving only the part of the map that is to be included in the mask. \* Control-clicking a region selects it \* Control-shift-clicking a region adds it to the current selection \* Clicking “Hide” hides a region without deleting or un-selecting it \* Clicking “Show” shows a region \* Clicking “Delete” deletes a region \* Clicking “Ungroup” splits a region into smaller subregions \* Clicking “Group” combines two or more regions into one larger region {% hint style="warning" %} There is no undo feature in Segger! We highly recommend that you click “Hide” before deleting a region. If more is hidden than you expect, you can click “Show” and ungroup the region before trying again. If you go straight to “Delete”, you’ll have to start over! {% endhint %} To make a mask around the head, hide all of the regions corresponding to the foot, body, and arm. The final Segger model looks like this, with the map in displayed grey: ![](https://guide.cryosparc.com/files/KTvMuJw2u6AGEgQ9KR07) Only the Segger regions corresponding to the head remain Next, the Segger model must be converted into an \`.mrc\` file which CryoSPARC can read. To do this, first select the remaining regions by control-click and dragging over them. Then click File > Save selected regions to .mrc file… in the Segger panel. You can name this file anything you like. This saves an \`.mrc\` map file with only the selected regions included. However, it is the wrong box size! {% hint style="info" %} In this guide we call the volume we just generated a “mask base” because we will use it to create a mask, but it has not yet been dilated and padded, and so should not be used as a mask in any refinements. {% endhint %} To make sure the mask base is on the same box as the input map, it must be resampled. Luckily, ChimeraX has a function to do this. In the example command below, #4 should be your mask base .mrc volume and #1 should be your original, unblurred volume. Change the numbers as necessary to match your work. \`\`\` volume resample #4 ongrid #1 \`\`\` Now your mask base and volume are in the same box size! To upload the mask base, it must first be saved to the local computer. In the command below, we save the mask (#5, adjust as necessary) to the desktop with a filename indicating \* On what job is this mask based? \* Project 300 (P300), job 3 (J3) \* Which regions of the map are included in this mask base? \* Head \`\`\` save ~/Desktop/P300-J3\_head.mrc #5 \`\`\` Note that this naming convention is entirely optional, you can choose any name you like. Regardless of the name, the mask base can now be uploaded to the compute system which runs CryoSPARC. In our case, mask bases are stored in separate directories per-target, but you can use any organizational scheme that helps you! \`\`\`sh scp ~/Desktop/P300-J3\_head.mrc {username}@{host}:~/masks/tri-snRNP/ \`\`\` ### Dilate and pad the base to create a mask Back in the CryoSPARC UI, run an \[Import Volumes\](/processing-data/all-job-types-in-cryosparc/import/job-import-3d-volumes.md) job to import the mask base you just uploaded. You can leave everything else as default, including that we are importing a map. Since the mask base has not yet been binarized and padded, we don’t want to accidentally use it where we need a mask! ![](https://guide.cryosparc.com/files/wp2WLG24eYUXHhTNbNOL) The mask base imported into CryoSPARC {% hint style="info" %} Binarization is the process of converting a map (which has smoothly varying values ranging from, typically, 0.0 to 1.0) to a mask that has only 0.0 or 1.0. Padding is the process of adding a soft edge to the binary mask to reduce ringing artifacts. {% endhint %} The next step is dilating and padding the mask base to produce our final mask. Create a Volume Tools job and connect the imported volume as the Input Volume and change the following settings | Parameter | Value | | --------------------- | ----- | | Type of output volume | mask | | Threshold | 0.05 | | Dilation radius | 5 | | Soft padding width | 17 | These settings will binarize our mask so that everything we included during segmentation (which all has a value greater than 0.05) is set to 1.0 and everything else is 0.0. Then, the mask will be expanded with 1.0 outward by 5 pixels (7 Å). This setting is the Dilation radius. We pad the mask to make sure that all of the information in the volume is covered by 1.0 once the alignment improves and the amount of the head we can see increases. Finally, the mask is padded with a soft edge that is 17 pixels (23.8 Å) wide. This is the Soft padding width parameter. This is a bit wider than the minimum we recommend (in this case, 13 pixels), but it is generally better to start with too large of a soft edge and decrease it if alignments don’t improve. {% hint style="warning" %} It is absolutely critical that any mask which is used to cut through map density has a soft edge. See the \[Mask Creation page\](/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera.md#why-do-masks-need-a-soft-edge) for more discussion of artifacts caused by masks with a hard edge. {% endhint %} Launch the Volume Tools job. It should run relatively quickly. Once it is complete, download the result and open it in the same ChimeraX window as your map. It should cover the head but not the rest of the tri-snRNP. As you contour the mask down, it should slowly expand away from your selection. ![](https://guide.cryosparc.com/files/IiPwHNJ8yvU6Qhlw3XuV) The mask (blue mesh) slowly expands as the contour is decreased from 1.0 to 0.0. \## Important Parameters Before creating our first Local Refinement job, we will cover a few commonly-changed parameters. A full discussion of all the settings is available in the \[main job page\](/processing-data/all-job-types-in-cryosparc/local-refinement/job-new-local-refinement-beta.md). ### Rotation and Shift Search Extents One of the major differences between Local Refinement and other types of refinement is that Local Refinement uses our existing knowledge about the particle poses, rather than starting from scratch. During each iteration, the refinement algorithm checks what the particle’s pose currently is. Then it checks the poses within a certain distance from that starting pose to see which one matches the volume best. The distance the algorithm checks is the Search Extent. For example, we know the head is not currently well aligned, but it is also not totally out of alignment. In other words, we expect that there is a moderate amount of rotation and movement, so the algorithm should check poses that are a moderate distance away from the current pose. If, however, we were aligning the body (which is already quite well-aligned), we could reduce the search extents significantly. Counterintuitively, if you are working with a small flexible domain (such as the arm), you may want to reduce the search extents even if you believe the domain is quite flexible. When the algorithm is aligning a small domain it doesn’t have much information to work with. Only letting it move a small distance from the initial alignment prevents it moving these small domains far away from the main bulk of the protein due to nearby noise. ![](https://guide.cryosparc.com/files/po4bXPXbT4D4Ge7tC9w6) Poorly-aligned particles will require larger search extents than particles which are already well-aligned. \### Fulcrum Position #### Why set a fulcrum? Consider the alignment algorithm again: 1. Mask out the subvolume 2. Search local translations and rotations for a better pose 3. Generate a new volume and repeat It’s step 2 that’s important here: when the particle is rotated, what is it rotating around? ![](https://guide.cryosparc.com/files/lTFqy1WjdYTpLmosul8h) The result of a 20° rotation depends on the fulcrum about which the object rotates. There’s no obvious best answer, so by default we rotate around the center of the mask. This works well when \*\*the mask covers a large proportion of the total volume\*\*. The head domain does not cover a large proportion of the volume. In this case, it might be better to rotate the particles around the center of the box, since that’s closer to the real hinging motion we expect to see. Rotation around the box center tends to work better when \*\*the mask covers a moderately-sized proportion which rotates relative to the main volume\*\*. But there’s no reason we have to pick one of these two points. Our intuition tells us that \*\*the head ought to rotate around some point on the head/body interface\*\*. In the next step you will pick a point on this interface and set that as the fulcrum. #### Finding the fulcrum Look back at your mask base in ChimeraX. To set the fulcrum, CryoSPARC needs the coordinates (in pixels, counting from the corner of the box) of the point we want to rotate about. So we need to determine the coordinates of a position on the surface of the interface, which is the edge of our mask base. ![](https://guide.cryosparc.com/files/Zw01qTStzQ3mpqliK4Sd) The fulcrum should be somewhere on the interface between the head and the rest of the tri-snRNP molecule. To get these coordinates in ChimeraX: 1. Navigate to the “Markers” ribbon menu. 2. Click “Surface” in the “Place markers” group. 3. Orient the mask so that you can see the surface you want to place the fulcrum on. 4. Right-click the surface to place a marker. 5. Read the coordinates in Å from the log. 6. Divide the coordinates by the pixel size (which is available with the command \`info #1\`, replacing \`#1\` with the correct number for your map) to get the pixel coordinates. ![](https://guide.cryosparc.com/files/P6PgvcqrGWk43Tfrhqoj) ChimeraX reports the position of markers in Å, which must be converted to pixels using the map's pixel size. In this example, the marker was placed at \`(249.5, 250.1, 253.3) Å\`, so the fulcrum position will be \`(178.2, 178.6, 180.9) px\`. ## Build the Job It is finally time to build the first Local Refinement job! Create a Local Refinement and connect the particles and volume from the Non-Uniform Refinement to the correct slots. Then connect your mask to the static mask slot. Finally, set the fulcrum you calculated in the last step. {% hint style="info" %} CryoSPARC expects the fulcrum in the form x,y,z, with no parentheses or spaces. For example, \`178.2,178.6,180.9\` {% endhint %} Leave all the other parameters set to their default and launch the job! ## Diagnostic Plots Local Refinement takes about as long as other refinement jobs, depending on the search extents and the quality of the initial alignment. Once the first iteration finishes, you will have access to several diagnostic plots. These plots are useful in assessing job progress and ensuring that parameters were set as expected. The first three plots that Local Refinement shows you are slices through the real space of your map, the Fourier space of your map, and the real space of your mask. ![](https://guide.cryosparc.com/files/C5wnUGWEo1wsCcWRGfKJ) Slices through the map in real space. ![](https://guide.cryosparc.com/files/rHhlNyef6BZoGChquh7x) Slices through the map in Fourier space. ![](https://guide.cryosparc.com/files/1nU9DV3185pUdoZkYiYP) Slices through the mask in real space. These plots largely exist to give you a sense of how the refinement is progressing without having to download the map at each stage of the refinement. One annotation to note is the pair of white dotted lines in the map and mask slices, one vertical and one horizontal. The intersection of these lines shows you the fulcrum point. Note that the fulcrum looks like it’s positioned at the head/body interface as expected! ![](https://guide.cryosparc.com/files/eAstizH9kx3i1K5zHxDZ) The Gold Standard Fourier Shell Correlation (GSFSC) plot. Plots of the Gold Standard Fourier Shell Correlation (GSFSC) demonstrate the correlation between the two independent half maps and determine the resolution to which we can trust our maps. It’s not uncommon for the unmasked GSFSC curve to be relatively poor during a Local Refinement. The alignment ignores everything outside the mask, which means the score of a given pose is not affected by even significant mismatches outside the mask. ![](https://guide.cryosparc.com/files/4Q2isWde3l7vzEbHZ4UD) The Guinier plot. A Guinier plot visualizes the contribution of a given resolution shell to the final map. As discussed in \[Rosenthal and Henderson\](#references), this plot is used to determine the optimal sharpening factor (the B-factor). The “sharp” map output has this B-factor applied to it, but you can always generate maps with other sharpening factors using the Sharpening Tools job. ![](https://guide.cryosparc.com/files/i9kxPZ7fUutp9YeTnPug) The noise model. The noise model is an important component of any cryoEM image processing algorithm. Briefly, the noise model is used to modulate the penalties associated with poor correlation in a frequency shell by the expected quality of signal in that frequency shell. Put another way, if a particular frequency is very noisy, it should not surprise us when the 3D model does not agree well with the images in that frequency. Noise models for cryoEM generally have a high peak at the low resolutions (left), rapidly drop in the moderate resolutions (middle), and steadily rise as the resolution increases. ![](https://guide.cryosparc.com/files/joJQMWHdiuDQ1z21UI2L) The distribution of particle viewing directions. ![](https://guide.cryosparc.com/files/Mr5Q3u459hTr1lXfHIWr) The posterior precision plot. The viewing direction and posterior precision distributions are used to determine whether a particle stack suffers from orientation bias. The direction distribution directly plots the number of particles with a given pose. The posterior precision distribution is a measure of how confident we are in the volume’s quality when viewed from each direction. As long as your lowest and highest values in this plot are within an order of magnitude, your dataset likely samples all orientations enough to avoid significant anisotropy. ![](https://guide.cryosparc.com/files/sd1SAgWNLDAkCPSlXqbh) The distribution of angle and shift changes. At this early iteration, shifts are high and running up against the maximum shift extent. These histograms display how much each particle moved during this iteration. In this first iteration, particles are moving a lot and (especially in the shifts), bumping up against our search extent. However, right now the map of this region is not very good (remember that the input map was lowpass filtered, so these particles are aligning to a 12 Å map). If we see large peaks at the edge of our search extents in late iterations, we will have to consider re-running the job with larger search extents. ![](https://guide.cryosparc.com/files/pGmtG1AvhJuf75rCMNcZ) Per-particle scale distribution. In this example, per-particle scale has not been optimized, so all particles have a scale of 1.0. Finally, per-particle scale is a way of accounting for the fact that different images will have different absolute contrast due to different ice thickness, defocus, etc. Generally one should only refine per-particle scale when looking at the entire volume, so we have not refined these values. Thus, all particles are still at 1.0. ## Results - Local Refinement ![](https://guide.cryosparc.com/files/qCgKZIthhX8IYU5dZRWd) The final distribution of angle and shift changes. Note that the early peak at the maximum shift change has disappeared. These look good: they are both smooth, and there are no large spikes at the edge of our search parameters. There are some particles shifting all the way out to 10 pixels so future similar jobs may benefit from increasing the shift search extent, but this looks fine for now! The subvolume is indeed quite flexible, with thousands of particles rotating 20° or more! Next, take a look at the GSFSC curve: ![](https://guide.cryosparc.com/files/itL09NX6GayufDWR6Ntr) The GSFSC curves for the final iteration It is not surprising that the unmasked FSC curve is poor, since we’re only aligning a small subvolume. It’s good that all three curves are smooth and decrease all the way to zero. The mask in this case may have been a little tight — the Corrected curve does not closely track the Tight curve until higher resolutions. However, it does “catch up” eventually, so these results are likely still trustworthy. {% hint style="info" %} When performing Local Refinements, pay attention to the Corrected GSFSC curve. If it does not closely track the Tight curve, your mask may be too tight. See the \[Mask Creation page\](/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera.md) for more information. {% endhint %} Finally, download the sharp map. This map has the sharpening factor from the Guinier plot automatically applied and will give you a good sense of the quality of the alignment. ![](https://guide.cryosparc.com/files/t6PoyVdRIGGxFjLGa36q) The result of performing a Local Refinement with a mask around the head. This is a dramatic improvement in how much of the head is visible. Note, though, that the maps of the body and foot are much worse — when we align the head, the head/body flexibility causes these domains to blur out instead! The GSFSC resolution (4 Å) is already slightly improved over that of the published result (4.2 Å) for the head. However, there is another step which may further improve the results. ## Particle Subtraction Recall that Local Refinement aligns a masked volume to the full particle images. In this case, the masked volume is just the head, while the images contain all four domains. Many of the particle images have the body, foot, or arm directly above or below the head. When the electron beam passes through, these other domains “cast a shadow” on the image of the head domain. This can hurt the alignment, since the masked volume does not have information from these domains but the images do. To fix this problem, we can first project the volume of the foot, body and arm (but not head) for each image, and then subtract this projection from each image. This leaves images containing only the information from the head domain, the same as our masked volume. The efficacy of this technique depends on the quality of the subtracted domains’ alignment. Subtracting a blurry or flexible domain from images will leave shadows and other artifacts, which wouldn’t improve the results. Ultimately, whether Particle Subtraction helps or hurts with a particular dataset is empirical: you won’t know until you try! ### Mask Creation For this job, we will make a mask using the same process as for the head, except including everything other than the head. Be sure you build this mask using the Non-Uniform Refinement, since we need the body and foot to be well-aligned! ![](https://guide.cryosparc.com/files/XBNp1w5AsMlkkIWqPiBP) The mask for particle subtraction, which excludes the head domain. {% hint style="info" %} If in the future you find that you are often performing both Particle Subtraction and Local Refinement, going through the process of map segmentation twice can be irritating. To avoid this, after saving your first mask (of the region you want to keep), you can delete the regions used to create the mask. Finally, show all regions to bring back the hidden regions. You can then save these regions to make your mask for Particle Subtraction without having to re-select anything. More tips on mask generation can be found in \[our guide page\](/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera.md). {% endhint %} For this mask you should not dilate the base at all, since you do not want to subtract surrounding noise from the particle images. A soft edge is still necessary. For this example, we used the recommended minimum soft padding of $$5 \\times{} \\frac{\\mathrm{resolution}}{\\mathrm{apix}}$$ which works out to be 12 pixels. Plug the particles and volume from the Non-Uniform Refinement into a Particle Subtraction job along with the mask you just made. All default parameters are fine, so go ahead and launch the job! Once the job completes, we recommend performing a Homogeneous Reconstruction Only job to ensure that the results are as expected. This step is optional, as you won’t use the resulting map for anything, but it only takes a few minutes to run and can save you a lot of time if you catch a failed subtraction before running a whole refinement! The reconstructed map from the subtracted particles in this example looks like this: ![](https://guide.cryosparc.com/files/vJMQT2djKKIeCObh1Wdx) The results of subtracting the arm, body, and foot domains. Clearly, the body and foot have been subtracted successfully. At lower contours there is still some remaining signal from the arm. This isn’t entirely surprising, since the arm’s alignment was bad to begin with. In any case, the arm is small, so we have successfully subtracted most of the volume that lies outside our mask. Use these subtracted particles in a Local Refinement to see if you can improve the resulting map. Clone your previous local refinement and replace the particles with the new, subtracted particle stack. You should also slightly increase the shift search extent to 14 A, since some particles were against the edge of the extent in the first refinement. Leave all other settings as you had them previously and launch the job. ## Results - Local Refinement after Particle Subtraction With our masks, particle subtraction improved the GSFSC resolution by an additional 0.1 Å, which is within the realm of how much any two reconstructions might differ by chance. More importantly, signal subtraction kept the GSFSC curve higher in the middle resolutions, which has a significant impact on the overall quality of the map despite the similarity of the GSFSC resolution. When directly comparing maps with and without particle subtraction, it appears that some regions benefit greatly from subtracting away the body and foot: ![](https://guide.cryosparc.com/files/0q6Vzlh9hwwFnWkheAYb) A comparison of the local refinement with (purple) and without (cyan) particle subtraction shows significant improvement in this region, with side chains visible and more connected density in the particle subtracted map. while other regions only benefit modestly: ![](https://guide.cryosparc.com/files/IUydpAQbS9pyXV9b8kZh) A comparison of the local refinement with (purple) and without (cyan) particle subtraction shows only modest improvement in this region. In the end, like many other steps of a cryoEM workflow, the optimal combination of subtraction, masking, and parameters must be determined empirically for each dataset. ## Conclusion Local Refinement is an essential tool in the analysis of targets with rigid domains separated by a hinge. In each step of a Local Refinement, the optimal pose for a masked subvolume is found for a given set of particle images, which may or may not have signal from other regions of the particle subtracted. This leaves three major domains of optimization for the user: 1. The mask 2. Search extent and other refinement parameters 3. Particle subtraction These domains can be optimized together or independently, and often several iterations are necessary to achieve the best result. ## Exercises ### The foot domain [](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp.md#docs-internal-guid-da9bd864-7fff-1258-2029-53da86db3392) Perform a local refinement of the foot domain. The published map for the foot domain alone is at 3.7 Å. We were able to improve the map of the foot all the way to 3.4 Å using the same techniques as for the head domain! ![](https://guide.cryosparc.com/files/d3rSLgAKKw1M5PlyHQjX) Significant improvement of the foot domain before (purple) and after (cyan) Local Refinement. \### Investigate mask parameters Make a series of masks using the same mask base but varying the dilation radius and soft padding width. Run Local Refinements with all settings the same, except using a different mask in each (you may want to use a subset of particles to speed things up). Compare the results. Do you notice any trends? Which mask do you consider optimal for this refinement? Are the same settings best for other domains? ### Masking small domains What is the smallest domain you can mask before the results become unreliable? Why do you think it’s harder to align smaller domains? Can you think of any settings you could change to improve the results? ## References 1. Thi Hoang Duong Nguyen et al., “Cryo-EM Structure of the Yeast U4/U6.U5 Tri-snRNP at 3.7 Å Resolution,” \*Nature\* 530, no. 7590 (February 1, 2016): 298–302, . 2. Grigore Pintilie and Wah Chiu, “Comparison of Segger and Other Methods for Segmentation and Rigid-Body Docking of Molecular Components in Cryo-EM Density Maps.,” \*Biopolymers\* 97, no. 9 (September 2012): 742–60, . 3. Peter B. Rosenthal and Richard Henderson, “Optimal Determination of Particle Orientation, Absolute Hand, and Contrast Loss in Single-Particle Electron Cryomicroscopy,” \*Journal of Molecular Biology\* 333, no. 4 (October 31, 2003): 721–45, . --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-yeast-u4-u6.u5-tri-snrnp.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures.md). # Tutorial: Tips for Membrane Protein Structures Membrane proteins are an increasingly important class of targets for cryo-EM in academia and industry. These targets are often small (<100kDA in molecular weight), flexible, and have a large micelle in the transmembrane region. Here, we list a few suggested tips for working with these targets in CryoSPARC, sorted by the different stages of processing. ![](https://guide.cryosparc.com/files/JDYHVKcZ5cQuJdzhs6Ov) One type of membrane protein: the Cannabinoid Receptor 1-G GPCR complex, (Kumar et al., 2019). Data from EMPIAR-10288. Density shown at two different thresholds to illustrate the micelle regions. ![](https://guide.cryosparc.com/files/4FOWrvDGWLaGE5X67Xno) \### Pre-Processing Generally, pre-processing steps remain unchanged from other nominal cryoEM pipelines—namely, we recommend the use of the \[\*\*Patch Motion Correction\*\*\](/processing-data/all-job-types-in-cryosparc/motion-correction/job-patch-motion-correction.md) and \[\*\*Patch CTF\*\*\](/processing-data/all-job-types-in-cryosparc/ctf-estimation/job-patch-ctf-estimation.md) jobs with no salient modifications to the parameters. We find that per-particle CTF refinements (post-3D refinement) rarely improve final structures due to the low amount of signal present per-particle in the micrograph for small membrane targets. Nevertheless, per-particle CTF refinement may be useful to try once a sufficiently detailed structure is refined. ### Particle Picking Particle picking can be one of the most challenging parts of working with membrane proteins. \[\*\*Blob Picker\*\*\](/processing-data/all-job-types-in-cryosparc/particle-picking/job-blob-picker.md) \*\*/\*\* \[\*\*Template Picker\*\*\](/processing-data/all-job-types-in-cryosparc/particle-picking/job-template-picker.md) For crowded micrographs, the following two parameters can substantially affect picking performance: \* \`Particle diameter\` \* \`Min. separation dist\` \* Reducing this for crowded datasets may help pick out more true particles. Other potentially useful picking jobs: \* Neural-network-based particle picking techniques such as \[\*\*Topaz\*\*\](/processing-data/all-job-types-in-cryosparc/deep-picking/topaz.md) or \[\*\*Deep Particle Picker\*\*\](/processing-data/all-job-types-in-cryosparc/deep-picking/deep-network-particle-picker.md) can be useful when a large portion of particles are difficult to identify visually \* \[\*\*Blob Picker Tuner\*\*\](/processing-data/all-job-types-in-cryosparc/particle-picking/job-blob-picker-tuner.md) can also be quite useful for crowded micrographs. Be sure to choose approximately 100 manual picks, focusing on picks that are ‘clumped together’ and originating from micrographs that span a wide range of defocus values. \[\*\*Extract from Micrographs\*\*\](/processing-data/all-job-types-in-cryosparc/extraction/job-extract-from-micrographs.md) We suggest using an extraction box size that is approximately 2-3 times larger than the particle diameter. To account for signal displacement caused by the CTF, Rosenthal and Henderson (2003) suggest a box size of: $$ D + 2 R = D + 2 (\\lambda \\Delta F / d), $$ where $$D$$ is the diameter of the particle, $$\\lambda$$ is the electron wavelength, $$\\Delta F$$ is the defocus value and $$d$$ is the resolution. Note that the radius of displacement, $$R$$, is not a function of the particle diameter, and therefore this formula may result in a box size that is 4-5 times larger than the particle diameter when $$D$$ is relatively small (as is typically the case for membrane proteins). Many of the CryoSPARC algorithms (e.g., 2D classification, ab-initio), however, are tuned for particle images with a box size that is 2-3 times larger than the extent of the particle. Furthermore, substantial computational savings can be achieved by using a smaller box size at early stages of processing where there are potentially many particles (millions). With small membrane proteins at reasonable concentrations, it is common to have many particles (several hundred) per micrograph and therefore very large particle sets in initial classification. To address this, a suggested pipeline is: 1. Extract particles with smaller box size (1.5X - 2X particle extent), 2. Perform multiple rounds of 2D classification, ab-initio, 3D classification, and initial (heterogeneous) refinements, selecting the best particles to carry forward 3. Re-extract surviving particles with a larger box size (2X - 3X particle extent) to reasonably account for all the information spread due to the CTF, and finally 4. Perform high resolution refinement(s). ### Particle Curation During 2D classification, a number of parameter changes can help improve performance for membrane targets: \[\*\*2D classification\*\*\](/processing-data/all-job-types-in-cryosparc/particle-curation/job-2d-classification.md) \* \`Force Max over poses/shifts\` \* By turning this off, 2D classification will automatically marginalize over the poses and shifts of each particle. For small particles, the uncertainty over poses and shifts can be substantial, and account for this through marginalization over these unknowns can be beneficial. Marginalization will add computational cost, but can help improve classification results in general when SNR is low. When this option is used, 2D classes will appear more “radially blurred” with less streaky or noisy artefacts towards the periphery. \* \`Number of iterations\` \* Increasing the default value of 20 may help improve classes. \* \`Batch size\` \* Empirically, users have found that doubling the initial value to 400 is sometimes beneficial. \* \`Circular mask diameter\` \* This can help account for crowding by masking out any information outside of a circular region in each particle image. For small particles with a lot of crowding, this can be necessary to ensure classification is based on view/conformation rather than arrangement of neighbours. ### Reconstruction & Refinement #### A note about masks First and foremost, all masks applied during 2D-to-3D processing should be smooth (i.e., contain no sudden 'cliffs' where the mask drops from a value near 1 to a value near 0) to avoid ringing effects. This is because sharp masks, when applied to half-maps during refinement jobs, can increase the likelihood of overfitting by introducing artifactual signal that is common to both half-maps. If you are generating masks using Chimera(X) (e.g., by following our \[tutorial\](/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera.md)), be sure to use the\[ Volume Tools\](/processing-data/all-job-types-in-cryosparc/utilities/job-volume-tools.md) job to add a sufficient soft padding width. As noted by the \[mask generation tutorial\](/processing-data/tutorials-and-case-studies/mask-selection-and-generation-in-ucsf-chimera.md#importing-and-processing-the-masks-in-cryosparc), a useful rule of thumb is to keep the mask padding width proportional to the achieved resolution in Angstroms. As long as the \*soft padding width\* is sufficient, and the mask covers the desired region of structure (while "cutting" through minimal density), the \*threshold value\* and \*dilation radius\* may be set as needed in order to generate a mask of the desired size. Furthermore, it is especially important for membrane target masks to not be overly 'tight' to the structure. For such small proteins, a tight mask can more easily lead to a situation where a refinement 'overfits' to junk/noise (cf. \[Common Failure Modes\](#common-failure-modes)). In general, avoid creating a mask that is similar in shape to the secondary structure of the protein, and err on the side of loose (but nevertheless smooth) masks for all processing. \[\*\*Particle subtraction\*\*\](/processing-data/all-job-types-in-cryosparc/local-refinement/job-particle-subtraction-beta.md) In general, we find that particle subtraction can only help in very specific situations. Namely, if your structure contains two very rigid subunits, one large and one small. In this case, subtracting the larger subunit can improve the resolution of the smaller unit, if particle alignments are sufficiently well resolved for this subtraction to accurately remove the larger subunit signal. We strongly recommend avoiding the subtraction of micelles -- these structures are generally disordered, and it is very difficult to subtract them from particle images without removing other useful signal. Instead, consider the use of \[local refinement\](/processing-data/all-job-types-in-cryosparc/local-refinement.md) with non-uniform refinement and marginalization turned on. \[\*\*Ab initio\*\*\](/processing-data/all-job-types-in-cryosparc/3d-reconstruction/job-ab-initio-reconstruction.md) \* \`Initial / maximum resolution\` \* For smaller membrane proteins, it is often useful to set the initial and maximum resolutions to smaller numerical values (e.g., 9Å and 7Å). This is because smaller particles appear as featureless blobs at lower resolutions and there will not be enough information to align particles and recover the structure. \[\*\*Non-uniform refinement\*\*\](/processing-data/all-job-types-in-cryosparc/3d-refinement/job-non-uniform-refinement-new.md) Non-uniform refinement can significantly improve refinements for targets that contain micelles and for smaller proteins. Consider the following two modifications if refinement results are poor: \* \`Initial lowpass\` \* Empirically, increasing this resolution (e.g., to a lower numerical value such as 15Å) may improve results for smaller targets. \* \`Static masking\` \* For small, low-SNR particles, dynamic masking may perform poorly. Instead, supplying a soft, static mask may improve the final refinement. \[\*\*Local refinement\*\*\](/processing-data/all-job-types-in-cryosparc/local-refinement.md) Local refinement can also be quite useful for membrane targets. Note that local refinement masks \*\*must\*\* be softly padded, especially when cutting into density (even a micelle). A few salient parameters to consider: \* \`Rotation/Shift search extent\` \* When using smaller masks, tighter orientation search extents generally produce better results. \* \`Marginalization\` \* (Default on) Marginalization over poses and shifts can greatly improve alignments for smaller targets. \* \`Non-uniform refine enable\` \* (Default on) Non-uniform refinement can help account for disordered regions (such as micelles and flexible/floppy appendages). \* \`Rotation/Shift gaussian prior widths\` \* In cases of small molecules, small masks, or poor SNR, local refinement may benefit from the introduction of gaussian priors around each particle's initial orientation parameters. The utility of these priors is commensurate with the quality of the initial alignments. ### Heterogeneity Analysis CryoSPARC includes a wide assortment of tools for assessing and separating heterogeneous datasets. For high-resolution refinement of any protein, it is critical to ensure that the dataset is as homogeneous as possible; this often entails both particle curation (junk removal) and pruning of heterogeneity. In addition to 2D Classification and Ab-Initio Reconstruction, several other job types for heterogeneity analysis are highlighted below, along with important parameters to consider. \[\*\*Heterogeneous Refinement\*\*\](/processing-data/all-job-types-in-cryosparc/3d-refinement/job-heterogeneous-refinement.md) \* \`Force hard classification\` \* Hard classification can improve results for low-SNR particles, especially when the target contains a static (well-resolved) domain connected to a flexible/heterogeneous domain (such as a micelle). \[\*\*3D Classification\*\*\](/processing-data/all-job-types-in-cryosparc/variability/job-3d-classification-beta.md) \* \`Force hard classification\` \* Similar to Heterogeneous Refinement, force hard classification can help isolate regions of heterogeneity. \* \`RMS convergence criterion\` \* For low-SNR particles, the standard class switching criterion may lead to more F-EM iterations than necessary and cause processing to take longer. Consider turning this secondary criterion on to save computational cost. \[\*\*3D Variability\*\*\](/processing-data/all-job-types-in-cryosparc/variability/job-3d-variability.md) 3D Variability (3DVA) analysis can be an especially important tool for heterogeneity analysis of small membrane targets. The 3DVA publication includes results on the Cannabinoid Receptor 1-G GPCR, which show that 3DVA can resolve two different bending motions of the 53kDa transmembrane region of the protein. When running 3DVA, be sure to supply a soft solvent mask to ensure that the job does not resolve variation due to the micelle. It is often advantageous to use a mask that \*excludes\* the micelle, nanodisc, or other disordered regions, in order to force the algorithm to focus only on variability within the ordered region. ### Common Failure Modes \* "Spiky" densities like the one shown below are often a sign that there are many junk particles in the dataset — this can be especially prevalent in membrane protein datasets where particle picking is difficult. In these cases, it is often helpful to further “purify” the dataset, by either: \* performing additional 2D classification rounds, or \* running ab-initio reconstruction with multiple classes, then using the resulting volumes (including junk classes) to initialize heterogeneous refinement or 3D classification jobs and processing all the particles. Particles that fall into intact classes where the protein density is strong can be used for further refinements and particles falling into other classes can be discarded. This “junk-sorting” in 3D can often separate junk particles more effectively than 2D classification. ![](https://guide.cryosparc.com/files/H8bgLBIkJR6XJ5S2htNB) A ’spiky’ hyaluronan synthase (the same density shown at two different thresholds for clarity) resolved from one class of a 3D classification job. Data from EMPIAR-11030 (Maloney et al., 2022). \### Useful Discussion Threads \* \* \* \### Citations Kumar, Kaavya Krishna, et al. "Structure of a signaling cannabinoid receptor 1-G protein complex." \*Cell\* 176.3 (2019): 448-458 Maloney, Finn P., et al. "Structure, substrate recognition and initiation of hyaluronan synthase." \*Nature\* 604.7904 (2022): 195-201. Rosenthal, Peter B., and Richard Henderson. "Optimal determination of particle orientation, absolute hand, and contrast loss in single-particle electron cryomicroscopy." \*Journal of Molecular Biology\* 333.4 (2003): 721-745. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-tips-for-membrane-protein-structures.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide.md). # New Live Session: Start to Finish Guide ## Start to Finish Guide This guide covers the end-to-end workflow for using CryoSPARC Live. In addition to the below, we recommend checking out the \[CryoSPARC Live Walkthrough\](/live/tutorial-videos.md): {% embed url="" %} Processing EMPIAR-10288 in CryoSPARC Live. {% endembed %} ## 1. Create New Session > From the \*\*CryoSPARC Live Sessions View,\*\* create and configure new sessions and view a summary of existing sessions. ➡️ Navigate to the Sessions View by clicking on the CryoSPARC Live icon on the sidebar. ![](https://guide.cryosparc.com/files/Pc5EV6aXOtaEv5Rq2kXm) ➡️ Click on the \*\*New Session\*\* button in the header, which opens a panel in the sidebar. ![](https://guide.cryosparc.com/files/CIGEapnJcvM5FzSGsjfq) ➡️ Select or enter the CryoSPARC Project number where you would like this Session to be created. Enter a Title for the Session. {% hint style="warning" %} \*\*The project you select must already exist.\*\* You may need to create a new Project in the CryoSPARC interface, if one does not already exist.\\ \\ \*\*Recommended scenarios for creating new Projects/Sessions:\*\* \* \*\*Collecting new data for the first time on a new target molecule:\*\* Create a new Project and a new Session within it. \* \*\*Collecting data a second time on the same sample/target\*\* (potentially the same or different grid from the same batch, potentially on a different day): Use the existing Project and existing Session where you processed the first set of images. In the existing Session, create a new exposure group (see Configure New Session, below) and start the Session again, causing it to read in the new set of images. \* \*\*Collecting data on a new sample/grid/preparation of the same target molecule:\*\* Use the existing Project, but create a new Session. This allows easy re-use of 3D volumes, 2D templates, and easy combining of particle images downstream.\\ You can create multiple Sessions within a project, for example if collecting/processing new data from a similar sample. The recommended workflow is to create a new Project for each new unrelated sample on which you are collecting data. \*\*Only users who own a particular Project or have a Project shared with them can see CryoSPARC Live Sessions within those Projects.\*\* To add a user to a Project, navigate to the Project Details Panel in your regular CryoSPARC instance and click \`Share With Users\` to select the user you wish to give access. {% endhint %} ➡️ Click Create Session. This will open your new session. All new sessions by default are set to Paused status. ## 2. Configure New Session > On the \*\*Configuration Tab\*\*, enter required parameters (or load a saved Configuration Profile) and select compute resources. New Sessions will open on the \*\*Configuration Tab\*\* by default. In order to Start this New Session, you must first type in (or load from a profile - see \[below\](#load-configuration-profile)) a few required parameters in three sections (Configuration, Parameters, and Exposure Groups). Required parameters are outlined in red and are also summarized in the \*\*Start Checklist.\*\* !\[\](/files/-MNiw5jwzZ\_86don4mKR) ### Configuration Section #### Compute Resources These are parameters relating to the GPU resources on which CryoSPARC Live jobs will run. For more details on hardware requirements for Live, please see: \[Prerequisites and Compute Resources Setup\](/live/prerequisites-and-compute-resources-setup.md) ![](https://guide.cryosparc.com/files/qnO2Is1OFFsuXDWEarRe) ➡️ \*\*Select/enter the following required parameters\*\* \* \`Preprocessing Lane\`: Select a Preprocessing Lane and a Number of Preprocessing GPU Workers. This is effectively the number of GPU workers that will carry out motion correction, CTF estimation, particle picking and extraction in parallel concurrently. {% hint style="info" %} It's possible to adjust the number of GPU workers allocated for the preprocessing stage throughout the lifecycle of the session. For example, at the start of the session, you can allocate four GPUs to quickly extract particles from exposures, then lower the number of workers to two or one when resources are needed for the reconstruction stage. For information on the minimum number of GPU workers you need to assign, see: \[Prerequisites and Compute Resources Setup\](/live/prerequisites-and-compute-resources-setup.md) {% endhint %} \* \`Reconstruction Lane\`: Select a Reconstruction Lane where Streaming 2D classification and Streaming Refinement jobs will be launched. \* \`Auxiliary Lane\`: Select an Auxiliary Lane where Ab-initio Reconstruction and Generate Templates jobs will be launched. \* (Optional) \`Use SSD\`: If the disk you are using for the project directory is already an SSD, you don't need to copy the files to another SSD. Turn off \`Use SSD\` if you do not wish to copy files over (for larger datasets, it can take some time to write all files over to the SSD before processing can stream in new collected particles). \* (Optional) \`Priority\`: Unless specified, all jobs in the Live Session will run according to the Session-level priority. For details on Priority Queuing in CryoSPARC, please see the \[Priority Queuing Tutorial\](/setup-configuration-and-management/software-system-guides/tutorial-priority-job-queuing.md#cryosparc-live-session-priority). \* (Optional) (v5.0+) \`Workers per GPU\` : Specifies the number of Live preprocessing worker jobs to launch per GPU worker. On some systems, launching multiple preprocessing workers per GPU can improve throughput without requiring additional GPU hardware. #### Run Configuration ![](https://guide.cryosparc.com/files/ClrRMLQlbcYN3Mevm03I) \* (Optional) \`Exposure Processing Priority\` : Determines the order in which exposures are processed or reprocessed. \* Normal: Exposures are processed in ascending UID order (i.e. earlier exposures are processed first), however exposures are reprocessed in descending UID order (i.e. recent exposures are reprocessed first). \* Oldest: The oldest exposure found will be (re)processed first. \* Latest: The latest exposure found will be (re)processed first. \* Alternate: Alternate between Oldest and Latest priority modes. This mode helps ensure that some recent exposures are always being processed, even if there is a backlog of older exposures that need to be reprocessed. \* (Optional) (v5.0+) \`Delay worker startup until ready\` : Whether to wait until there is at least one exposure ready to process before queueing Live Worker jobs. Turning on this option allows you to start a Live session before any data has been collected, as long as you know where the movie files will be saved by the microscope control software. The Live session will not consume GPU resources or license tokens while waiting for data to appear. In this way, multiple Live sessions can be started in anticipation of data that will arrive later (e.g. from a multi-grid collection setup) and in conjunction with the next option (auto pause), data from multiple collections can be processed in an unattended fashion. \* (Optional) (v5.0+) \`Auto Pause Mode\` : Automatically pause the session when there are no remaining exposures to process, and a sufficient amount of idle time has passed since the last processed exposure was found. \* Standard: Pause the session immediately after the idle timeout has expired. This will kill any running 2D/3D streaming jobs at the time auto pause is triggered. \* Graceful: Once the idle timeout has lapsed, also wait for 2D/3D streaming jobs to finish processing the available particles before pausing the session. \* (Optional) (v5.0+) \`Auto Pause Timeout\` : The amount of idle time that must pass after the last found exposure before automatically pausing. While a session is running, jobs launched by Live can be viewed by clicking on the \*\*'x Active Jobs'\*\* button in the footer of the CryoSPARC Live application. You can also see Live jobs in the Resource Manager tab in your regular CryoSPARC instance. To quickly navigate to the Resource Manager, click on any job in the CryoSPARC Live application footer. !\[\](/files/-MNiwB338-A\_2X0hSoRR) This will open the Active Jobs modal. !\[\](/files/-MNiwDuxU\_j\_EBVoWEys) {% hint style="info" %} To adjust the Number of Preprocessing GPU workers or any other Compute Resources settings while a Session is Running, you will need to first Pause the Session, update the configuration and then Start the Session again. {% endhint %} ### Parameters Section These are data processing parameters that will be engaged across Microscope/Camera, Motion Correction, CTF Estimation, Blob Picker, Template Picker, and Particle Extraction once the Session is Started. !\[\](/files/-MNiwH6IsFlAns6F8DVQ) ➡️ \*\*Step 1: At a minimum, you need to enter the following 7 required parameters which are outlined in red.\*\* \* \*\*Microscope/Camera Parameter\*\* \* \`Raw pixel size (A)\`: The raw pixel size of the input movie data. For super-resolution data, this should be the super-resolution pixel size (i.e. not the camera native pixel size). For EER data, this should be the camera native pixel size, and you should also modify the corresponding EER parameters for upsampling and dose fractionation. \* \`Accelerating voltage (kV)\`: The accelerating voltage of the microscope collecting the data. \* \`Spherical abberation (mm)\`: The spherical aberration of the microscope collecting the data. This should be zero if there is a Cs-corrector. \* \`Total exposure dose (e/A^2)\`: This should be the total electron dose across each movie (i.e. not the per frame dose). \* Blob Picker \* \`Minimum particle diameter (A)\`: This can initially be set as the minimum dimension of the particle expected. This can be fine-tuned later on the Picking tab. \* \`Maximum particle diameter (A)\`: This should be set equal to or slightly larger than the maximum dimension of the particle expected. This can be fine-tuned later on the Picking tab. \* Particle Extraction \* \`Extraction box size (pix)\`: This should be a box size in pixels for extracting particles. Generally, this should be set to about twice the particle diameter, and should be a set to a number that has 2,3,5,7 as its prime factors. Typical box sizes are between 256 and 640 pixels. Note that the box size is in pixels after Fourier-cropping (if enabled in motion correction). Mathematically good numbers are: \* \`\`\` 32, 36, 40, 42, 48, 56, 60, 64, 70, 72, 80, 84, 90, 96, 100, 108, 112, 120, 128, 144, 160, 180, 192, 200, 216, 224, 240, 256, 270, 288, 300, 320, 324, 336, 384, 400, 432, 448, 450, 512, 576, 640, 648, 672, 720, 768, 784, 810, 864, 882, 1024, 1152, 1280, 1296, 1344, 1440, 1568, 1620, 1728, 1792, 2000, 2048 \`\`\` {% hint style="info" %} If you would like to save micrographs or particles in \[float16 format\](/processing-data/tutorials-and-case-studies/tutorial-float16-support.md) , you can enable the toggle titled "Save results in 16-bit floating point" under "Motion correction", "Particle extraction", or both. (CryoSPARC v4.4+) {% endhint %} To view advanced parameters, click \*\*Show Advanced\*\*. You can also filter by a parameter name. !\[\](/files/-MNiwKllgJwgdioY2\_XA) ➡️ \*\*Step 2: Apply (i.e., Save) your parameters\*\* \* Click \`Apply to All\` to save your parameter entries for all exposures (existing and incoming) in the Session. \*\*This is the recommended course of action when starting a New Session.\*\* \* If your Session is already running and you wish to have new parameter changes apply only to new exposures coming in after that point, then you may instead wish to click \`Apply to Future\`. ### Exposure Group Section This section covers parameters that will tell CryoSPARC Live where to find new movie files, which files to read, whether to apply a gain reference and/or defect file (if available), and how to handle multiple Exposure Groups, if applicable. {% hint style="info" %} By default, there is always at least one Exposure Group in a Session. Exposure groups are collections of exposures that have the same optical parameters. You can use exposure groups for multiple purposes including: \* A new exposure group per grid \* A new exposure group for each beam-shift position in a template \* A new exposure group for different squares on a grid \* A new exposure group for a new data collection session, perhaps on a different day After enabling an exposure group, you can choose to continuously listen to the specified directory and filename wildcard filter for new exposures, or ignore the group if you no longer want to include a set of exposures for processing. {% endhint %} !\[\](/files/-MNiwO6liZwbhArYTJWi) ➡️ \*\*Step 1: Edit Parameters for Exposure Group 1\*\* \* \`Enable continuous import\`: Toggling this on will allow new exposures added to the directory, to be processed as they are found. Be default, this is enabled. If you are creating more than one exposure group and the initial exposure groups are not expected to contain any new images that Live has not already found, you can disable continuous import from the older exposure groups to save disk operations needed to check for new files periodically. \* (Optional) \`Ignore exposures from group\`: This is useful only if you have multiple Exposure Groups configured, and you would like to ignore all exposures from a particular Group. This might be used if you find that all the images from a group are of poor quality, for example. \* \`Directory to watch\`: The file system location where the exposures are being written or are already saved. This is a directory on the filesystem, \*\*not\*\* a wildcard path. \* (Optional) \`File name wildcard filter\`: You can optionally filter by wildcard to select files within the Directory to watch, that match a specific extension, e.g., \`\*.mrc\`. This is important if multiple types of files will be saved in the same directory. \* (Optional) \`Search recursively\`: This option will traverse all subdirectories within the directory specified in \`Directory to watch\` that matches the \`File name wildcard filter\` value. This can be helpful if your data collection software writes files to multiple sub-folders, for example. Leaving this option turned off will tell the file engine to only search for files matching the \`File name wildcard filter\` value in the top level directory. If using Search Recursively, then in \`Directory to watch\`, ensure you choose the highest-level folder to which movies are being written for the data collection session. For example, for EPU, specifying \`../Images\_Disc1/Data\` as the \`Directory to watch\` value and \`FoilHole\_\*\_Data\_\*\_\*\_\*\_\*.mrc\` for the filter value when \`Search Recursively\` is enabled allows you to get around creating an exposure group for every grid square. \* \`Gain Reference Path\`: Enter or select an absolute path to the gain reference, if the gain reference file is available. The gain reference should be in \`.mrc\` or \`.gain\` (for EER data) format. \* \`Defect File Path\`: Enter or select an absolute path to the defect file, if the defect file is available. Note that zeros in the gain reference are also treated as defects and corrected in the same way. To add a new Exposure Group, click \*\*'New'\*\*, ➡️ \*\*Step 2: Click Enable (i.e., Save) for each Exposure Group added\*\* You must click Enable for each newly added Exposure Group. This saves the parameters you entered. Once the exposure group is enabled, its parameters are locked in and cannot be changed unless the Session is cleared. For more information on clearing a session, please see: \[Live Jobs and Session-Level Functions\](/live/live-jobs-and-session-level-functions.md) !\[\](/files/-MNiwRmPwcWkDKxQop\_b) ### Load Configuration Profile {% hint style="warning" %} CryoSPARC Live \[configuration profiles\](https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide#load-configuration-profile) are not backwards compatible when created in v5.0+: profiles created in previous versions will be retained when updating to v5.0, but new profiles created in v5.0 will be dropped if downgrading to v4.7 or older. {% endhint %} To speed up the configuration a new Live session, you can save all or some of the parameters of an existing session into a configuration profile and apply the profile to future sessions. Configuration profiles can store: \* Compute resource preferences \* Exposure groups \* Parameter sections such as microscope params #### \*\*Creating a profile\*\* In any Live session that has been configured with all required parameters, you can choose to save a subset or all of that configuration as a profile. ➡️ Navigate to the 'Configuration' section. ➡️ Click the Profiles dropdown, then 'Save current configuration'. ➡️ Choose a title, and one or more sections to save to that profile. !\[\](/files/-MNiwqtlQojPywqtfxKW) #### \*\*Applying a profile\*\* ➡️ Navigate to the 'Configuration' section. ➡️ Click the Profiles button to see a list of saved profiles. !\[\](/files/-MNiwvGaHXkWRB3lADOs) ➡️Click \*\*View and Apply\*\* to preview. If a profile contains exposure groups, it will create a new exposure group in the session. ➡️Click \*\*Apply Profile\*\* to load the profile into the current Session. !\[\](/files/-MNiwyS7vQwcEnaOzLk\_) ## 3. Start Session and View Exposures > As the session progresses, view diagnostic plots and metadata for each exposure. Start Manual Picking anytime, and optionally Reject or Reprocess individual exposures. Once you have configured and saved all required parameters and exposure groups, the Session can be started anytime. !\[\](/files/-MNix15bp3Hdm6nGKGrw) ➡️\*\*Step 1: Start Session\*\* ➡️Click \*\*Start Session\*\* (from the Configuration Tab, or from the Header). !\[\](/files/-MNix44qV45j1m3tHBxO) As exposures are captured, detected, and loaded into CryoSPARC Live, a visualization/thumbnail for each appears sequentially from left to right in the top bar. Near the bottom of each thumbnail, a blue progress bar indicates the status of pre-processing (motion correction, CTF estimation, picking, extraction, etc). Rejected exposures are indicated with a red “X”. Exposures are processed in they order they are found, as follows: patch motion correction (global and local) > patch CTF estimation > particle picking > particle extraction. Exposures being processed are indicated with a flashing blue outline and a blue progress indicator labelled with the current processing stage (e.g., Motion Correction). !\[\](/files/-MNix73nO4PLE2ZBjAQH) Once a session is started, you can see sessions statistics at a glance in the sidebar on the left. This includes the number of exposures found, processed, accepted, rejected, and the rates at which exposures are being found and processed. !\[\](/files/-MNixDtmu\_p8FIDC4QXf) ➡️\*\*Step 2: View Individual Exposures information\*\* Clicking on the \*\*Individual Exposure Tab\*\* will bring up diagnostic plots for the exposure that is currently selected in the Exposure Feed. !\[\](/files/-MNixGnlDMcM6c-8jVwk) You can traverse the Exposure Feed by clicking on a thumbnail or using the navigation buttons at the top. The arrows allow for traversing through exposures including the ability to follow the latest exposure, or navigate to a specific exposure using its ID. !\[\](/files/-MNixLM6i--SPbL5NXPA) {% hint style="info" %} \*\*Clicking on the |<-- button will follow the latest incoming exposure.\*\* {% endhint %} On the right side of the \*\*Individual Exposure Tab\*\* is the Exposure Viewer. You can adjust the Low Pass Filter (LP Filter) slider (top right of the exposure canvas) as required. Buttons are available in the bottom left for zooming, panning and resetting the view. If an exposure has failed processing for any reason, the corresponding \*\*traceback/error message\*\* will be displayed in the Individual Exposure Tab. ➡️\*\*Step 3: Reject individual exposures and/or reset failed exposures\*\* If you wish to \*\*Reject\*\* an individual exposure, click on the dropdown menu above the Exposure Viewer and click \*\*'Reject'\*\*. Alternatively, use the keyboard shortcut by pressing \*\*"R"\*\* on the keyboard. A red \*\*'M'\*\* (for manually rejected) will appear on the thumbnail of any manually rejected exposure. !\[\](/files/-MNixOktjbR9M65xI83x) To un-reject an individual exposure, or to reset an exposure that has failed, click \*\*'Unreject'\*\* or \*\*'Reprocess exposure'\*\* from the same dropdown menu. !\[\](/files/-MNixSFB8TxlLiZ3RRr2) \*\*➡️Step 4: Modify Exposure Processing Priority\*\* The priority in which exposures are processed by the preprocessing GPU worker(s) can be modified at any time. For more information, see the following section: {% content-ref url="/pages/-MNipixNnwl\\\_RERAifqp" %} \[Live Jobs and Session-Level Functions\](/live/live-jobs-and-session-level-functions.md) {% endcontent-ref %} ## 4. Exclude Poor Quality Exposures from Downstream Processing > The \*\*Overview Tab\*\* contains time-series plots depicting the evolution of computed data attributes. Use these to optionally exclude groups of exposures based on various thresholds. The \[\*\*Overview Tab\*\*\](/live/ui-overview.md#overview-tab) contains a number of plots that are useful for assessing the quality of processed images. You can hover over any individual exposure (dot) on any graph to view a small thumbnail and other details of the exposure. Clicking on any dot will cause the selected exposure to be displayed in the exposure viewer and selected in the feed. !\[\](/files/-MNixWrKnCztzTg6S-XA) Use the threshold sliders or the input fields to optionally exclude groups of exposures that do not meet desired criteria. For example, you may wish to exclude exposures with a calculated CTF Fit (Å) value greater than 4Å from downstream processing in Live (i.e., further particle picking, inclusion of the corresponding particles in 2D classification and refinement). The exposures that are rejected in this way will still be available for later downstream processing in CryoSPARC or for export - they are not deleted. {% hint style="info" %} You can apply, adjust and clear thresholds \*\*at any time\*\* during a Live Session. Particles from accepted micrographs will be used for downstream 2D and 3D processing. If a micrograph is rejected, its particles will be removed from consideration downstream and outputs that depend on particles such as the 3D refinement density map and FSC curves will be re-calculated at the next update. {% endhint %} ➡️There are two ways to adjust a threshold on a particular attribute of micrographs.\\ \*\*Graphically\*\*: Click on \*\*'Select Thresholds'\*\* in the bar above the overview plots to set the plot mode to selection. Then, click and drag over a vertical selection of the plot for the attribute you wish to filter, to select a range that includes the data you wish to \*\*keep\*\*. When you release the mouse button after dragging, you will see your selection appear on the graph as a \*\*green\*\* highlight, and the threshold slider above the attribute will change to the new values you have selected.\\ \*\*Manually\*\*: Use the threshold sliders or input boxes to enter the exact values you wish to use for filtering for a given attribute. You will see a highlighted region appear on the graph in \*\*green\*\* corresponding with the slider position. !\[\](/files/-MNix\_ZztLAMCidejc4R) ➡️Once you see a region highlighted in \*\*green\*\*, you can finalize your selection by clicking \*\*'Set Threshold'\*\* for that attribute. Otherwise you can click 'Cancel'. Once you set the threshold, all existing micrographs outside the set range will be \*\*threshold rejected\*\* as will any new micrographs that are processed. !\[\](/files/-MNixcLuzE2a\_5tqoTTl) In the exposure feed, rejected exposures will have a reject icon. You can apply thresholds on multiple attributes, e.g., on both CTF Fit (Å) and on Defocus Range (Å). Exposures that are rejected for any reason will show as red dots in all graphs. !\[\](/files/-MNixhQH7UI\_Cjs2xhZu) ➡️\*\*To clear thresholds\*\*, click \*\*'Clear'\*\* on any attribute and this will reset the exposures that were previously rejected. It is possible to both manually reject and threshold reject an exposure. ## 5. Browse and Download Exposure Stats > The \*\*Browse Tab\*\* contains various exposure statistics that can be plotted, compared and downloaded. At any time during a session, you can view exposure statistics, filter values and download a \`.csv\` file containing all of the data. ➡️Navigate to the \[\*\*Browse Tab\*\*\](/live/ui-overview.md#browse-tab) to view statistics and scatter plots comparing any two attributes. !\[\](/files/-MNixkuWDu4stdQEG-5p) ## 6. Fine-tune Particle Picking > Perform manual, blob or template-based particle picking from the \*\*Picking Tab\*\* as the session progresses, with the option to set new particle score thresholds and re-extract particles during preprocessing, which will be fed into downstream steps. ### Default: Blob Picker The Blob Picker is the default picker enabled in CryoSPARC Live for a new session. The CryoSPARC Live GPU Workers will perform blob-based picking and extraction on all incoming movies using the \`Minimum particle diameter (A)\`, \`Maximum particle diameter (A)\` and \`Extraction box size (pix)\` that you provided on when configuring the Session, until and unless Blob Picker settings are adjusted or the active picker is changed. ➡️\*\*Step 1: View Blob Picks\*\* Blob picks \*\*"B"\*\* are displayed in yellow in the exposure viewer. Picking statistics are displayed on the Blob Picker button and can also be viewed by clicking on the dropdown. !\[\](/files/-MNixrr\_sCRxufyjSLZd) {% hint style="info" %} To hide or view blob picks, click on the Blob Picker button at the top of the Exposure Viewer. By default, CryoSPARC Live displays pick locations as dots. \*\*Shift + Click\*\* on the Blob Picker button at the top of the Exposure Viewer to cycle through circular, square and dot pick markers. You can also adjust how picks are displayed using the expansion menu on the right of the picker buttons. \*\*For a full list of keyboard shortcuts, click on the Main Menu in the top left corner of any Session.\*\* {% endhint %} ➡️\*\*Step 2: Adjust Blob Picking Parameters\*\* As more exposures are processed, you may wish to adjust Blob Picker parameters to obtain better picks. !\[\](/files/-MNixuzWjh2ZneWpYhTM) You can adjust the \`Minimum particle diameter (A)\`, \`Maximum particle diameter (A)\`, blob shape (circular, elliptical, blob, or a combination of them), \`Lowpass filter to apply (A)\` and the \`Min. separation distance (diameters)\` at any time during a session. ➡️Adjust the value(s) as desired and then click \*\*Activate for All\*\* or \*\*Activate for Future\*\* in order to trigger re-picking and re-extraction based on your changes. Or, use Test Adjustments, explained next. \* \`Activate for All\`: Applies the thresholds and. any changed parameters to all exposures (i.e., will cause all exposures to be re-processed with the changes) \* \`Activate for Future\`: Applies the thresholds and any changed parameters to all exposures that have not yet been processed. Useful if you do not want your changes to trigger reprocessing. ➡️\*\*Step 3: Test Adjustments (Optional)\*\* If you are not sure how your parameter changes might affect processing or if you would like to experiment, you can use \*\*'Test Adjustments'\*\* which will cause only the active (currently selected) exposure to be re-processed with the new picking parameter changes, and re-extracted. !\[\](/files/-MNixyGpjJ3ZrkReNGYt) After clicking 'Test Adjustments' you may have to wait a few seconds or minutes until one of the CryoSPARC Live GPU Workers becomes available to pick up the test micrograph. Once this process is complete, the new pick locations will appear on the active micrograph. Exposures to which a \*\*Test\*\* parameter has been applied are indicated with a purple "\*\*T"\*\* on their respective thumbnails. You can apply '\*\*Test Adjustments'\*\* on as many individual exposures as you like. !\[\](/files/-MNiy13076zb\_8gjYb-Q) ➡️Once satisfied with the new picker settings, click \*\*'Activate for All'\*\* or '\*\*Activate for Future'\*\* as desired. This will trigger re-picking and re-extraction. Unless one of the \*\*Activate\*\* buttons is clicked following \*\*Test\*\*, the exposure on which \*\*Test\*\* was run, will simply be excluded from any further processing (i.e., from particle extraction and steps downstream). To undo \*\*Test\*\* mode on a particular exposure, i.e., to reset it so that it can be included in further processing, click on the dropdown above the Exposure Viewer and click Reprocess exposure. ➡️\*\*Step 4: Filter and Extract Blob Picks\*\* Along with picker parameters, you will need to adjust the Normalized Cross-Correlation (NCC) and Power Score thresholds, similar to \`Inspect Picks\` in CryoSPARC, to adjust and exclude poor quality picks. !\[\](/files/-MNiyAqoxwQfmBQGy1mv) ➡️Adjust the thresholds or input new values as desired and then click \*\*Confirm Thresholds\*\* or Cancel if you wish to revert to the previous setting. As you drag the sliders, you will see picks appear and disappear in real time on the exposure viewer. Typically, you will want to adjust the NCC and power sliders until only "true" particles visually remain. For the Blob Picker, the power scores are often most useful. Removing low-power picks will exclude picks in empty patches of ice, smaller contaminants, and broken particles. Removing high-power picks will exclude carbon edges, ice crystals, gold particles, overlapping particles, etc. ➡️Click \*\*Extract for All\*\* or \*\*Extract for Future\*\* to trigger re-extraction. The extraction step will attempt to extract all particles falling within the new thresholds, and will remove particles that are too close to the edges of the exposures. {% hint style="info" %} Note that if you make changes to 'Adjust Blob Picker' parameters and 'Filter and Extract Blob Picks' extraction thresholds, \*\*both\*\* 'Activate for All' and 'Extract for All' buttons will become available. Clicking \*\*either\*\* button will cause \*\*both\*\* new settings to be applied. {% endhint %} ### Manual Picker The Manual Picker can be engaged anytime during a Session. To pick particles, navigate to the Exposure Viewer, click the "arrow+" button in the bottom right and begin selecting particles by left-clicking. To remove manual picks, either right-click, or click the "arrow-" button and then left-click over the picks you wish to remove. Manual picks are displayed in a table on the Manual Picker tab. Clicking on a table row will navigate to the exposure indicated. !\[\](/files/-MNiyENeBn2CnNdsgGdO) ➡️To extract Manual Picks, click \*\*Start Manual Extraction\*\* on the Manual Picker tab. These picks can be fed into the Template Picker (by generating templates, below), or simply exported. ### Template Picker The Template Picker can be engaged anytime after starting a Session. To do so, you can either choose to generate templates from available blob picks or manual picks in the session, or load in available templates from any existing CryoSPARC Project/Job. ➡️\*\*Step 1: Create or Load Templates\*\* You can either generate templates using \*\*2D Classification from Latest Exposures\*\* or \*\*Load Existing Templates\*\*. !\[\](/files/-MNiyIOYWD0-\_-vE9EEW) ➡️To \*\*generate templates\*\*, specify: \* Classes (number of classes for 2D classification) \* Exposures (particles from this many exposures will be included) \* Classify (whether to classify the existing Blob Picks, Template Picks or Manual Picks) ➡️Click Generate Templates, which will start a 2D Classification job on the Auxiliary Lane that will classify the particles. Once complete, you will be able to select the desired classes below to use as templates for the Template Picker. ➡️To \*\*Load Templates\*\* that you already have available, enter the CryoSPARC Project ID and Job ID corresponding to the available particles/templates. These will be loaded and then they can be selected. If you have already started Streaming 2D Classification in CryoSPARC Live (eg using blob picks) you can enter the project ID and job ID of the streaming 2D class job to use the current templates to instantiate the template picker. ➡️\*\*Step 2: Select Templates\*\* Select as many templates as desired by clicking on each class. !\[\](/files/-MNiyMPhnzahEo6tpZUU) ➡️\*\*Step 3: Adjust Template Picker\*\* ➡️Enter (or adjust) the Particle Diameter (A) \*\*(required)\*\*. If desired, adjust the Lowpass filter to apply (A), Ang. sampling (degrees) and Min. separation dist (diameters). The particle diameter should be set equal or larger than the longest expected dimension of the particle. You can Test Adjustments if desired, and then click \*\*Apply to All\*\* or \*\*Apply to Future\*\* to trigger template-based picking using your parameters. !\[\](/files/-MNiyR55v7jIuqplINfb) ➡️\*\*Step 4: Filter and Extract Templates\*\* You will need to adjust the Normalized Cross-Correlation (NCC) and Power Score threshholds, similar to \`Inspect Picks\` in CryoSPARC, to adjust and exclude poor quality picks. See the \[similar section for Blob picking here.\](#default-blob-picker) !\[\](/files/-MNiyVElfDTJ0Z\_qWX8y) ➡️Adjust the thresholds or input new values as desired and then click \*\*Confirm Thresholds\*\* or Cancel if you wish to revert to the previous setting. ➡️Click \*\*Extract for All\*\* or \*\*Extract for Future\*\* to trigger re-extraction. The extraction step will attempt to extract all particles falling within the new thresholds, and will remove particles that are too close to the edges of the exposures. ## 7. Start Streaming 2D Classification > 2D classification is performed seamlessly while a session is in progress. Newly available particles can be classified into existing classes as they are extracted, or 2D classification can be re-started at any time. You can start Streaming 2D Classification at any time once you have started the session. You may wish to start 2D Classification early on into a session to see more about the visual quality of the particles extracted thus far and make choices about tuning picking parameters or about the sample itself. !\[\](/files/-MNiyaeorCTveI8s6Eo-) ➡️\*\*Step 1: Start Streaming 2D Classification\*\* ➡️On the 2D Classification Tab, click the \*\*Gear\*\* icon and enter the number of Classes. \* Alternatively, you can \*\*Build with Custom Parameters\*\* by clicking on the "hammer" button. This will create a new Streaming 2D Classification job in the CryoSPARC Project where the Live Session is housed. \* \*\*Navigate to the CryoSPARC Project\*\* and find the new job (set to Building status). Enter any custom parameters you wish to change in the Job Builder. \*\*DO NOT launch the job from the regular CryoSPARC interface.\*\* \* \*\*Return to the CryoSPARC Live interface > 2D Classification Tab\*\* and click Queue to launch the job. ➡️Click Start. The job will start and take in all the particles that have been extracted so far. The most recently extracted set of particles from each micrograph will be used. Therefore if a micrograph was originally picked with the blob picker, then re-picked with the template picker, the template picked particles will be used. ➡️\*\*Step 2: Select 2D Classes\*\* !\[\](/files/-MNiyeGEaFhXt6A8IUIC) Once the first 20 iterations of classification are complete, the job will display interactive 2D classes that can be selected. These displayed classes will update as new particles arrive. ➡️Click to select the desired class averages. Use the buttons above the class averages to sort and select, or right click on any class average to display a menu of actions. All particles falling within the selected class averages will be used for Ab-Initio Reconstruction and Streaming Refinement. !\[\](/files/-MNiyiP5UYs3fMiBAQlx) {% hint style="info" %} \*\*2D Classes will continue to update with new particles every few minutes, as new particles arrive.\*\* The rate of updating depends on how fast new particles are coming and, and how long it takes to update classes. Once 10,000 new particles are seen, Streaming 2D classification will also go back and re-classify all existing particles into the updated classes, which can take several minutes. Since the 2D templates that are resolved will change only slowly as new particles are seen, the selection of 2D classes that are made will persist over updates of streaming 2D classification. However, it can be a good idea to return to the streaming 2D class tab periodically to check if any classes should be newly selected or unselected. While waiting for new particles, the Streaming 2D Classification job will enter \`Waiting\` status. {% endhint %} ➡️\*\*Step 3: Stop, Re-Start or Attempt Resume (Optional)\*\* !\[\](/files/-MNiymJcEq9w7v3py1Uc) \*\*Stop\*\* At any time during 2D Classification, you can also choose to \*\*Stop\*\* the job. This will kill the Streaming 2D Classification job. If you wish to start another Streaming 2D job from scratch, you will need to configure the number of classes and click \*\*Start\*\*. \*\*Re-Start\*\* After streaming 2D class has been stopped or killed, you can "force" Re-Start 2D Classification. This will start off 2D classification from scratch without reference to previous results. \*\*Attempt Resume\*\* Alternatively, if the number of classes has not changed, you can attempt to resume streaming 2D classification from the previous results that are currently displayed. This will work if the previous 2D class job was killed or stopped after writing out some results, and if the particle box size, pixel size, and number of classes has not changed. If resuming fails, 2D class will start from scratch. ## 8. Start Ab-Initio Reconstruction > On the \*\*Ab-Initio Tab\*\*, generate an initial model from available particles or load an initial model to use for refinement. Once you have run Streaming 2D Classification and selected some classes, the particles falling into those class averages are available for Ab-Initio Reconstruction. !\[\](/files/-MNiyqWXPMTDTSi6gMgn) ➡️\*\*Step 1: Configure Ab-Initio Reconstruction or Load Volume\*\* ➡️Click on the \*\*Gear\*\* icon to configure Ab-Initio Reconstruction. !\[\](/files/-MNiyuIvORTyJS64q9MH) ➡️Enter the \`Number of Classes\` if you wish to resolve more than 1 class, \`Symmetry\` (optional, we recommend using the default C1 as it is not necessary/recommended to enforce symmetry during ab-initio reconstruction), and the number of \`Particles\` you wish to use for Ab-Initio (also optional, default 100,000). Note that if you use multiple classes at this stage, you can select one as the initial model for streaming 3D refinement, \*\*but all particles will be used for refinement\*\*. Streaming heterogeneous refinement is not currently available in Live. \* Alternatively, you can \*\*Build with Custom Parameters\*\* by clicking on the "hammer" button. This will create a new Ab-Initio Reconstruction job in the CryoSPARC Project where the Live Session is housed. \* \*\*Navigate to the CryoSPARC Project\*\* and find the new job (set to Building status). Enter any custom parameters you wish to change in the Job Builder. \*\*DO NOT launch the job from the regular CryoSPARC interface.\*\* \* \*\*Return to the CryoSPARC Live interface > Ab-Initio Tab\*\* and click Queue to launch the job. ➡️Click \*\*Queue\*\* to launch the job. Volume slices will display in the CryoSPARC Live interface and the 3D volume will be available to view in the Volume Viewer as the reconstruction progresses. !\[\](/files/-MNiyz6nQfxwuEZIT\_Ex) !\[\](/files/-MNiz10tvyieGrbnhyDH) You can also view the progress of any job in Live by clicking on the job number in the sidebar or on the relevant tab, to expand the streamlog view. For example, we clicked into the Ab-Initio job from the sidebar: !\[\](/files/-MNiz4-hgRRUuURhkVhb) \*\*Alternatively, you can load an existing volume by entering the Project UID and Job UID corresponding to the location of the initial model in your CryoSPARC instance, and then click Load.\*\* {% hint style="info" %} The volume viewer can be rotated and zoomed by dragging with the left mouse button or scrolling respectively. Holding shift and dragging with the left mouse will pan the view. {% endhint %} !\[\](/files/-MNiz9AJDbwh\_X5RgwWc) To download the volume, click 'Download map' on the bottom right hand corner of the Volume Viewer. Alternatively, you can find the volume in the CryoSPARC Project where the Live Session is housed. ➡️\*\*Step 2: Select a Volume for Refinement\*\* ➡️Select one volume from Ab-Initio to use for Streaming Refinement. Click on the volume slices image to select the class. !\[\](/files/-MNizE8AFTh37i4bKtXq) ## 9. Start Streaming Refinement > Refinement in CryoSPARC Live operates in a streaming manner, taking into account new particles that become available after preprocessing. Once the Ab-Initio job has been completed (or a volume has been loaded) and one volume has been selected in the Ab-Initio Tab, you can start a Streaming Refinement job. ➡️\*\*Step 1: Configure Streaming Refinement\*\* ➡️Navigate to the Refinement Tab. Click on the \*\*Gear\*\* icon to configure refinement parameters. !\[\](/files/-MNizHtiqQZ4Q0bet8iF) ➡️Specify the Symmetry for refinement if known/required. Initial volumes will be automatically aligned to the symmetry axes if symmetry is specified. \* Alternatively, you can \*\*Build with Custom Parameters\*\* by clicking on the "hammer" button. This will create a new Streaming Refinement job in the CryoSPARC Project where the Live Session is housed. \* \*\*Navigate to the CryoSPARC Project\*\* and find the new job (set to Building status). Enter any custom parameters you wish to change in the Job Builder. \*\*DO NOT launch the job from the regular CryoSPARC interface.\*\* \* \*\*Return to the CryoSPARC Live interface > Refinement Tab\*\* and click Queue to launch the job. ➡️Click \*\*Queue\*\* to launch the job. Various plots will display in the CryoSPARC Live interface and the refined 3D volume will be available to view in the Volume Viewer as the reconstruction progresses. !\[\](/files/-MNizM6LHs0IkdDYXlfC) !\[\](/files/-MNizOrolXSS-fFML6mo) !\[\](/files/-MNizTKeSMyXPWDA\_kTq) To download the volume, click 'Download map' on the bottom right hand corner of the Volume Viewer. Alternatively, you can find the volume in the CryoSPARC Project where the Live Session is housed. {% hint style="info" %} \*\*As new particles are picked, extracted and classified, they will be picked up by the Streaming Refinement job so that the refinement volume will update over time in the Volume Viewer.\*\* Streaming 3D refinement will update the 3D map, FSC resolution estimate, and diagnostic plots repeatedly as new images are collected. When sufficient new particles are available that pass the previous filtering stages and 2D class selection, refinement will backtrack to a lower resolution map, and re-perform refinement of all particles until convergence. The rate of updating will depend on the number of particles already collected and the current resolution. While waiting for new particles, the Streaming Refinement job will enter \`Waiting\` status. {% endhint %} ## 10. End the Session > What to do once you have finished processing in Live. ### Pause Session and Auto Pause Once you have completed processing in CryoSPARC Live for the day/dataset, you can pause the session to free up compute resources by clicking \*\*Pause Session\*\* in the header. Pausing will retain all Session configuration and parameters and all results. Running preprocessing workers will be allowed to finish their current exposure and then gracefully exit. Running 2D or 3D streaming jobs will be killed and marked as completed so that their latest results become available to use for further processing in CryoSPARC or resuming in CryoSPARC Live. !\[\](/files/-MNizZ6mDpz9cj8k\_5Ni) #### Auto Pause (v5.0+) You can also configure the session to automatically pause if no new exposures are found or processed for a specified amount of time. This will free up the compute resources in use by the session for use in other jobs or sessions automatically. Two modes of Auto Pause are available: \* Standard: Pause the session immediately after the timeout has expired. This will kill any running 2D/3D streaming jobs. \* Graceful: After the idle timeout has lapsed, also wait for 2D/3D streaming jobs to finish before pausing the session. ![](https://guide.cryosparc.com/files/fdXG8hukZAA9tOVJnfz6) In combination with the \[\`Delay worker startup until ready\`\](#run-configuration) option (also new in v5.0+), auto pause allows you to create and start multiple Live sessions at the same time, each pointing to a different on-disk directory where microscope control software will write out new images as they are acquired. For example, in a multi-grid acquisition setting, one session can be created for each grid that will be imaged. All sessions can be started, but none will consume hardware resources until data appears in their respective directory. At that point, the session will automatically queue Live workers to process the data. Once new data stops being added to the directory, the session will automatically pause, freeing up resources for the next session to process its data. ### Mark as Complete If you have finished with a Session and do not plan to return to it, you may wish to \*\*Mark as Complete\*\* from the header. This will set the session to completed status and separate it from running and paused sessions in the browse sessions page. A completed session can still be started again and no results are lost. For more information on Session-Level Functions, please see: \[Live Jobs and Session-Level Functions\](/live/live-jobs-and-session-level-functions.md) ## 11. View/interact with Outputs and Perform Further Processing > Interacting with outputs and performing advanced processing. ### Continue Processing Directly For any Session, you can navigate into the CryoSPARC Project where the Live Session is housed and use outputs from Streaming 2D Classification or Streaming Refinement (e.g., particles, volumes, templates, etc), directly for further processing. Using particles from Streaming 2D classification will allow using the 2D alignments and assignments downstream (e.g., for select 2D or re-centering extraction). Using the particles from Streaming Refinement will allow using the 3D alignments downstream (e.g., for reconstruction, 3D Variability, CTF refinement, local refinement, etc). !\[\](/files/-MNizaqwM0OJtlXUJdls) {% hint style="info" %} Note that if you create a new job in CryoSPARC and use the outputs of a Live streaming job that is still running or waiting as input, the new job will remain Queued until the streaming job enters completed status. You can force this to happen by stopping the streaming job from the Live interface. The streaming job will be killed and marked as completed. You can then start streaming processing again from Live and this will create a new streaming job. {% endhint %} ### Export Exposures and/or Particles At any time during a Session, you can navigate to the \*\*Details\*\* Tab and click \*\*Export Exposures\*\* or \*\*Export Particles\*\* to cause the available exposures and/or particles to be made available in CryoSPARC. !\[\](/files/-MNizeP71GlNh7XOj\_Q9) No data is copied in this process. Rather, you will see a new job or jobs appear in the workspace in CryoSPARC corresponding to your Live session. !\[\](/files/-MNizhY4hvAcYhHnCBvL) These jobs will have outputs pointing to the data from CryoSPARC Live, and can be used as any other CryoSPARC job output for connecting to new jobs for more processing. \* \`CryoSPARC Live Exposure Export\` Job: This job will run in the CryoSPARC Project where the Live Session is housed and will separately output accepted, rejected, and manually rejected exposures for each Exposure Group in the Session. \* \`CryoSPARC Live Particle Export\` Job: This job will export all particles from the Live session that pass the threshold tests in picking and have been extracted. These are the same particles that would be seen by streaming 2D classification. These particles will not come with alignment or class assignment information, but do contain \`location\`, \`pick\_stats\`, \`blob\` and \`ctf\` outputs. You can continue processing in CryoSPARC from the outputs of the above jobs. !\[\](/files/-MNizlE5IHLDe2aMaHy1) ## 12. Record Notes and View Session Details > The \*\*Details Tab\*\* contains session history and a handy notes features. The Details Tab contains information about the user, session directory, session-level functions and start/pause history for the session along with a notes feature that includes checklists. !\[\](/files/-MNizoEFBeLcPfTabYhr) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/live/new-live-session-start-to-finish-guide.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import.md). # Tutorial: EPU AFIS Beam Shift Import ## Introduction Thermo Fisher’s \[EPU data collection software\](https://www.thermofisher.com/ca/en/home/electron-microscopy/products/software-em-3d-vis/epu-software.html) is a commonly used data acquisition software for single particle analysis. Typically, SPA data has been collected via manually moving the stage around, placing different regions within the hole at the optical center of the microscope. This is done in order to avoid the strong aberrations that result from off-axis use of the objective lens. However, collecting data in this manner induces delays from moving the stage and waiting for the stage to settle. Thus, advances such as \[EPU’s Aberration-Free Image Shift (AFIS) collection mode\](https://assets.thermofisher.com/TFS-Assets/MSD/posters/MM2019-poster-advances-SPA-data-acquisition.pdf) have allowed for targeting multiple holes without stage movement in-between each hole. Importantly, AFIS and associated microscope calibration service allow for targeting holes that don’t lie at the optical center of the microscope \*\*without\*\* inducing severe artefacts, and this significantly speeds up data collection. For many datasets collected via AFIS mode, it is still worthwhile to estimate residual higher-order aberrations such as coma via the Global CTF Refinement job: if there are any residual aberrations, correcting them may lead to improved structures. However, doing so requires grouping movies into subsets (\[Exposure Groups\](/processing-data/all-job-types-in-cryosparc/ctf-refinement/job-exposure-group-utilities.md)) based on similar optical conditions, which includes the amount of applied beam shift. Since refinement of higher-order CTF aberrations is done separately for each exposure group, the assignment of movies into exposure groups can have significant impacts on the aberration values, which will impact the resolution achieved by subsequent refinements. In CryoSPARC v4.4+, we have integrated the import of beam shift values from EPU sessions collected in AFIS (Aberration-free Image Shift) mode to allow for Exposure Group assignments based on applied beam shift. The following tutorial covers: \* how to import movies/micrographs with beam shift values \* how to assign movies/micrographs into exposure groups based on beam shift \* merging beam shift values into pre-v4.4 movie/micrograph datasets, without re-processing from scratch \* continuing processing data from live ## Use case #1: Clustering movies into exposure groups at import time {% hint style="info" %} This use case covers the situation where dataset processing starting in CryoSPARC from scratch, i.e., all processing steps post-motion correction (including particle picking) have not been done yet. For existing CryoSPARC exposure datasets, datasets processed in CryoSPARC Live, or datasets with existing particles, please refer to the subsequent use case #2. {% endhint %} ### Import Movies or Import Micrographs For movie or micrograph datasets collected via EPU’s AFIS mode, beam shift values can now be imported along with other metadata. As an example, here is a screenshot of an output data directory from a data collection session using EPU. Note that the movies we would like to import are in \`.eer\` format, and the associated files containing the beam shifts are in \`.xml\` format. Each EER movie has a corresponding \`xml\` file. ![](https://guide.cryosparc.com/files/JrI3f5uew85UHubNiFk5) In CryoSPARC’s Import Movies and Import Micrographs jobs, you will now notice an “XML Import” section that allows specification of an absolute path wildcard to the directory containing the XML files. This path is in addition to the wildcard expression pointing to the raw movie files. For the above example, we have set the two wildcard expressions to the following: \* Movies data path: \`/bulk9/data/EPU\_apof\_JLR/20230212a\_T64/Images-Disc1/GridSquare\_11564642/Data/\*.eer\` \* EPU XML metadata path: \`/bulk9/data/EPU\_apof\_JLR/20230212a\_T64/Images-Disc1/GridSquare\_11564642/Data/\*.xml\` Here they are the same paths with different file extension filters. There are 4 additional parameters that assist in finding correspondences between the \`.eer\` and \`.xml\` files. These parameters specify the number of characters to cut off of the beginning and end of the movie and XML filenames in order to match them to each other, one-to-one: \* \*\*Length of movie filename prefix to cut for XML correspondence:\*\* Use this field to specify the number of characters to cut off the prefix of the imported movie filename, to match with the XML filename. \* \*\*Length of movie filename suffix to cut for XML correspondence:\*\* Use this field to specify the number of characters to cut off the suffix of the imported movie filename, to match with the XML filename. \* \*\*Length of XML filename prefix to cut for movie correspondence:\*\* Use this field to specify the number of characters to cut off the prefix of the XML filename, to match with the imported movie filename. \* \*\*Length of XML filename suffix to cut for movie correspondence:\*\* Use this field to specify the number of characters to cut off the suffix of the XML filename, to match with the imported movie filename. ![](https://guide.cryosparc.com/files/C56EvJrl1Rh9a0blc2rV) Example illustrating how to determine the number of characters to cut for the movie and XML filenames. In this case, we need to trim the eight \`\_EER.eer\` characters off of the end of the movie filename, as well as the four \`.xml\` characters off of the XML filenames. Thus, we’ll set the movie suffix parameter to 8, the XML suffix parameter to 4, and leave the rest empty. After inputting the parameters and running the job, a scatter plot with the beam shift values will be displayed in the event log if the XML import was successful. The event log will also print an example of the movie and XML paths after applying the prefix/suffix trim, in order to ensure that these are aligned and match in structure. If any of the XML files are absent, corrupt, or missing beam shift values, they will be flagged as having missing beam shifts; in the example image, two exposures are missing beam shift values. \*\*Be sure to check the event log to see if the majority of exposures had successfully read beam shift values\*\*; if this is not the case, a warning will be displayed in orange highlight. ![](https://guide.cryosparc.com/files/IUEpeDEllwAYIBNyjYlp) \### Pre-processing Next, exposures must be pre-processed via motion correction (applicable to movies) and CTF estimation. CTF estimation is required to cluster exposures by the applied beam shift. The recommended motion correction job is Patch Motion Correction, and the recommended CTF estimation job is Patch CTF Estimation. ### Clustering via Exposure Group Utilities The next step is to cluster exposures into groups based on the applied beam shift. The main purpose of clustering is to ensure that exposures with similar beam shift values are placed into the same exposure group. This can be done via running Exposure Group Utilities in the \`cluster&split\` mode. ![](https://guide.cryosparc.com/files/j6SCv7mgn8e5LiDJuBWz) Example of a tree view for the workflow (starting from importing micrographs) until exposure group utilities. First, connect the outputted exposures from the Import Movies or Import Micrograph job above. Then, specify the “Input Selection” as \`exposure\`, and the “Action” as \`cluster&split\`. Finally, set the number of clusters. In this case, based on the beam tilt scatter plot above, we counted 61 clusters, which correspond to the 61 unique “rings” (each ring corresponding to 8 different collection sites arranged in a circle on one hole). Note that it is not necessary to ensure that the number of clusters matches the number of holes precisely. Indeed, depending on the layout and orientation of holes on the grid, the beam shift distribution may not form neat clusters and may appear more continuous. In any case, the following should be noted when choosing the number of clusters: \* With too few clusters, there will be greater intra-exposure-group variability in the beam shift, possibly leading to less accuracy when fitting the higher-order aberrations \* With too many clusters, there will be fewer exposures and particles per exposure group, possibly limiting the precision of the fit higher-order aberration values. In extreme cases, too few particles per exposure group could impact the stability of the Global CTF Refinement aberration fitting algorithm, as there is a minimum cumulative amount of signal in each exposure group that is needed to fit the aberration parameters. This is important to keep in mind, as aberration estimation is done \*independently\* for each exposure group. The “Clustering method” may also be tweaked. The most important factor when clustering exposures is that clusters are reasonably uniform in both: \* the number of exposures they contain, and \* the range of beam shift values they span The default of \`agglomerative\` clustering works well on a variety of datasets, but we also enable \`kmeans\`. K-means clustering works better when exposures’ beam shift values form isotropic clusters with most points located close to the mean, or when the spread over beam shifts is more “continuous” and doesn’t form neat discrete clusters. In these cases, k-means will ensure that clusters remain relatively uniform in the range of beam shift values that each cluster spans. Agglomerative clustering may perform better when clusters form more irregular shapes, such as the “rings” in this example. ![](https://guide.cryosparc.com/files/IcHssTC2OlQio8iz8CPR) Once the number of clusters is chosen, queue and run the job. At the first checkpoint, the exposure group clustering result is shown. If any exposures are missing beam shift values, they will be placed into their own separate exposure group, and the number of exposure groups outputted by the job in total will be one larger than the parameter value. ![](https://guide.cryosparc.com/files/7jy6SIFSUqj1O0caOci0) The output exposures are now ready for downstream processing, including motion correction, CTF estimation, and particle picking. Be sure to experiment with Global CTF Refinement to see if clustering particles into exposure groups helps obtain better resolutions. Note that only exposure groups with an adequate number of particles should have their aberrations refined, as Global CTF Refinement depends on having enough signal across the particle images in the exposure group. ## Use case #2: Importing beam shifts and clustering exposures \*after\* the original movie import This use case applies when beam shifts were not imported at the time of movie import, for example when: \* The mode of exposure import did not support the simultaneous import of beam shifts, such as the \*Import Movies\* job in a CryoSPARC version older than 4.4 or exposure import or pre-processing in CryoSPARC Live. \* One did not enable the \*Import Beam Shift Values from XML Files\* option when importing movies in CryoSPARC version ≥ 4.4. In this case, the following steps (outlined below) allow for re-clustering of exposure groups: \* Running an Import Beam Shifts job in order to retrieve the exposures’ beam shift values; \* Clustering the movies/micrographs into exposure groups via Exposure Group Utilities, with input particles provided to the job If you are importing fresh movies or micrographs into CryoSPARC v4.4+, Use case #1 covers the basic import case, which is recommended to read first. ### Import Beam Shift Navigate to the job builder, locate the new “Import Beam Shift” job under the imports section, and build an Import Beam Shift job. This is a new job created to add beam shift information to existing exposures datasets in CryoSPARC, without need for re-importing the movies/micrographs from scratch. Next, \*\*connect the existing movies or micrographs dataset from CryoSPARC as input to the Import Beam Shifts job\*\*. This may be a movie dataset exported from CryoSPARC Live, or a movie dataset processed in regular CryoSPARC. Ensure that the entire movie dataset is inputted to the job (i.e. if any exposure curation had filtered out some exposures, ensure to use exposures from upstream to that job). When connecting movies as input to the Import Beam Shifts job, the jobs will use the existing movies’ UIDs rather than generating new UIDs, like the other import jobs. These existing UIDs are required when updating particles’ exposure group assignments in Exposure Group Utilities, to match particles to the exposures that they came from. As in use case #1, provide the XML directory wildcard expression, that points to the directory containing the original XML files. These parameters are identical to those in Import Movies, and the instructions can be followed in \[use case #1 instructions\](https://www.notion.so/T-2825-Tutorial-EPU-AFIS-Beam-Shift-Import-7799197a2beb41c58fc307024a9556b2?pvs=21). If needed, specify the four “Length of movie/XML filename prefix/suffix…” parameters to correctly match movie filenames to XML filenames. Examples of the trimmed file-paths will be printed to the event log to help determine the number of characters. The values of these parameters is most quickly determined by running the job with all defaults, and observing the event log. For example, when connecting movies that were previously imported to CryoSPARC and running the Import Beam Shift job, the event log shows the following messages: ![](https://guide.cryosparc.com/files/42bBu3dYFifDrVLk8XAk) Here, the example movie/mic filename is the same as the XML filename, except for the trailing \`\_EER.eer\`. Due to these extra characters, the beam shift import was not successful, and CryoSPARC warned that it did not find the beam shifts associated with any of the 2797 exposures. To fix this, we can set the “Length of movie filename suffix to cut for XML correspondence” parameter to 8 to cut off the trailing 8 characters and find proper matches between the XML and movie files. Re-running the job, we see that the XML files were found for all but two exposures, which happen to be missing from this dataset: ![](https://guide.cryosparc.com/files/Eq7hJ2DwM4t89ItaI0QP) Finally, if the XML import was successful and beam shifts were present in the XML files, a beam shift scatter plot will be displayed in the event log as in use case #1. The UIDs and all input slots (e.g. motion correction or CTF estimation results) will have been pulled from the input dataset, meaning we do not have to repeat these steps if they have already been done. ### Clustering via Exposure Group Utilities If particles were already picked, we also do not have to repeat particle picking and can instead assign particles to exposure groups based on which exposures they came from. This can also be done via the Exposure Group Utilities job. In this case, we can \[use Exposure Group Utilities as described above\](https://www.notion.so/T-2825-Tutorial-EPU-AFIS-Beam-Shift-Import-7799197a2beb41c58fc307024a9556b2?pvs=21), with the following (\*\*bolded\*\*) modifications: \* Connect the output exposures from “Import Beam Shift” \*\*and the existing particle dataset\*\* to Exposure Group Utilities; \* As in use case #1, Set the “Input Type” to \`exposure\`, specify the “Action” as \`cluster&split\`, and specify the number of clusters and clustering method; \* Activate the “\*\*Correspond particles to exposures and enforce consistency of exposure group IDs\*\*” parameter \* \*\*If particles were previously split into more than one exposure group, set the “Combine strategy” to \`take\_mode\`\*\* \* This ensures that when particles from different exposure groups are combined into the same group, the aberration values for the entire group will be set to the mode (most common value) amongst particles in the group. Since exposure group clustering is done with the purpose of re-running Global CTF Refinement, aberrations will be re-refined and this is not a point of concern. In our case, particles were previously from only one exposure group, so we don’t need to change the combine strategy. Thus, we’ll run the job with the following input parameters: ![](https://guide.cryosparc.com/files/JwnJvQa4jvKW4mQOeijR) Checkpoints 1 and 2 will show the exposure and particle datasets’ exposure group information prior to clustering, respectively, in a table format in the event log. In most cases, particles and exposures will initially be all pooled into one exposure group, unless they were assigned different exposure group IDs upon import. Checkpoint 1 will also show the beam shift scatter plot labelled by the assigned exposure groups. Checkpoints 3 and 4 will show the exposure and particle datasets’ exposure group information after clustering. If “\*\*Correspond particles to exposures\*\*” was activated, the particles and exposures datasets should be consistent. ## Next Steps Once picked particles are obtained and a relatively high-resolution structure has been obtained, use Global CTF Refinement to fit higher-order aberration values. If you followed use case #2, it’s possible to do an apples-to-apples comparison of resolutions before and after clustering particles into exposure groups. This can be done by using two Global CTF Refinement jobs and two Homogeneous Reconstruction Only jobs along with a fixed mask. In our example dataset, the resolution improvements obtained via exposure group clustering were rather modest, indicating that the microscope was quite well calibrated. However, \[examples\](https://discuss.cryosparc.com/t/beam-tilt-refinement-by-image-shift-groups-for-datasets-acquired-with-the-leginon-appion-suite/10682) of more significant improvements have been previously documented on the forum. ![](https://guide.cryosparc.com/files/DOmGPG48ZcC4Kt5R5YcD) Example of processing movies exported from live. Movies were processed through “Import Beam Shifts” to tag them with beam shifts, then together with particles were passed through Exposure Group Utilities to cluster them based on beam shift. Global CTF Refinement was performed twice, once on the initial exposure group assignments (J135: all particles in one group) and once on the new assignments (J136). Two final reconstructions were done, with fixed poses and identical masks, to compute FSCs. ![](https://guide.cryosparc.com/files/pAUTdzbt8ubfGDilJxOd) A modest increase in resolution (1.53 Å) induced by exposure group clustering into 61 clusters, compared to the baseline of 1.54 Å associated with keeping all exposures in one cluster. Note that poses and masks were identical, thus the only variables changed between these two reconstructions were the fitted high-order aberrations from Global CTF Refinement. \## References \* Dustin Morado’s \[EPU\\\_group\\\_AFIS\](https://github.com/DustinMorado/EPU\_group\_AFIS) repository for clustering strategies, as well as his \[detailed forum post\](https://forum.scilifelab.se/t/creating-optics-groups-from-epu-afis-data-and-more/122) describing the motivation for exposure group clustering when collecting in AFIS mode. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-epu-afis-beam-shift-import.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement.md). # Tutorial: 3D Flexible Refinement {% hint style="warning" %} 3DFlex (BETA) is available in CryoSPARC v4.1+. {% endhint %} ## Overview 3D Flexible Refinement (3DFlex) is a motion-based deep generative model for continuous heterogeneity. It can model non-rigid motion and flexibility of a protein molecule across its conformational landscape, and can use the motion model to combine signal from particle images in different conformations to improve refinement resolution in flexible regions. ![](https://guide.cryosparc.com/files/j0nq9shHJ4PbGdfIwu9Z) The 3DFlex model represents the flexible 3D structure of a protein as deformations of a single \*\*canonical\*\* 3D density map \*V\*. Under the model, a single particle image is associated with a low-dimensional \*\*latent coordinate\*\* \*z\* that encodes the conformation for the particle in the image. A neural \*\*flow generator network\*\* \*f\\\_θ\* converts the latent coordinate into the flow field \*u\* and a convection operator then deforms the canonical density to generate a \*\*convected\*\* map \*W\*. This map can then be projected along the particle viewing direction determined by the pose \*φ\*, CTF-corrupted, and compared against the experimental image. {% hint style="warning" %} Complete details of the architecture and training of 3DFlex can be found in the \[bioRxiv preprint here\](https://doi.org/10.1101/2021.04.22.440893). {% endhint %} 3DFlex (BETA) is included in \*\*CryoSPARC v4.1\*\*. This tutorial shows how to run the new job types in CryoSPARC used for creating, training, and using a 3DFlex model. It also covers some of the practical aspects of using the algorithm such as parameter tuning and customizing inputs. Much of the content is covered in a tutorial video below. ## Example Results {% embed url="" %} This video shows results of 3DFlex on a dataset of 102,500 particle images of a tri-snRNP spliceosome particle (EMPIAR-10073). 3DFlex is run with a K=5-dimensional latent space, and different regions of the space correspond to different parts of the particle's conformational landscape. This video shows the output of the 3DFlex generative model as latent coordinates are varied along three axes (coordinates 1, 3, and 5). These dimensions encode non-rigid motion of the head region of the protein, where different parts and subunits move and bend relative to each other. {% endembed %} {% embed url="" %} 3DFlex applied to 58,433 particle images of a translocating ribosome (EMPIAR-10792). Traversing the latent space shows that 3DFlex has learned coordinated motion of multiple parts (e.g., large and small subunits, elongation factor, etc.) including the overall ratcheting motion of the ribosome. For this result, a segmentation was used to specify a tetrahedral mesh topology allowing adjacent subunits to deform separately (see Mesh Generation below). {% endembed %} {% embed url="" %} 3DFlex applied to 113,511 particle images of the SARS-CoV-2 spike protein (EMPIAR-10516). 3DFlex is run with a K=3 dimensional latent space and has learned a combination of motions of the RBD and NTD domains. The up-RBD in particular undergoes a lot of motion which limits its resolution in rigid refinement. In contrast, flexible refinement improves the resolution of the up-RBD. This result also used a segmentation to enable the adjacent RBD and NTD domains to deform separately (see Mesh Generation below). {% endembed %} {% embed url="" %} This video shows results of 3DFlex on a dataset of 200,000 particle images of a TRPV1 ion channel (EMPIAR-10059). 3DFlex is run with a K=2-dimensional latent space. The video shows the output of the 3DFlex generative model as latent coordinates are varied along each of the two dimensions. The first dimension reveals inward and outward coordinated bending of opposite flexible subunits in the soluble domain. The second dimension reveals twisting of the subunits around the pore axis. {% endembed %} {% embed url="" %} This video shows a comparison between the reconstructed density map from a conventional refinement and flexible refinement using 3DFlex for the TRPV1 ion channel. Map quality and local resolutions are substantially improved in the peripheral helices. Notably, local focused refinement using a mask around the flexible part cannot improve the reconstruction compared to a conventional refinement, because the flexible parts are non-rigid and too small for individual pose alignment. {% endembed %} {% embed url="" %} 3DFlex applied to 84,266 particle images of an αV β8 integrin (EMPIAR-10345). 3DFlex using two latent dimensions, learns large bending motions of the flexible arm of the integrin particle, as well as flexibility in the Fabs that are bound. {% endembed %} ## Installing 3DFlex (CryoSPARC v4.1–v4.3 only) {% hint style="info" %} All 3D Flex requirements are installed with CryoSPARC v4.4+. Skip this section unless you are running v4.1–v4.3 {% endhint %} 3DFlex job types are available in \*\*CryoSPARC v4.1+\*\* but in v4.1–v4.3, the new dependencies required for 3DFlex are not installed. To ensure a CryoSPARC worker can run 3DFlex, please see the following instructions: {% content-ref url="/pages/QJmYbRULZz7NCH2AKc9z" %} \[Installing 3DFlex Dependencies (v4.1–v4.3)\](/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement/installing-3dflex-dependencies.md) {% endcontent-ref %} ## Job Types The 3DFlex workflow in CryoSPARC involves five new job types. These jobs are described in more detail in the tutorial video below. \* \*\*3D Flex Data Prep\*\*: Prepares particles for use in 3DFlex training and reconstruction. \* 3D Flex Reconstruction cannot use box sizes larger than 440 pixels, so ensure that the downsample and crop settings of 3D Flex Data Prep produce final images no larger than 440 pixels. \* In CryoSPARC versions prior to v4.4, this job outputs pre-computed CTF values for use by downstream jobs. \* In CryoSPARC v4.4+, the job no longer outputs full-resolution CTF values and the downstream jobs now compute CTF values (including higher order aberrations) on the fly. This change reduces disk space and CPU RAM requirements substantially and allows for higher resolution reconstructions. \* \*\*3D Flex Mesh Prep:\*\* Takes in a consensus (rigid) refinement density map, plus optionally a segmentation and generates a tetrahedral mesh for 3DFlex. See Mesh Generation below. \* \*\*3D Flex Training\*\*: Uses a mesh and prepared particles (at a downsampled resolution) to train a 3DFlex model. Parameters control the number of latent dimensions, size of the model, and training hyperparameters. This job outputs checkpoints during training. \* \*\*3D Flex Generator:\*\* Takes in a checkpoint from training and generates volume series from it, to show what the model is learning about the motion of the particle. This job can be run while training is ongoing to see progress along the way. This job can also optionally take in a high-resolution density map (e.g., from 3D Flex Reconstruction) and will upscale the deformation model and apply deformations to the high resolution map. \* \*\*3D Flex Reconstruction\*\*: Takes in a checkpoint from training as well as prepared high-resolution particles and performs high-resolution refinement using L-BFGS under the 3DFlex model. This is the stage at which improvements to density in high-res regions are computed. Outputs two half-maps that can be used for FSC validation, sharpening, and other downstream tasks. ## Tutorial Video Please watch the following tutorial video that covers usage of 3DFlex. It explains details of the job types, parameter tuning, and other considerations. Most of these details are not currently in written form in the documentation so we encourage users to watch the entire video. {% embed url="" %} ## Mesh Generation {% hint style="info" %} The tetrahedral mesh is an important concept in 3D Flexible Refinement. We cover this topic in significantly more detail in the \[dedicated guide page on the topic\](/processing-data/tutorials-and-case-studies/tutorial-3d-flex-mesh-preparation.md). {% endhint %} As discussed in the preprint, regularization of deformations is critical for a method like 3DFlex. Without strong regularization, the deep generative model can easily overfit to noise in the data and learn unrealistic deformations. 3DFlex uses a tetrahedral mesh (similar to Finite Element Methods) to represent deformation, and applies a rigidity prior that encourages the model to avoid non-rigidity unless it is well supported by the data. In 3DFlex, we define a tetrahedral mesh (or tetramesh) using: \* a set of vertices \* a set of tetra cells, each connecting four vertices \* a “tetra index map”, which is an NxNxN map of indices indicating for each voxel, which tetra cell that voxel belongs to. The tetramesh is defined during the setup of a model. During training, the flow generator outputs a deformation field as a set of deformations of each vertex of the tetramesh, and the convection operator uses the tetra index map to determine how to convect the canonical density based on the movement of the mesh vertices. ### Mesh Topology By default, the 3D Flex Mesh Prep job will automatically generate a regular tetrahedral mesh of specified coarseness, and this typically yields good results, but the 3DFlex method works with any mesh geometry. The mesh topology can be adjusted to introduce additional inductive bias into the model. This is particularly useful for resolving motion of adjacent domains that move differently from each other. ![](https://guide.cryosparc.com/files/ijAHOptpv3XeYXOUGNto) For example, for the SARS-CoV-2 spike protein we obtained good results with a mesh constructed using a sub-mesh for each RBD and NTD domain, fused to a sub-mesh for the central trimer of S2 domains (Figure 11). To construct such a mesh, we provided coarse boundaries between adjacent RBD and NTD domains to the 3D Flex Mesh Prep job, along with the desired topology of the mesh (i.e. which parts are connected to which other parts). The job then automatically generates sub-meshes and fuses them together to form a complete mesh. Please watch the following tutorial video for details about how to use the 3D Flex Mesh Prep job to adjust mesh topology. The 3D Flex Mesh Prep job supports input of \`.seg\` files generated by \[UCSF Chimera’s Segger\](https://github.com/gregdp/segger) tool. This is the easiest way to denote coarse boundaries between segments. The job also supports input of your own custom \`.mrc\` files that you can create that label each voxel with a segment number. {% embed url="" %} The use of custom mesh topology provides helpful inductive bias but does not provide 3DFlex with information about the direction nor types of molecular motions present in the data. Rather, 3DFlex must still learn a non-linear non-rigid deformation from scratch across all mesh nodes jointly during training. Whether using a regular or custom mesh, there is substantial latitude in specifying the mesh. Where motions are smooth, the size and shape of mesh elements and their precise locations are not critical since they only serve to ensure the deformation is smooth, and the flow generator is able to displace the mesh elements (including changing their size or shape) during deformation. Likewise for custom meshes, the separation of subdomains does not need to be “exact” as the canonical voxel density values and structure within each region of the mesh are still learned from the data by 3DFlex. ### Rigidity Weights Along with the mesh topology, 3DFlex also defines rigidity weights for the mesh. The rigidity weight for each cell denotes the relative strength of the rigidity prior that should be applied to that cell. The overall strength of the prior is also a parameter (set at training time) but the relative rigidity is part of the mesh definition. For example, empty space between two subunits should not be very rigid and should be able to compress/expand allowing the subunits to move apart, while high density core parts of a subunit are more likely to remain rigid during deformation). By default, the 3D Flex Mesh Prep job will automatically generate rigidity weights based on the amount of density within each cell in the input consensus (rigid) refinement map. It is also possible to modify rigidity weights or provide custom rigidity weights to 3DFlex. \[See this example \*\*cryosparc-tools\*\* notebook\](https://tools.cryosparc.com/examples/3dflex-custom-latent-trajectory.html). ### Fully Custom Meshes It is possible to create and input fully custom meshes for 3DFlex using \*\*cryosparc-tools\*\*. \[This example notebook\](https://tools.cryosparc.com/examples/3dflex-custom-latent-trajectory.html) includes more details about how a mesh is defined and how to provide your own vertices, cells, tetra index map, and rigidity weights. ## Parameter Tuning Several parameters of the 3DFlex algorithm must be tuned for each dataset in order to give the best results. Details about parameter tuning are in the tutorial video. The important parameters to tune are: \* 3D Flex Mesh Prep: \* \`Base num. tetra cells\` controls the fineness of the tetramesh. Finer meshes allow for more detailed motion but reduce the regularization and with poor quality data or small particles can lead to overfitting. Note that currently 3D Flex Mesh Prep cannot create a mesh with \`Base num. tetra cells\` greater than 40. \* \`Segmentation\` and \`Rigidity weighting\` see Mesh Generation above. \* 3D Flex Training \* \`Number of latent dims\` usually best to start with 2, and increase if the data appears to have more complex motions (and sufficient signal to resolve more motions) \* \`Number of hidden units\` can be reduced to e.g., 32 to limit the capacity of the flow generator model for cases with simpler motion or where overfitting is a concern. \* \`Rigidity (lambda)\` controls the overall strength of the rigidity prior. This should be tuned carefully through empirical tests. When too high, the model will ignore more detailed motions in the data. When too low, the model may learn unrealistic motions due to noise in the data. \* \`Noise injection stdev\` controls the noise injected during latent inference. Higher values encourage more smoothness of the latent conformational landscape (i.e., nearby latent positions will encode similar conformations) but higher values also reduce precision in latent inference, potentially limiting how precisely flexible parts are aligned. \* \`Latent centering strength\` controls the strength of a prior that tries to ensure that latent coordinates are generally centered in the latent space and stay within the range (-1.5, 1.5). This must be tuned for each dataset if you see that latent coordinates are all close to zero or are all hitting the edge of the (-1.5, 1.5) domain. It does not have impact on the results or capacity of the model and is simply a nuisance parameter. \* 3D Flex Reconstruct \* \`Max BFGS iterations\` is set to 20 by default. This can be increased for large box sizes or very high resolution. Also, in some cases it is possible for the FSC curve after 3DFlex Reconstruct to not drop off to zero at high resolutions or appears clearly artefactual, which is an indication that the BFGS optimization has not fully converged. In these cases, it can help to increase this parameter to 40. \* \`Load all particles in RAM\` is a new option in CryoSPARC v4.4 that is off by default, meaning that particle images will be read from the project directory or from SSD cache during iterations of reconstruction, rather than being first pre-loaded into CPU RAM at the start of the job. Keeping this parameter off substantially reduces the CPU RAM requirements of the job, allowing for larger box size reconstructions. Turning the parameter on may improve speed. \* \`Cache particle images on SSD\` is a new option in CryoSPARC v4.4 that is on by default, causing particle images to be cached at the start of the job. Turning this off will cause particles to be read directly from project directories instead of being copied to the cache. ## Limitations 3DFlex is an advance in modelling continuous heterogeneity but it does have several limitations. The most important are listed here: \* Compositional heterogeneity. Being a motion model, 3DFlex currently does not have a way to cleanly represent compositional heterogeneity. It is able to move density around, but cannot delete or add density (the opposite of density-based methods like 3DVA, cryoDRGN, etc.). As such, when presented with data that contain compositional heterogeneity, it may result in strange effects. For example, a domain that is partially occupied in the data may be modelled by creating a deformation that “expands” that domain over a wide space, thereby causing the density to drop, appearing like that domain has been erased. This is obviously not ideal behavior and the 3DFlex model will waste capacity modelling this compositional change rather than conformational changes. Improving 3DFlex in compositional cases is an area of development. Currently we suggest using 3D Classification and Heterogeneous Refinement jobs to ensure that discrete compositional heterogeneity is separated as much as possible before inputting particles into 3DFlex. \* Intricate motions. Though 3DFlex does well in modelling motion even of relatively small parts of a particle, it is not yet capable of modelling highly intricate motions such as side chain or loop motion. These motions are far smaller than the setup of 3DFlex (e.g., using a tetramesh) can allow to be modelled. Furthermore, small motions and conformational changes are unlikely to even be statistically detectable in single particle data unless those motions happen in tandem with other larger changes in the molecule. \* Intermediate states with no data. 3DFlex is strongly biased to modelling motion, and so when presented with data with discrete heterogeneity, it will likely learn a model that maps the multiple discrete states together under deformations that unite them. However, if the data is discrete, there will not be any signal about the actual conformational states of intermediate positions between the discrete endpoints of motion. 3DFlex will still model these transitions, but it will only be guided by its rigidity prior for intermediate states that are not actually seen in the data. \* Interpretation of latent space. The interpretation of 3DFlex is also an interesting area for future work. It is unclear how one should relate the continuous probability distribution of particle images in the 3DFlex latent space to a physically meaningful notion of energy via a Boltzmann distribution. This is because the non-linear capacity of the flow generator means that relative distances and volumes (and hence probability density) in the latent space are arbitrary. ## Computational Considerations 3DFlex is relatively computationally demanding. It is GPU accelerated. Memory: \* GPU memory use is relatively limited during training time, but at reconstruction time the GPU must be able to fit at least 2x the size of a volume at the full resolution box size. We have not yet finely profiled memory usage so it may be more. \* CPU memory in CryoSPARC v4.4+: \* 3DFlex loads all particles into CPU memory at training time. This means you must have sufficient CPU RAM to fit the entire dataset (at the training box size). During 3D Flex Data Prep, you can limit the number of particles. During 3D Flex Reconstruction, particles are read from SSD cache by default and therefore do not need to all fit in CPU RAM. \* CPU memory in CryoSPARC prior to v4.4: \* 3DFlex loads all particles into CPU memory at training time and reconstruction time. This means you must have sufficient CPU RAM to fit the entire dataset (at the training box size for train time, and at the high resolution box size for reconstruction time). During 3D Flex Data Prep, you can limit the number of particles as well. \* 3DFlex does not yet use the CryoSPARC particle caching system. It reads particles directly from project directories into CPU RAM at the start of processing. Speed: \* Speed of 3DFlex training (and reconstruction) are primarily driven by two factors: the number of latent dimensions and the number of voxels (i.e. the volume) inside the solvent mask. Training time will increase approximately linearly with both of these factors. Therefore to speed up training, downsampling to a smaller size (while still retaining enough resolution for training to pick up secondary structure, etc.) is very helpful. Similarly, the solvent mask should not be made overly loose (though it should also be loose enough not to cut off any density in flexible regions that are not well resolved in the consensus rigid density). \* Performance appears to be more strongly affected by GPU performance than other CryoSPARC job types. We have not yet extensively characterized performance but newer/faster GPUs appear to provide substantial benefits. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-flexible-refinement.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement.md). # Tutorial: CTF Refinement ## Overview CTF Refinement includes two major components: local (per-particle) CTF refinement and global (per-group) CTF refinement. \`Local CTF Refinement\` adjusts each particle's defocus value to estimate the z-position of the particle in the sample/ice. \`Global CTF Refinement\` adjusts the higher-order CTF terms (beam-tilt, trefoil, spherical aberration, tetrafoil) across an entire group of images to find the optimum values, accounting for misalignment or aberrations in the microscope itself. In CryoSPARC, both local and global CTF refinement can be performed standalone (using aligned particles and a reference volume as input) or they can be performed on-the-fly during a 3D refinement, so that the values are iteratively optimized along with particle alignments. \*\*New:\*\* As of CryoSPARC v3.3+, \`Global CTF Refinement\` now supports the estimation and correction of anisotropic magnification present in the particle images. ## Local CTF refinement (per-particle defocus) This is a relatively straightforward optimization process of finding the optimal per-particle defocus for each particle in a dataset. Per-particle defocus refinement has been previously proposed and implemented in many other software packages for single particle EM (cisTEM, RELION, Thunder, etc.). \`Local CTF Refinement\` in CryoSPARC requires aligned particle images and a 3D reference (two half-maps), ideally already at a high resolution. Experimental particle images are compared against the 3D reference from their half-set, from the best known pose, at various defocus levels, and the best defocus is selected. The optimal defocus ideally corresponds to the height of the particle in the sample/ice. Since each particle can be at a different height and ice thicknesses can be 10 times larger than the particle diameter in many cases, per-particle defocus refinement can often make a large difference in the accuracy of CTF correction for each particle. However, it generally works best for larger, highly rigid, high quality samples that already reach relatively high resolutions (better than 4A). In general, it is a good idea to try local CTF refinement on every dataset, and to use a homogeneous (gold-standard) refinement to check whether the overall resolution increased or decreased. ### Run Local CTF Refinement Create a \`Local CTF Refinement\` job using the job builder and connect particles from a previously-run refinement (the particles must have \`alignments3D\` defined). Also connect the refined volume from the \*same\* refinement job. You can optionally connect a separate mask input, otherwise the \`mask\_refine\` that is included in the volume input from the previous refinement will be used by default. !\[Job builder for Local CTF Refinement\](/files/-MNdqPytWo-Pksh92U8E) The most important parameters to adjust are: \* Minimum fit res (A): controls the minimum resolution used for fitting. Generally CTF refinement should be done only with medium to high resolution signal, as low resolution signal can throw off CTF fits. For smaller particles, change this to a higher resolution. \* Maximum fit res (A): controls the maximum resolution used for fitting. Higher resolution signal is better for CTF refinement, until there is too much noise present in the half-maps. Leave this blank to have the maximum resolution automatically determined via FSC between the two input half-maps. \* Defocus search range: controls how far above and below the current defocus to search for the optimal defocus of each particle. If you used \`Patch CTF Estimation\` previously in cryoSPARC, this value can be made relatively small, about the same as the thickness of ice you expect to have in the sample, since the input defocus values will already be fairly accurate. !\[\](/files/-MNdsjFEbHK7GaiJQv6V) Once the job is run, several diagnostic plots will be created that show the progress of CTF refinement. !\[\](/files/-MNdsmAkUDORNsWUlDTQ) Plots of per-particle defocus error landscapes show the change in log-likelihood across the range of tested defocus values. The curves should like like these above, showing a clear minimum near 0 change in defocus. The X-axis is in units of Angstroms. The Y-axis is in log units, so each change of 1 unit corresponds to a change of 1/e^1 = 0.367 in probability. Therefore, plots with a minimum that is hundreds of units deep indicate that we are highly confident about the optimal defocus value. On the other hand, plots with very shallow minima (tens of units) indicate uncertainty in the optimal defocus. !\[\](/files/-MNdsptcmzFmozYxebiE) Histograms showing the change in per-particle defocus across all the particles in the half-set indicate the total amount of deviation from the input defocus parameters that was achieved by CTF refinement. The histogram should generally be very peaked near zero and should not have heavy tails. Heavy tails, or the presence of many particles having optimal defocus values at the ends of the search range indicates that defocus refinement was not very confident or accurate. ## Global CTF Refinement (per-exposure-group beam-tilt, trefoil, spherical aberration, tetrafoil, and anisotropic magnification) Ultra high resolution cryo-EM structures require correcting for electron-optical aberrations and microscope misalignments that result in nuanced "high-order" terms in the Contrast Transfer Function (CTF). These higher order terms (corresponding with beam tilt, trefoil, spherical aberration, tetrafoil) can only be detected at very high resolution, and cannot easily be estimated by straightforward measurements on the microscope. Therefore, the strength of each of these aberrations must be estimated from single particle data itself, by refining the corresponding CTF parameters against a high-resolution reference map. This process of high-order aberration estimation and correction was pioneered by (Zivanov et al. 2019) in RELION 3.1. While microscope misalignments can result in higher order terms affecting the CTF, microscopes occasionally show magnification anisotropy. The result of this anisotropy is that micrographs are slightly distorted by a linear transformation (or "stretch") in the image plane. Unlike higher order aberrations, anisotropic magnification cannot be corrected by better microscope alignment, and must either be estimated using the diffraction pattern of known crystalline samples, or by projection-matching using a high quality reference map. As of CryoSPARC v3.3+, the latter method of anisotropic magnification estimation and correction is now supported, which also follows closely the developments made by Zivanov et al. High-resolution signal is typically required to estimate any anisotropy, and unless the anisotropy is extreme, correcting for it will typically only improve maps that have already reached a fairly high resolution. Furthermore, errors in defocus and astigmatism due to magnification anisotropy are also corrected when fitting the magnification matrix. CryoSPARC v2.12+ contains a GPU accelerated implementation of high-order aberration estimation and correction. In all cases, estimation is done by directly maximizing the likelihood of observing the experimental images given a 3D reference map, using \*LBFGS\*. Images collected on a given microscope generally will have related CTF parameters for higher-order aberrations and anisotropic magnification. The images that are related (same grid, same image shift position, etc.) can be grouped into "exposure groups" so that they can all be refined at once, with more signal. Creation and management of exposure groups is explained in the next section. Like local CTF refinement, \`Global CTF Refinement\` generally works best with larger, more rigid particles. However, \`Global CTF Refinement\` does use signal from all the particles in an exposure group, and so can detect beam tilt and other aberrations even with smaller/flexible structures. ### Run Global CTF Refinement Create a \`Global CTF Refinement\` job using the job builder and connect particles from a previously-run refinement (the particles must have \`alignments3D\` defined). Also connect the refined volume from the \*same\* refinement job. You can optionally connect a separate mask input, otherwise the \`mask\_refine\` that is included in the volume input from the previous refinement will be used by default. !\[Job builder for Global CTF Refinement\](/files/-MdIwrGz50JE1RuBPa1w) The most important parameters to adjust are: \* Number of iterations: controls the number of iterations of CTF refinement that are done. It is important that the number of iterations is at least 2 when anisotropic magnification is being estimated together with the other aberrations. This is to allow the aberrations to be fit to the data while accounting for anisotropic magnification. If only aberrations are being estimated, 1 iteration is usually sufficient. \* Minimum fit res (A): controls the minimum resolution used for fitting. Generally global CTF refinement should be done only with medium to high resolution signal, as low resolution signal can be unreliable. For smaller particles, change this to a higher resolution. \* Maximum fit res (A): controls the maximum resolution used for fitting. Higher resolution signal is better for CTF refinement, until there is too much noise present in the half-maps. Leave this blank to have the maximum resolution automatically determined via FSC between the two input half-maps. \* Fit Tilt/Trefoil/Spherical Aberration/Tetrafoil: Select which higher-order aberrations should be refined. Tilt and Trefoil are 3rd order and require less high resolution signal to accurately detect, compare to spherical aberration and tetrafoil which are 4th order. In some cases, optimizing the 4th order terms can be detrimental, especially if per-particle defocus or the 3rd order terms are not yet correctly refined. Note: as of v4.0, only the third-order aberrations (Tilt and Trefoil) and fit by default, whereas the fourth-order aberrations (Spherical Aberration and Tetrafoil) are \*not\* fit by default. \* Fit Anisotropic Magnification: Activate to enable the estimation of anisotropic magnification. Note that this is \*inactive\* by default, since significant anisotropic magnification is a relatively rare phenomenon compared to beam tilt and other aberrations. !\[\](/files/-MNdt0BLYoavbgO9fjYH) Once the job is run, several diagnostic plots will indicate the phase delay and fit diagnostics of each type of aberration. !\[\](/files/-MNdt304-EDwtidHJcN1) For each order of aberration (odd and even), three plots are made. The first shows the phase error data that is measured from all the particles in aggregate. On the left is the full phase error, on the right are the masked out terms that will be used for fitting. For the odd terms, the aberrations appear as anti-symmetric patterns of delay (blue) and advancement (red) of the diffracted beam. The second plot shows the fit predicted values of the phase delay, after refining CTF parameters. The third plot shows the residual phase error between the data and fit, which should only contain noise indicating a good fit. Similar plots are made for the even terms in the CTF. Note that odd terms are optimized from zero each time the \`Global CTF Refinement\` job is run, meaning that the plots will always show aberrations in the measured data (first plot). Even terms, on the other hand, are optimized starting from their current input values. Therefore if \`Global CTF Refinement\` is run twice, the second time, the even terms will show nearly zero aberration in the measured data (since the input CTF parameters are already nearly correct). Note that in the output log of \`Global CTF refinement\` the units of each aberration parameter are printed. Beam-tilt is internally parameterized in Angstroms rather than radians, as converting to the latter requires a non-zero spherical aberration coefficient. Values in milli-radians are printed in cases where the spherical aberration is non-zero. #### Anisotropic Magnification If magnification correction is enabled, three plots are also made akin to that of the aberration plots. The first plot shows the predicted displacement per-pixel based on an unconstrained least-squares fit, which should typically show a linear trend. The absence of a trend indicates that the anisotropy is not significant, and a non-linear trend could indicate that the anisotropy is severe (so that multiple iterations are necessary to converge), or that there are other systematic effects in the data. The second plot shows the fitted displacement values based on the current estimate of the magnification matrix, and the third plot shows the residual (i.e. the unconstrained displacements minus the linear fit). Note that two sets of plots are made, showing the displacements in the x direction and the y direction separately. Similar to the even aberrations, anisotropic magnification is optimized from its current values at the start of the iteration. This is done because unlike the refinement of odd and even aberrations, the refinement of anisotropic magnification involves approximations to the log likelihood objective function, and this approximation improves as the magnification matrix converges. As well, all high-order CTF parameters are fit to the current values of the magnification matrix. For these reasons, it is recommended to perform at least 2 iterations of CTF refinement when fitting aberrations together with anisotropic magnification. After two iterations, the residual anisotropy should typically be very small. !\[Anisotropic magnification plot for EMPIAR-10395.\](/files/-MdIYjiJkgrcsJcGoTHv) Above is an example of the anisotropic magnification plot from the first iteration, for \[EMPIAR-10395\](https://www.ebi.ac.uk/pdbe/emdb/empiar/entry/10395/). On the left are the displacement plots for the Y coordinate from the first iteration, indicating that there is a significant linear trend in the predicted displacements at each voxel. On the right are the plots from the final iteration, showing a residual with no linear trend, hence no fit. The absence of a fit in the final iteration plot indicates that the anisotropic magnification matrix has converged. This job was run with three iterations. ## Ewald Sphere Correction Both Local and Global CTF Refinement may be run with Ewald Sphere correction enabled. This means that estimation of per-particle defocus and high-order aberration parameters can be done while accounting for the curvature of the Ewald Sphere. Generally, this does not significantly impact the outcome unless previous reconstructions have shown that Ewald Sphere correction results in a measurable resolution increase. To use this feature in either Local or Global CTF Refinement jobs, activate the \`Account for EWS curvature\` parameter and make sure to set the \`EWS curvature sign\` to the correct value of curvature determined from previous reconstructions with Ewald Sphere enabled. For more information on how to obtain these reconstructions and the curvature sign, please refer to the Ewald Sphere Correction tutorial for a detailed walkthrough. ## On-the-fly CTF refinement in homogeneous refinement In CryoSPARC v2.12+, both \`Local CTF Refinement\` and \`Global CTF Refinement\` can be run as standalone jobs. However, since the refinement of these parameters is very fast, they can also be run on-the-fly during iterations of \`Homogeneous Refinement\`. In the new \`Homogeneous Refinement\` job in v2.12+, there are new parameters to enable local and/or global CTF refinement. CTF refinement is carried out iteratively with refinement of 3D poses and the 3D map, starting once the initial refinement is converged. The new \`Homogeneous Refinement\` job in v2.12+ will create plots similar to the standalone CTF refinement jobs, and the final CTF parameters after refinement will be outputted along with the 3D alignments of particles. ## Non-uniform refinement with high-order CTF correction In CryoSPARC v2.12+, the \`Non-uniform Refinement\` job has been updated to use the new GPU code that supports higher-order CTF correction, but this is \*\*NOT\*\* enabled by default. You must turn on the \`Enable higher-order CTF\` parameter in \`Non-uniform refinement\`. Please also note that legacy refinement jobs will not support the correction of high-order CTF aberrations or anisotropic magnification. On-the-fly CTF refinement cannot be done during a \`Non-uniform Refinement\`, so particles should be processed through the standalone \`Local CTF Refinement\` then \`Global CTF Refinement\` jobs first. ## Exposure Groups In CryoSPARC v2.12+, particles, movies, and micrographs are organized into "Exposure Groups", which allow images with the same microscope configuration (beam tilt, image shift, etc) to have their CTF refinement done independently in a streamlined manner. This section describes the tools in CryoSPARC to create and manage exposure groups. ### \*\*At Import time\*\* When you import a dataset (movies, micrographs or particles) in CryoSPARC v2.12+, the set of imported data is automatically set with a new "exposure group ID". This ID is \*\*unique within a project\*\* (the group ID increments with each import job, starting from zero) unless overridden using the \`Override Exposure Group ID\` parameter. Using this method, you can import your datasets separately based on their beam tilt groups, or any other groups where you would like to use, and the grouping of imports will be retained even if the datasets are merged later on in processing. ### \*\*Using the \`Exposure Groups\` utilities\*\* If you have a dataset that was imported prior to v2.12, or a dataset that contains multiple exposure groups and you would like to separate each of the groups in the dataset, you can use the \[\`Exposure Group Utilities\` Job\](https://cryosparc.com/docs/reference/jobs#exposure-group-utilities). This job allows you to view, split, and combine datasets into one or more exposure groups. To split a dataset into exposure groups, can select which file path attribute of the dataset will be used to identify unique groups. For example, in EPU, when capturing multiple images per hole, each shift position should be separated as a separate group. The groups can be identified by the first section of numbers right after the the word "Data" in the file path, as outlined below: FoilHole\\\_21256428\\\_Data\\\_\*\*21254194\*\*\\\_21254195\\\_20190622\\\_0517\\\_Fractions.mrc Knowing this, we can separate our exposure (or particle) dataset into unique exposure groups. Input your data into the \`Exposure Group Utilities\` job, and select the \`split\` mode. Use the parameters \`Field to use to split Dataset\`, \`Start Slice Index\`, and \`Number of characters to Consider\` (\[more information here\](https://cryosparc.com/docs/reference/jobs#exposure-group-utilities)) to create unique tokens out of the file paths available. The job automatically creates and sets exposure groups for these tokens: !\[\](/files/-MNdtD4as3nWvWzaXx9x) You can choose to output each exposure group separately by using the \`Split Outputs by Exposure Group\` parameter. You can also combine multiple exposure or particle datasets by connecting them all into the respective input slot in the Exposure Groups Utilities job. Using the \`combine&set\` mode and \`Set Exposure Group Value\` parameter, you can combine all connected datasets and set their exposure group to the same value. Note that when this happens, the job will check that the CTF values across the exposure group are consistent- you can decide what the job will do if it finds inconsistent values using the \`Combine Strategy\` parameter. ### For advanced users Another way to modify the exposure group ID for a dataset is to Export the job that created the dataset (or create a .csg file manually) \[(view data management tutorial)\](https://cryosparc.com/docs/tutorials/data-management) and modify the .cs file directly. You will have to modify the field \`ctf/exp\_group\_id\` (and \`mscope\_params/exp\_group\_id\` for movie/micrograph datasets or \`location/exp\_group\_id\` for particle datasets) for all items inside the dataset. You can set these columns with the desired group identifiers, which do not need to be sequential but do need to be unique. If your dataset does not have this result slot (which may be the case for jobs \*\*not\*\* processed by Patch CTF Estimation or imported prior to v2.12), you will have to first add the field, then modify the fields. See the python (2.7) example below. \`\`\`python #import the modules from cryosparc\_compute import dataset from cryosparc\_compute import common #load the dataset particle\_dataset = dataset.Dataset.load() #add missing fields (this example is for particle datasets) particle\_dataset = common.create\_missing\_fields\_in\_dataset(particle\_dataset, 'ctf', 'particle.ctf') #set the exposure group id particle\_dataset\['ctf/exp\_group\_id'\]\[:\] = 2 \`\`\` You can then re-import this dataset using the Import Result Group job. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-ctf-refinement.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/guide-performance-benchmarking-v4.3.md). # Guide: Performance Benchmarking (v4.3+) ## Overview After installing CryoSPARC and verifying the instance is working correctly (see \[Installation Testing\](https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/guide-installation-testing-with-cryosparcm-test)), use the Performance Benchmarking job to measure the performance of your system and compare it against references provided by Structura and your own past benchmarks. The new “Benchmark” job is available in the CryoSPARC job builder and can be run on any worker lane connected to your CryoSPARC instance. ![](https://guide.cryosparc.com/files/v32tYPEkJfmJtpwk2yMH) The “Benchmark” job will make sure the benchmark data exists in the right location (and downloads it if it doesn’t), and runs the three benchmark tests in serial (CPU, Filesystem and GPU benchmarks) as specified. {% hint style="warning" %} CryoSPARC v5.0+ performance benchmark system is not backwards compatible with earlier versions. This means that performance benchmarks recorded in v5.0+ will be dropped when downgrading to v4.7 or below. v5.0 instances include new updated reference performance benchmarks on bare metal and AWS node types. {% endhint %} ### Benchmark Data The benchmark data (17GB, compressed) is required to be downloaded and extracted into a location accessible by the job in order to run the benchmarks. As a convenience, this is automatically done by the Benchmark job when the required data does not exist in the project directory. The benchmark data can also be manually downloaded via the link provided below. Once manually downloaded and extracted, the absolute path to the folder can be specified in the “Benchmark Data Directory” parameter. \[Click here to download the benchmark data package directly from cloud storage.\](https://s3.us-east-1.wasabisys.com/cryosparc-performance-benchmark-data/performance\_benchmark\_data\_v1.tar.gz) The benchmark data package contains movies, particles and volumes required for each of the tests. An abridged directory listing can be seen below: \`\`\` . ├── class2D\_test │   └── maps.mrc ├── gpu\_engine\_test │   ├── abinit\_particles.cs │   ├── abinit\_volume.mrc │   └── J586 │   └── extract │   ├── 001411154804159785773\_14sep05c\_c\_00003gr\_00014sq\_00005hl\_00003es.frames\_patch\_aligned\_doseweighted\_particles.mrc │   ├── (...)\_14sep05c\_c\_00003gr\_00014sq\_00006hl\_00003es.(...).mrc │   ├── (...)\_14sep05c\_00024sq\_00003hl\_00005es.(...).mrc │   ├── (...)\_14sep05c\_c\_00003gr\_00014sq\_00007hl\_00005es.(...).mrc │   ├── (...)\_14sep05c\_c\_00003gr\_00014sq\_00006hl\_00005es.(...).mrc │   ├── (...)\_14sep05c\_c\_00003gr\_00014sq\_00005hl\_00002es.(...).mrc │   ├── (...)\_14sep05c\_c\_00003gr\_00014sq\_00009hl\_00004es.(...).mrc │   ├── (...)\_14sep05c\_c\_00003gr\_00014sq\_00011hl\_00003es.(...).mrc │   ├── (...)\_14sep05c\_00024sq\_00004hl\_00002es.(...).mrc │   ├── (...)\_14sep05c\_c\_00003gr\_00014sq\_00011hl\_00002es.(...).mrc │   ├── (...)\_14sep05c\_c\_00003gr\_00014sq\_00008hl\_00005es.(...).mrc │   ├── (...)\_14sep05c\_c\_00003gr\_00014sq\_00004hl\_00004es.(...).mrc │   ├── (...)\_14sep05c\_c\_00003gr\_00014sq\_00010hl\_00002es.(...).mrc │   ├── (...)\_14sep05c\_c\_00003gr\_00014sq\_00007hl\_00004es.(...).mrc │   ├── (...)\_14sep05c\_00024sq\_00006hl\_00003es.(...).mrc │   ├── (...)\_14sep05c\_c\_00003gr\_00014sq\_00005hl\_00005es.(...).mrc │   ├── (...)\_14sep05c\_00024sq\_00003hl\_00002es.(...).mrc │   ├── (...)\_14sep05c\_c\_00003gr\_00014sq\_00006hl\_00002es.(...).mrc │   ├── (...)\_14sep05c\_c\_00003gr\_00014sq\_00002hl\_00005es.(...).mrc │   └── (...)\_14sep05c\_c\_00003gr\_00014sq\_00011hl\_00004es.(...).mrc ├── gpu\_fsc\_test │   ├── half\_map\_A.mrc │   └── half\_map\_B.mrc ├── movies │   ├── eer │   │   ├── FoilHole\_2669035\_Data\_2668380\_2668382\_20200703\_235726\_Fractions.mrc.eer │   │   ├── FoilHole\_2669035\_Data\_2668383\_2668385\_20200703\_235738\_Fractions.mrc.eer │   │   └── FoilHole\_2669035\_Data\_2671097\_2671099\_20200703\_235716\_Fractions.mrc.eer │   ├── mrc │   │   ├── 17jul30a\_b\_00007gr\_00002sq\_v01\_00002hl16\_00002edhiii.frames.mrc │   │   ├── 17jul30a\_b\_00007gr\_00002sq\_v01\_00002hl16\_00004edhiii.frames.mrc │   │   └── 17jul30a\_b\_00014gr\_00001sq\_v01\_00002hl16\_00005edhiii.frames.mrc │   └── tiff │   ├── FoilHole\_21044295\_Data\_21043958\_21043960\_20210422\_050646\_fractions.tiff │   ├── FoilHole\_21044296\_Data\_21043958\_21043960\_20210422\_050719\_fractions.tiff │   └── FoilHole\_21044297\_Data\_21043958\_21043960\_20210422\_051105\_fractions.tiff └── picking\_test ├── 014750136049583239150\_14sep05c\_00024sq\_00004hl\_00002es.frames\_background.mrc ├── 014750136049583239150\_14sep05c\_00024sq\_00004hl\_00002es.frames\_patch\_aligned\_ctf\_spline.npy ├── 014750136049583239150\_14sep05c\_00024sq\_00004hl\_00002es.frames\_patch\_aligned\_doseweighted.mrc ├── exposure\_dataset\_for\_picking.cs ├── picking\_templates.mrc └── templates\_dataset\_for\_picking.cs \`\`\` #### Particle Data Source Particles previously processed in CryoSPARC from a subset of movies in EMPIAR-10025: {% embed url="" %} #### Movie Data Sources \*\*TIFF:\*\* 3x K3 Super-resolution (11520, 8184) 70 frames: 1.16GB each from EMPIAR-10721: {% embed url="" %} \*\*MRC:\*\* 3x K2 (3710, 3838) 44 Frames 1.2GB each from EMPIAR-10249: {% embed url="" %} \*\*EER:\*\* 3x Falcon 4 (4096, 4096) 48 Frames: 500MB each from EMPIAR-10612: {% embed url="" %} ### Sharing Data with Structura Biotechnology (Optional) In each job, there is a parameter (”\*\*Share benchmark data with Structura Biotechnology”,\*\* disabled by default) to allow uploading of benchmark data to Structura’s servers. The data sent includes timings and hardware information, but does not include any user identifiable information. An example of the data uploaded can be seen below: \`\`\` { "type" : "gpu", "timings" : { "fsc\_spherical" : { "put\_mapr\_on\_gpu" : 0.5858473777771, ... }, "fsc\_loose" : { "put\_mapr\_on\_gpu" : 0.677032470703125, ... }, ... }, "cryosparc\_version" : "v4.1.0", "created\_at" : 1666985248.02303, "instance\_information" : { "platform\_node" : "server\_hostname", "platform\_release" : "4.15.0-142-generic", "platform\_version" : "#146~16.04.1-Ubuntu SMP Tue Apr 13 09:27:15 UTC 2021", "platform\_architecture" : "x86\_64", "cpu\_model" : "Intel(R) Xeon(R) CPU E5-1630 v4 @ 3.70GHz", "physical\_cores" : 4, "total\_memory" : "62.80GB", "available\_memory" : "52.17GB", "used\_memory" : "9.81GB", "ofd\_soft\_limit" : 1048576, "ofd\_hard\_limit" : 1048576 }, "job\_params" : { "benchmark\_data\_dir" : null, "gpu\_num\_gpus" : 1, "send\_data" : true, "test\_random" : true, "test\_sequential" : true, "use\_all\_gpus" : false, "use\_ssd" : true }, "gpu\_name" : "NVIDIA GeForce GTX 1080 Ti", "gpu\_bus\_id" : "0000:02:00.0" } \`\`\` The data sent includes timings and hardware information, but does not include any user identifiable information. \*\*Structura will use this data to maintain aggregate statistics about CryoSPARC performance in the wild and help us focus our optimization efforts on the jobs and codepaths with the most benefit to users. Users who do upload benchmark data should not expect any direct response from Structura.\*\* ## Filesystem Benchmark The filesystem benchmark employs a sequential read test for movies, and both a sequential and random read test for particles simulating real CryoSPARC workflows to benchmark the filesystem where the benchmark data exists. Turn off the parameter “Use SSD for Tests” to disable the use of the caching system when performing the particle read tests. The job will instead report the time it takes to read the particles in a sequential and random pattern from the project directory instead of a local cache device. ### A Note About Filesystem Caching On Linux, the “page cache” is an area of unused memory that is used to store data that the OS reads for later rapid retrieval. For example, when you read a 1GB file twice, the second access of the file will be faster, since the file blocks come directly from the cache in memory instead of the hard disk or SSD. The OS automatically frees up data stored in the page cache as more memory is requested by other applications. The Benchmark job attempts to drop files that it uses from the page cache by using the \[\`posix\_fadvise\`\](https://linux.die.net/man/2/posix\_fadvise) function to declare that the files “will not be accessed in the near future” (\`POSIX\_FADV\_DONTNEED\`). Doing so allows subsequent runs of the Benchmark job to be reproducible (meaning that the numbers reported by the job won’t be skewed by faster read times) without having to manually drop the page cache. {% hint style="info" %} \*\*Dropping the page cache:\*\* The following two commands first instruct the kernel to \[write dirty pages to disk\](https://linux.die.net/man/8/sync), then \[drop the page cache\](https://www.kernel.org/doc/Documentation/sysctl/vm.txt): \`sudo bash -c 'sync; echo 1 > /proc/sys/vm/drop\_caches'\` {% endhint %} Note that there still may be other caches in play if your data is hosted on other machines (e.g., a storage cluster’s cache). ### Sequential Read Test - Movies To benchmark sequential reading, which is relevant in the early stages of data processing, three different types of movies (TIFF, EER and MRC) are read and timed. The test reports the averages of the total I/O time taken. This measures the performance of the storage volume on which the movies are located. To benchmark a storage volume that is different from the project directory, copy the benchmark data to the new location and specify it in the “\*\*Benchmark Data Directory”\*\* parameter. For more information on the sources of each of the movies, see \[Benchmark Data\](#benchmark-data). For TIFF and EER movies, only the time it takes for the system to read the data into memory is recorded. The time it takes for the movies to be decompressed (which is always performed when reading TIFF and EER movies) is timed but not recorded. ### Sequential Read Test - Particles To benchmark sequential particle reads, which are relevant during some parts of particle processing, a small particle stack (50,000 particles with shape (256,256) across 500 files) is randomly generated using \[\`numpy.random.randn\`\](https://numpy.org/doc/stable/reference/random/generated/numpy.random.randn.html), written to the project directory, cached onto the cache device (if available and enabled), then read (in a sequential pattern) back into memory. The time it takes the system to cache the particles (\`particle\_cache\_time\`), read the particles sequentially (\`particle\_sequential\_read\_time\`) and the rate at which the particles are read (\`particle\_sequential\_read\_rate\`) are recorded. ### Random Read Test - Particles To benchmark random reads, which are relevant during most parts of particle processing, the same particle stack created during the sequential read test is used, but this time the particles are read in a random pattern. The time it takes the system to read the particles randomly (\`particle\_random\_read\_time\`) and the rate at which the particles are read (\`particle\_random\_read\_rate\`) are recorded. ## CPU Benchmark The CPU Benchmark reads the same TIFF and EER movies from the Filesystem test (see \[Sequential Read Test\](#sequential-read-test-movies)), but instead reports the time it takes to decompress the movies, which is heavily dependent on CPU and Memory performance. ![](https://guide.cryosparc.com/files/K8STzarbDTQM7emp9swy) Note that in order to measure decompression time, the CryoSPARC environment variable \`CRYOSPARC\_TIFF\_IO\_SHM\` must be set and turned on (which by default, it is). \[See Environment Variables\](https://guide.cryosparc.com/setup-configuration-and-management/management-and-monitoring/environment-variables#cryosparc\_worker-config.sh). This parameter tells the IO system to first copy the contents of TIFF and EER files to \`/dev/shm\` (a temporary file storage system backed by RAM) before decompressing it, allowing the system to distinguish IO time from decompression time. Note that this parameter also increases performance on some networked file systems. Decompression time is averaged from three runs of different movies and reported as \`tiff\` and \`eer\` in the CPU tab of the Benchmark viewer. ## GPU Benchmark The GPU benchmark executes a collection of functions from CryoSPARC jobs on each of the worker’s GPUs (unless the “\*\*Number of GPUs to benchmark\*\*” parameter is specified, in which case only the specified number of GPUs are benchmarked), and times them. The tests include: 1. FSC Calculations using different masks: \* Spherical Mask (\`fsc\_spherical\`) \* Loose Mask (\`fsc\_loose\`) \* Tight Mask (\`fsc\_tight\`) \* Noise Sub Mask (\`fsc\_noisesub\`) 2. Non-Uniform Refinement’s core algorithm (\`matched\_cv\_filter\_estimation\`) 3. Particle picking’s core algorithm (\`picking\`) 4. CryoSPARC’s core alignment and reconstruction algorithm (”Engine”), tested with various parameter combinations: \* Particles in cache \* Using 1 CPU thread \* Using a trilinear interpolation kernel \* Using C1 symmetry \* Using Pose Maximization \* Test A (\`disk\_single\_linear10\_max\_C1\`) \* Using a tricubic interpolation kernel \* Using C1 symmetry \* Using Pose Maximization \* Test B (\`disk\_single\_linear20\_max\_C1\`) \* Using 2 CPU threads \* Using a trilinear interpolation kernel \* Using C1 symmetry \* Using Pose Maximization \* Test C (\`disk\_multi\_linear10\_max\_C1\`) \* Using a tricubic interpolation kernel \* Using C1 symmetry \* Using Pose Maximization \* Test D (\`disk\_multi\_linear20\_max\_C1\`) \* Particles in memory \* Using 1 CPU thread \* Using a trilinear interpolation kernel \* Using C1 symmetry \* Using Pose Maximization \* Test E (\`memory\_single\_linear10\_max\_C1\`) \* Using Pose Marginalization \* Test F (\`memory\_single\_linear10\_marg\_C1\`) \* Using D7 symmetry \* Using Pose Maximization \* Test G (\`memory\_single\_linear10\_max\_D7\`) \* Using a tricubic interpolation kernel \* Using C1 symmetry \* Using Pose Maximization \* Test H (\`memory\_single\_linear20\_max\_C1\`) \* Using Pose Marginalization \* Test I (\`memory\_single\_linear20\_marg\_C1\`) {% hint style="info" %} As of CryoSPARC v4.4+, \`memory\_multi\_\*\` (particles in memory + multithreaded) tests have been removed. {% endhint %} #### Non-Uniform Refinement’s core algorithm The core algorithm used in Non-Uniform Refinement performs multiple data transfers to and from the GPU, while performing hundreds of GPU-accelerated FFTs. This test stresses the memory performance of the GPU and PCIe bandwidth of the CPU, and is limited by the performance of a single CPU core. #### CryoSPARC’s core reconstruction algorithm The different parameter combinations specified for the core reconstruction algorithm tests code paths used by various CryoSPARC jobs including Homogeneous Refinement, Non Uniform Refinement, 3D Classification, and more. The test name (e.g., \`memory\_multi\_linear20\_marg\_C1\`) is comprised of the following parameters used to perform the test: \\\*\\\*\\\*\\\*\\ \* Particle Location: \* Particles can either be stored on the cache device (SSD) if caching is enabled, or read into memory. When particles are in memory, IO time becomes negligible. \* Number of CPU Threads: \* CryoSPARC’s core algorithm can be run with one or two threads. Most of the time, it’s run with two threads, so that particle IO and GPU computation is performed concurrently. Note that in tests using 2 CPU threads, some timing numbers are not accurate due to the concurrency of the computation, which is why only the \`overall\` time is reported. In these cases, it’s best to compare the timings from the corresponding single-threaded test. \* Interpolation Kernel: \* CryoSPARC’s refinement and classification algorithms use two main interpolation kernels: trilinear (\`linear10\`) and tricubic (\`linear20\`) to interpolate values of the 3D density in Fourier space. Interpolation is necessary when rotating and projecting the 3D density, which is used in the orientation search step in most refinement/classification/variability jobs. Trilinear interpolation is significantly less computationally expensive than tricubic interpolation, requiring only 8 array accesses (vs. 64) of the underlying 3D density. Trilinear interpolation is also hardware-accelerated on NVIDIA GPUs through CUDA, whereas tricubic interpolation is not. \* Pose Assignment Method: \* Non-Uniform Refinement supports either pose “\`max\`imization” or “\`marg\`inalization” during the reconstruction of the 3D density from the particle images. Maximization means that each particle is assigned a single 3D pose and shift during reconstruction. Alternatively, marginalization allows each particle to be assigned multiple 3D poses and shifts, each being weighted by their relative likelihoods under the image formation model. Maximization is usually sufficient, but for small particles or noisy datasets, marginalization helps to account for uncertainty in estimating the poses. When reconstructing the 3D density, maximization only has to insert each particle image into the 3D reconstruction \*\*once;\*\* marginalization is more computationally expensive because it requires inserting each image into the reconstruction multiple times. \* Symmetry Operator: \* CryoSPARC’s core reconstruction algorithm supports many symmetry operators, but C1 and D7 were chosen for these benchmarks as a way to turn “off” and “on” the code paths respectively that enable symmetry. ## Interpreting Results Using The Benchmark Viewer To view and compare previous Benchmark results and reference benchmarks provided by Structura, navigate to the “Benchmarks” tab inside the “Manage” panel. ![](https://guide.cryosparc.com/files/suIvaRUOtO9vnu8eDYPJ) ![](https://guide.cryosparc.com/files/IQhOD2K811MWF7bgotNS) Under each sub-tab (CPU, File System, GPU, Extensive Validation), there will be reference benchmarks provided by Structura which can be used as a comparison against benchmarks run on the current instance. To compare multiple references, select them from the table using the checkboxes and click the “Compare” button on the top right side of the screen. ![](https://guide.cryosparc.com/files/pSIKXA4BgbPaIplPx9nU) In the comparison view, each benchmark is a column, and their timings are listed as rows. The overall time that the benchmark took is listed under the “Time” sub-column (\*\*A\*\*), and the portion of how long it took relative to the other timings is represented as a percentage in the “Pct” sub-column. When a benchmark (column) is selected, it becomes the base “Reference” \*B1\* for the “Speedup” columns \*B2\*, which are available for all other benchmarks in the comparison view. The “Speedup” is calculated as $$ Reference(seconds)/Current(seconds), $$ which helps to easily glean how much faster or slower a timing is \*in comparison to\* the reference. When a timing is hovered over, its details will be displayed in the “Benchmark Details” section on the right side of the page (\*\*C\*\*). ![](https://guide.cryosparc.com/files/6pyW4iwNuURBrrf3Wp1R) To view more detailed timings (available for the GPU benchmark only), click on the “+” button to expand a row (\*\*D\*\*). These sub-timings are the low-level functions that get called inside of CryoSPARC’s core reconstruction algorithm. The “Tags” column (\*\*E\*\*) indicates what hardware component each function’s speed is most dependent on. For example, for the \`setup\_scales\` sub-timing, the relevant component tags are “PCIe Latency/Bandwidth Speed” and “GPU/CPU Memory Allocation” because the function allocates space for a float32 array in CPU and GPU memory, fills it with data in CPU Memory, then downloads the contents of the array from CPU memory to it’s corresponding location on GPU memory. When a “download” happens, this occurs over the PCIe lanes that connect the CPU to the GPU, where the link speed (determined by e.g., PCIe Gen. 3 on most GPUs and PCIe Gen. 4 on NVIDIA Ampere and Ada architectures) matters the most in determining how fast this happens. ### Component Tags | Tag Name | Most likely bottleneck | | --- | --- | | CPU Performance | \- single core clock speed | | GPU Performance | \- float32 performance and memory bandwidth | | PCIe Latency/Bandwidth Speed | \- PCIe generation (3, 4) and number of lanes per slot (x8, x16) | | GPU/CPU Memory Allocation | \- general cpu/gpu performance | | Input/Output Speed | \- random read speeds of the storage device where particle images are located | \### Raw Data At the end of the benchmark job, results are saved as a JSON and CSV in the job directory. The exact path of the files can be seen at the end of each test in the job’s Event Log. \`\`\` Writing benchmark data to /bulk9/data/dev\_projects/CS-peformance-benchmark/J73/J73\_fs\_benchmark\_data.json Writing benchmark data to /bulk9/data/dev\_projects/CS-peformance-benchmark/J73/J73\_fs\_benchmark\_data.csv \`\`\` To view the original Benchmark job that a benchmark was created from, right click on the column header and select “Show job in sidebar”. The JSON and CSV results can also be downloaded from this context menu. ![](https://guide.cryosparc.com/files/GKXz2KBfnpU9d4d8Rbgk) \## Performance Benchmarking Entire Jobs with the Extensive Validation Job The Extensive Workflow job is now called the Extensive Validation job (v4.3.0+). The Extensive Validation job is a job that creates and queues other jobs in a pre-defined workflow. Workflows available are for the \[EMPIAR-10025\](https://www.ebi.ac.uk/empiar/EMPIAR-10025/) and \[EMPIAR-10305\](https://www.ebi.ac.uk/empiar/EMPIAR-10305/) datasets, which are downloaded when the job is run. If you run the Extensive Validation job in "Benchmark" mode, each job defined in the workflow will run in sequence. This will allow you to compare the overall performance of each job in the Benchmark UI, along with the CPU, Filesystem, and GPU performance benchmarks. First, create an “Extensive Validation” job and select “Benchmark” as the value for the “Run Mode” parameter: ![](https://guide.cryosparc.com/files/Eu7lgVvdwpNbIhWYZecp) In “Benchmark Mode”, jobs that support multi-GPU parallelization (such as Patch Motion Correction, Patch CTF Estimation, and 2D Classification) can be allocated multiple GPUs. To allocate multiple GPUs, specify a number greater than 1 for the “Number of GPUs to use” parameter field, and either select a lane or specify the exact GPUs using the “Run on specific GPUs” tab in the Resource Selection panel. ![](https://guide.cryosparc.com/files/fXvQFlTNgJw8GPaXnIo5) For more information on the jobs that are launched by the Extensive Validation job in benchmark mode, see the Extensive Validation documentation here: {% content-ref url="/pages/-MNhrOW1ytZ8-U3WkV0r" %} \[Guide: Verify CryoSPARC Installation with the Extensive Validation Job (v4.3+)\](/setup-configuration-and-management/software-system-guides/tutorial-verify-cryosparc-installation-with-the-extensive-workflow-sysadmin-guide.md) {% endcontent-ref %} ## Appendix ### Drop the page cache First, instruct the kernel to \[write dirty pages to disk\](https://linux.die.net/man/8/sync), then \[drop the page cache\](https://www.kernel.org/doc/Documentation/sysctl/vm.txt): \`sudo bash -c 'sync; echo 1 > /proc/sys/vm/drop\_caches'\` --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/setup-configuration-and-management/software-system-guides/guide-performance-benchmarking-v4.3.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. --- # Unknown \> For the complete documentation index, see \[llms.txt\](https://guide.cryosparc.com/llms.txt). Markdown versions of documentation pages are available by appending \`.md\` to page URLs; this page is available as \[Markdown\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-3d-classification-beta.md). # Job: 3D Classification ## Description 3D Classification, first introduced in v3.3, can help discover discrete heterogeneity in single particle cryo-EM datasets. This job currently implements a version of 3D classification \*without alignment —\* a classification routine that can complement the \[Heterogeneous Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-heterogeneous-refinement) and \[3D Variability\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-3d-variability) jobs in finding new discrete classes of data. In CryoSPARC v4.0, 3D Classification was updated with several notable improvements, including FSC regularization, focus and solvent mask inputs, new convergence criteria, and a number of new diagnostic plots and outputs. {% hint style="info" %} Note that in CryoSPARC v4.0+, cloning a 3D classification job that was created in CryoSPARC v3.3 will fail to launch due to a change in the inputs and parameters of the job type. Instead, please create a 3D classification job from scratch in v4.0 and re-connect the desired inputs and set parameters. {% endhint %} Under the hood, 3D Classification uses a combination of Online and Full-Batch Expectation Maximization (O-EM, and F-EM, respectively). These algorithms alternate between (1) computing the most likely class assignments for each particle image in a batch based on known 3D class volumes, and (2) updating each 3D volume based on these assignments. Please also refer to the \[3D Classification tutorial\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/tutorial-3d-classification-beta), which has been updated for v4.1 with new considerations and example datasets. ## Input \* Particles (with \`alignments3D\`) \* \\\[Optional\] Initial Volumes \* To be used with the \`input\` initialization mode. The number of initial volumes must match the number of classes. \* \\\[Optional\] Solvent mask \* If not supplied, a solvent mask is computed by dilating and soft-padding the consensus volume. \* \\\[Optional\] Focus mask \* If not supplied, only the solvent mask will be used (i.e., the focus mask will be set to a volume of all ones). ## Commonly Adjusted Parameters ### Number of classes Number of classes to use in job. Note that the 3D Classification requires far less computational effort than jobs which perform particle alignments (such as \[Heterogeneous Refinement\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-heterogeneous-refinement)), so a far greater number of classes can be requested for the same computational cost. ### Filter resolution {% hint style="info" %} In v4.5 this parameter name and its default value changed from \`Target resolution\`, default 6 Å, to its current name of \`Filter resolution\`, default unset. {% endhint %} Classification is performed at this resolution. This parameter must be set for the classification to run. Results are best when the resolution is just high enough to see the difference of interest. For instance: \* 3-6 Å for small changes in density (presence/absence of ligand), \* 6-10 Å for conformational changes between one domain in relation to another, \* and >10 Å for presence/absence of a domain or binding partner. For more information on selecting a filter resolution, \[see the TRPV5 case study\](https://guide.cryosparc.com/processing-data/tutorials-and-case-studies/case-study-pseudosymmetry-in-trpv5-and-calmodulin-empiar-10256#filter-resolution-and-hard-classification). This parameter also controls the box size and pixel size of the output class volumes. To reconstruct classes at their extracted box size, use the \[Heterogeneous Reconstruction Only job\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-heterogeneous-reconstruction-only). ### Use latent mixing coefficients {% hint style="info" %} This parameter is new in CryoSPARC v5.0, and is turned off by default. In older versions of CryoSPARC, 3D Classification behaves in the same way as if this parameter is turned off. {% endhint %} By default, 3D Classification assigns a particle amongst classes based solely on how well the particle matches each class's volume. This corresponds to assuming that there is no available information that indicates how large each class is. When this parameter is turned on, 3D Classification instead treats the current class sizes at a given iteration as a representative example of the underlying, true class sizes (formally, it applies a prior over class posteriors based on the current sizes of classes). This means that when a particle matches equally well to the volumes of two classes, the algorithm assigns a higher probability to the larger class. Turning this parameter on encourages diverse class sizes and reduces the likelihood of encountering a great number of similar-looking classes with equal particle counts. ### O-EM learning rate init For a fixed O-EM batch size and epoch value, larger values will generally result in fewer populated classes. ### Symmetry Enforce point-group symmetry during back-projection of every class volume. ### Generate solvent mask from consensus {% hint style="info" %} This parameter is new in CryoSPARC v5.0. In older versions of CryoSPARC, the if a focus mask was provided and a solvent mask was not, the focus mask would be used for both. {% endhint %} By default, 3D Classification automatically generates a solvent mask from the consensus volume. If this parameter is turned off, a spherical mask is used instead. ### Use FSC to filter each class FSC filtering may be turned off to match the filtering behaviour of 3D classification in CryoSPARC v3.3. ### Convergence criterion (%) Primary stopping criterion — percentage of particles that have switched classes across F-EM iterations. Increasing this value may result in ‘early stopping’ of the optimization. ### RMS Density change convergence check {% hint style="info" %} The default value for this parameter changed in v4.5 {% endhint %} If some particles have high probability of being in two or more different classes, the primary switching criterion may result in several F-EM iterations where a substantial number of particles switch classes but the class volumes do not differ significantly. To prevent unnecessary computation, this secondary criterion tracks the root mean square difference of the real-space class volumes across iterations. The job will converge when either criterion is satisfied. ### Per-particle scale Per-particle optimization can be turned off and scales can be set to their upstream values (\`input\`) or to a constant value of 1.0 (\`none\`). ### Force hard classification Turn off weighted back projection — this may improve performance for small(er) targets where the standard optimization may ‘smear’ a portion of particles across several classes. ### Reorder classes by size {% hint style="info" %} The default value for this parameter changed in v4.5 {% endhint %} By default, output classes are not reordered during 3D Classification. This means that the output Class 0 refers to the same volume as class 0 during classification. If \`Reorder classes by size\` is turned on, classes will be reordered according to their size (i.e., number of assigned particles) at the end of classification, prior to output generation. This means that the output Class 0 is the class with the most particles, which is not necessarily the same as class 0 while the job was running. To avoid potential confusion regarding class outputs, this option must be turned off if \[\`Keep intermediate results\`\](#keep-intermediate-results) is turned on. ### Keep intermediate results {% hint style="info" %} In versions of CryoSPARC older than v5.0, this parameter is called \`Output data after every F-EM iter\` {% endhint %} This option may be useful for larger datasets where one may want to monitor the 3D volumes prior to the completion of the job. This option can only be turned on if \[\`Reorder classes by size\`\](#reorder-classes-by-size) is turned off. ## Output \* All particles \* All volumes \* \*\*This output is new in CryoSPARC v4.5+\*\* and is a volumes group output. It includes a \`series\` result that contains a downloadable zip file of all volumes. See \[documentation for volumes groups\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/utilities/split-volumes-group) for more details. \* Solvent mask (passthrough input or auto-generated) \* Consensus volume \* Focus mask (passthrough input if provided) \* Particles for each class \* 3D volumes for each class ## Common next steps \* \[Job: Heterogeneous Reconstruction Only\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/3d-refinement/job-heterogeneous-reconstruction-only) \* This job can be useful to reconstruct classes at a larger box size than the one set by the 3D classification target resolution. \* \[Job: Regroup 3D\](https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-regroup-3d) \* For large sets of classes (e.g., 50+), this job can quickly group these classes into a smaller set of 'superclasses' based on real-space voxel correlations. \* Further classification of subsets of classes ## \*New in CryoSPARC v4.0+\* A number of significant improvements to 3D Classification were added in CryoSPARC v4.0. We list them below. ### Algorithmic Changes \* \*\*Per-particle scale optimization (v4.1+)\*\* \* By default, 3D Classification will perform per-particle scale optimization before starting the main EM classification loop. \* \*\*FSC-based filtering (v4.0+)\*\* \* By default, during both O-EM and F-EM iterations, 3D Classification will filter each class volume by its intra-class FSC curve. \* \*\*Convergence criteria (v4.0+)\*\* \* F-EM iterations will conclude when one of two stopping criteria is met: \* % of particles that switch classes (primary stopping criterion) \* weighted mean RMS density change falls below a threshold (optional, secondary criterion) \* \*\*Separate focus and solvent mask inputs (v4.0+)\*\* \* 3D Classification accepts two different types of masks. A solvent mask, $$S$$, and a focus mask, $$F$$. During optimization we use the following real-space volume for all likelihood computations of class $$k$$: $$ V\\\_k \\leftarrow S \\\* (F \\\* V\\\_k + (1-F)\\\*\\bar{V}), $$ where $$\\bar{V}$$ is the consensus reconstruction. If $$F$$ is not provided, we set $$F = 1$$ and apply $$V\\\_k \\leftarrow S \\\* V\\\_k$$. Otherwise, we also plot real-space slices and projections of the mask overlayed on the consensus volume map: ![](https://guide.cryosparc.com/files/J7Qx5Q7tsZsJ6R3qlkaf) Focus mask overlayed on real-space slices. \* \*\*Filtered consensus volume output (v4.4+)\*\* \* The consensus map is now filtered in accordance to its FSC. The resulting map is output by the job for inspection. ### Diagnostic plots Starting with CryoSPARC v4.0, 3D Classification outputs several new diagnostic plots listed below. #### \*\*Per-particle Class ESS Histogram (added in v4.0)\*\* \*This histogram can help diagnose poor classification results by showing if some particles have significant probability mass in more than one class. The ESS (Effective Sample Size) is a measure of how many classes each particle appears to belong to with significant probability. And ESS of 1.0 indicates that a particle is completely confidently assigned to only one class. An ESS of 2.0 would mean that a particle belongs with substantial probability to two classes. When many particles have a large ESS (> 1), this indicates that there is significant uncertainty in classification, any the classes may be overlapping or similar.\* ![](https://guide.cryosparc.com/files/C1vMORKlWwte6eqHLFDG) \#### \*\*Difference from Consensus Real-Space Slices (added in v3.3, updated in v4.0)\*\* \*This plot shows the real-space difference between the consensus map and each class map, regularized by the class FSC (if FSC regularization is turned on). This can quickly show areas of heterogeneity.\* ![](https://guide.cryosparc.com/files/xEzUo0qYtDB1sfYZjbVM) \#### \*\*Class Flow Diagram (added in v4.0, updated in v4.1)\*\* \*This diagram shows how many particles switched classes across F-EM iterations (output starts at the second F-EM iteration). An edge, (i,j), is drawn with a thickness, colour, and opacity defined by the amount of particles that switch from class i to class j.\* ![](https://guide.cryosparc.com/files/DuWb3Nwguv3RXxtrShE5) \#### \*\*Class Flow Matrix (added in v4.1)\*\* \*This diagram visualizes class flow in a matrix format. Each column represents a 1D distribution of the particles in a given class at the current F-EM iteration. Each row represents the class which the particles belonged to at the previous iteration. In other words, each square in this grid represents an edge in the bipartite class flow graph above. This form of class flow can be useful in visualizing 'minor' edges that are difficult to see in the bipartite graph, and it can greatly improve clarity for class flow with large (25+) numbers of classes.\* !\[\](/files/15f2alsThANWFzUnzvkD) #### \*\*Class Assignment Histogram (added in v3.3, updated in v4.0)\*\* \*This histogram now includes both total assignments and the ‘effective size’ of the class. The latter is a sum of the probability mass in that class. When the assignments and effective size bars are differently sized, this indicates that there is uncertainty in the classification, as many particles have probabilities that are spread out between classes (an effect included in the effective size) compared to the class where they have the maximum probability (the assignments).\* ![](https://guide.cryosparc.com/files/GSqy7MzsRdOAstBk3AUI) \--- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the \`ask\` query parameter: \`\`\` GET https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/variability/job-3d-classification-beta.md?ask= \`\`\` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections. ---