# Table of Contents - [The Internals of Deno](#the-internals-of-deno) - [Audience | The Internals of Deno](#audience-the-internals-of-deno) - [Reviews | The Internals of Deno](#reviews-the-internals-of-deno) - [Translations | The Internals of Deno](#translations-the-internals-of-deno) - [Formats | The Internals of Deno](#formats-the-internals-of-deno) - [Contents | The Internals of Deno](#contents-the-internals-of-deno) - [1.1 Introduction | The Internals of Deno](#1-1-introduction-the-internals-of-deno) - [1.0 Cover page | The Internals of Deno](#1-0-cover-page-the-internals-of-deno) - [1.5 The Deno Company | The Internals of Deno](#1-5-the-deno-company-the-internals-of-deno) - [1.6 Deno's source | The Internals of Deno](#1-6-deno-s-source-the-internals-of-deno) - [1.4 Releases | The Internals of Deno](#1-4-releases-the-internals-of-deno) - [1.2 History of Deno | The Internals of Deno](#1-2-history-of-deno-the-internals-of-deno) - [1.3 About Deno | The Internals of Deno](#1-3-about-deno-the-internals-of-deno) - [2.1 Architecture | The Internals of Deno](#2-1-architecture-the-internals-of-deno) - [1.7 What's next | The Internals of Deno](#1-7-what-s-next-the-internals-of-deno) - [2.6 TSC/SWC | The Internals of Deno](#2-6-tsc-swc-the-internals-of-deno) - [2.3 Programming Languages | The Internals of Deno](#2-3-programming-languages-the-internals-of-deno) - [2.0 Cover page | The Internals of Deno](#2-0-cover-page-the-internals-of-deno) - [2.7 Rusty_v8 | The Internals of Deno](#2-7-rusty-v8-the-internals-of-deno) - [2.2 Overall architecture | The Internals of Deno](#2-2-overall-architecture-the-internals-of-deno) - [2.5 OPs | The Internals of Deno](#2-5-ops-the-internals-of-deno) - [2.10 What's next | The Internals of Deno](#2-10-what-s-next-the-internals-of-deno) - [3.1 Threading model | The Internals of Deno](#3-1-threading-model-the-internals-of-deno) - [3.4 What's next | The Internals of Deno](#3-4-what-s-next-the-internals-of-deno) - [4.4 What's next | The Internals of Deno](#4-4-what-s-next-the-internals-of-deno) - [2.4 Deno components | The Internals of Deno](#2-4-deno-components-the-internals-of-deno) - [3.0 Cover page | The Internals of Deno](#3-0-cover-page-the-internals-of-deno) - [4.2 Print | The Internals of Deno](#4-2-print-the-internals-of-deno) - [2.9 V8 | The Internals of Deno](#2-9-v8-the-internals-of-deno) - [4.0 Cover page | The Internals of Deno](#4-0-cover-page-the-internals-of-deno) - [3.2 Default threading model | The Internals of Deno](#3-2-default-threading-model-the-internals-of-deno) - [5.1 Hello world program | The Internals of Deno](#5-1-hello-world-program-the-internals-of-deno) - [4.1 The bridge | The Internals of Deno](#4-1-the-bridge-the-internals-of-deno) - [3.3 Asynchronous green threads | The Internals of Deno](#3-3-asynchronous-green-threads-the-internals-of-deno) - [5.2 Basic hello world | The Internals of Deno](#5-2-basic-hello-world-the-internals-of-deno) - [4.3 Encode and decode | The Internals of Deno](#4-3-encode-and-decode-the-internals-of-deno) - [5.0 Cover page | The Internals of Deno](#5-0-cover-page-the-internals-of-deno) - [2.8 Tokio | The Internals of Deno](#2-8-tokio-the-internals-of-deno) - [5.5 CLI Factory | The Internals of Deno](#5-5-cli-factory-the-internals-of-deno) - [5.4 Module Specifier | The Internals of Deno](#5-4-module-specifier-the-internals-of-deno) - [5.18 What's next | The Internals of Deno](#5-18-what-s-next-the-internals-of-deno) - [5.9 Run main module | The Internals of Deno](#5-9-run-main-module-the-internals-of-deno) - [5.10 Load module | The Internals of Deno](#5-10-load-module-the-internals-of-deno) - [6.1 Imports and ops | The Internals of Deno](#6-1-imports-and-ops-the-internals-of-deno) - [5.16 Instantiate module | The Internals of Deno](#5-16-instantiate-module-the-internals-of-deno) - [5.11 Recursive module loading | The Internals of Deno](#5-11-recursive-module-loading-the-internals-of-deno) - [5.3 Main program of Deno | The Internals of Deno](#5-3-main-program-of-deno-the-internals-of-deno) - [7.4 What's next | The Internals of Deno](#7-4-what-s-next-the-internals-of-deno) - [6.10 What's next | The Internals of Deno](#6-10-what-s-next-the-internals-of-deno) - [5.6 Permissions | The Internals of Deno](#5-6-permissions-the-internals-of-deno) - [6.2 Hello world program v2 | The Internals of Deno](#6-2-hello-world-program-v2-the-internals-of-deno) - [6.0 Cover page | The Internals of Deno](#6-0-cover-page-the-internals-of-deno) - [6.9 Debug logs | The Internals of Deno](#6-9-debug-logs-the-internals-of-deno) - [5.14 Transpile | The Internals of Deno](#5-14-transpile-the-internals-of-deno) - [7.1 Introduction | The Internals of Deno](#7-1-introduction-the-internals-of-deno) - [6.7 Evaluate module | The Internals of Deno](#6-7-evaluate-module-the-internals-of-deno) - [5.13 File fetching | The Internals of Deno](#5-13-file-fetching-the-internals-of-deno) - [6.3 Module graph with imports | The Internals of Deno](#6-3-module-graph-with-imports-the-internals-of-deno) - [7.0 Cover page | The Internals of Deno](#7-0-cover-page-the-internals-of-deno) - [6.8 Sync OPs | The Internals of Deno](#6-8-sync-ops-the-internals-of-deno) - [Afterword | The Internals of Deno](#afterword-the-internals-of-deno) - [7.3 Session storage | The Internals of Deno](#7-3-session-storage-the-internals-of-deno) - [5.8 JS Runtime | The Internals of Deno](#5-8-js-runtime-the-internals-of-deno) - [5.12 Module graphs | The Internals of Deno](#5-12-module-graphs-the-internals-of-deno) - [5.15 Register / compile module | The Internals of Deno](#5-15-register-compile-module-the-internals-of-deno) - [6.4 Transpile | The Internals of Deno](#6-4-transpile-the-internals-of-deno) - [5.7 Main Worker | The Internals of Deno](#5-7-main-worker-the-internals-of-deno) - [5.17 Evaluate module | The Internals of Deno](#5-17-evaluate-module-the-internals-of-deno) - [7.2 Local storage | The Internals of Deno](#7-2-local-storage-the-internals-of-deno) - [6.5 Registration and instantiation | The Internals of Deno](#6-5-registration-and-instantiation-the-internals-of-deno) - [6.6 Registration of ops | The Internals of Deno](#6-6-registration-of-ops-the-internals-of-deno) - [1.0 Cover page | The Internals of Deno](#1-0-cover-page-the-internals-of-deno) - [2.0 Cover page | The Internals of Deno](#2-0-cover-page-the-internals-of-deno) - [7.0 Cover page | The Internals of Deno](#7-0-cover-page-the-internals-of-deno) - [5.0 Cover page | The Internals of Deno](#5-0-cover-page-the-internals-of-deno) - [4.0 Cover page | The Internals of Deno](#4-0-cover-page-the-internals-of-deno) - [3.0 Cover page | The Internals of Deno](#3-0-cover-page-the-internals-of-deno) - [6.0 Cover page | The Internals of Deno](#6-0-cover-page-the-internals-of-deno) - [Afterword | The Internals of Deno](#afterword-the-internals-of-deno) --- # The Internals of Deno ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252F7FqkU5giGDJ0botCuvMk%252Fdeno%2520book%2520cover%25202024.png%3Falt%3Dmedia%26token%3De23b1d3b-b9ac-4853-ad58-68d0dae287b9&width=768&dpr=4&quality=100&sign=88fd48ca&sv=2) [](https://choubey.gitbook.io/internals-of-deno#the-internals-of-deno) The INTERNALS OF DENO ------------------------------------------------------------------------------------------------- _The_ _**one and only (free or not) book**_ _in market covering the internals of Deno._ #### [](https://choubey.gitbook.io/internals-of-deno#by-mayank-choubey) By Mayank Choubey #### [](https://choubey.gitbook.io/internals-of-deno#id-3rd-edition-january-2024) 3rd edition - JANUARY 2024 **Updated for Deno 1.40** * * * If you have any feedback for the book, including reviews, please send your note here: author.mayank.c@gmail.com. [NextAudience](https://choubey.gitbook.io/internals-of-deno/audience) Last updated 1 year ago This site uses cookies to deliver its service and to analyze traffic. By browsing this site, you accept the [privacy policy](https://policies.gitbook.com/privacy/cookies) . AcceptReject --- # Audience | The Internals of Deno This book is designed for individuals who possess a strong foundation in JavaScript and TypeScript and are eager to explore the internal mechanics of Deno. Specifically, this book caters to readers who: * Have experience working with Deno and its capabilities, including creating and managing projects, modules, and dependencies * Have written and executed programs within the Deno environment, leveraging its features and tools * Demonstrate a keen interest in understanding the underlying architecture and operational mechanics of Deno, including its runtime environment, security features, and performance optimization techniques * Seek to gain insight into how Deno manages code execution, including the processing of TypeScript and JavaScript files, module resolution, and error handling * Aspire to comprehend the intricate architecture that underlies Deno, including its component interactions and internal workflows * Desire to unveil the hidden mechanisms that power Deno's actions, including its use of V8, Rust, and other underlying technologies * Yearn to gain a deeper understanding of Deno's design principles, trade-offs, and future development directions **Important Note** This book assumes prior knowledge of Deno, JavaScript, and TypeScript. It is not intended as an introductory resource for those new to Deno or programming. Instead, its purpose is to provide an in-depth exploration of Deno's internal workings, addressing the questions and curiosities of readers already familiar with the basics. Numerous excellent resources are available for those looking to get started with Deno, and this book builds upon that foundation. [PreviousThe Internals of Deno](https://choubey.gitbook.io/internals-of-deno) [NextReviews](https://choubey.gitbook.io/internals-of-deno/reviews) Last updated 1 year ago --- # Reviews | The Internals of Deno Review by a **Sr. Developer at Walmart** _While conducting a search for in-depth information on Deno's internal workings, I came across this comprehensive book. I was pleased to find that it provides a clear and concise explanation of the details I needed to understand, making it an accessible resource for anyone seeking to explore Deno's inner mechanics. This book is a must-read for individuals curious about the intricacies of Deno's architecture and operations, providing valuable insights that will deepen their understanding of this powerful technology._ Review by a **Sr. QA at a leading streaming provider** _As a Quality Assurance professional, I have always been intrigued by the inner workings of various technologies. My fascination with the complexity of internal mechanisms led me to explore Deno's internals, and this book served as a valuable resource in that endeavor. Although some concepts pushed the boundaries of my existing knowledge, this book provided me with a deeper understanding of Deno's architecture and operations, further solidifying my appreciation for the intricacies of software design._ * * * If you find this book helpful in your journey to understand Deno's internals, I encourage you to share your thoughts and feedback with me. Your input is invaluable in refining and improving the content for future readers. Please feel free to email your comments, suggestions, and testimonials to _author.mayank.c@gmail.com_. I welcome your feedback and look forward to hearing about your experiences. When submitting your review, you may also include any personal information you'd like to share, such as your name and profession, which will be featured in the testimonial section. Your support and contributions are greatly appreciated, and I will regularly update the testimonials page with new reviews to inspire and motivate other learners. [PreviousAudience](https://choubey.gitbook.io/internals-of-deno/audience) [NextTranslations](https://choubey.gitbook.io/internals-of-deno/translations) Last updated 1 year ago --- # Translations | The Internals of Deno The translations for this book has been available in the following languages: * **Chinese**: Translation done by JiQingYun. The book is available [here](https://mp.weixin.qq.com/mp/appmsgalbum?__biz=MzkyMjQzNjMxNQ==&action=getalbum&album_id=3106828545089208325&uin=&key=&devicetype=iMac+MacBookPro18%2C3+OSX+OSX+14.1.2+build(23B92)&version=13080512&lang=zh_CN&nettype=WIFI&ascene=7&session_us=gh_a4b3390dab25&fontScale=100) . * **Spanish**: Inviting people to translate book in Spanish For translation permissions, please email to author.mayank.c@gmail.com. [PreviousReviews](https://choubey.gitbook.io/internals-of-deno/reviews) [NextFormats](https://choubey.gitbook.io/internals-of-deno/formats) Last updated 1 year ago --- # Formats | The Internals of Deno If you like to download a PDF for this book, you can use the following link: [![Logo](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2Fcfl.dropboxstatic.com%2Fstatic%2Fmetaserver%2Fstatic%2Fimages%2Ffavicon.ico&width=20&dpr=4&quality=100&sign=fd7c6bec&sv=2)InternalsOfDeno.pdfDropbox](https://www.dropbox.com/scl/fi/79bd2jcfekkr8zv5i6rv4/InternalsOfDeno.pdf?rlkey=o7l4z1lplpeo8z2yx0e9pctz3&dl=0) **Note:** The PDF formatting is not correct in some places. [PreviousTranslations](https://choubey.gitbook.io/internals-of-deno/translations) [NextContents](https://choubey.gitbook.io/internals-of-deno/content) Last updated 1 year ago --- # Contents | The Internals of Deno [Chapter 1 - INTRODUCTION](https://choubey.gitbook.io/internals-of-deno/introduction/chapter-cover-page) [Chapter 2 - ARCHITECTURE](https://choubey.gitbook.io/internals-of-deno/architecture/chapter-cover-page) [CHAPTER 3 - THREADING MODEL](https://choubey.gitbook.io/internals-of-deno/threading-model/threading-model) [CHAPTER 4 - BRIDGE](https://choubey.gitbook.io/internals-of-deno/bridge/chapter-cover-page) [CHAPTER 5 - FOUNDATIONS](https://choubey.gitbook.io/internals-of-deno/foundations/chapter-cover-page) [CHAPTER 6 - IMPORTS AND OPS](https://choubey.gitbook.io/internals-of-deno/import-and-ops/chapter-cover-page) [CHAPTER 7 - LOCAL AND SESSION STORAGE](https://choubey.gitbook.io/internals-of-deno/chapter-7-local-and-session-storage/7.0-cover-page) [PreviousFormats](https://choubey.gitbook.io/internals-of-deno/formats) [Next1.0 Cover page](https://choubey.gitbook.io/internals-of-deno/introduction/chapter-cover-page) Last updated 1 year ago --- # 1.1 Introduction | The Internals of Deno Deno is a secure runtime environment designed specifically for executing JavaScript and TypeScript code. It was released nearly a decade after the initial release of Node.js, building on the foundations laid by its predecessor. Node.js, a widely adopted runtime for JavaScript, enables developers to utilize JavaScript on the server-side, contributing to its popularity. JavaScript, a programming language with a rich history, remains the most extensively used language globally. Node.js has a vibrant community, offering a wide range of robust tools and libraries, which facilitate the development of web applications. Its adaptability, user-friendly nature, and community support have contributed to its widespread adoption. Deno, developed by Ryan Dahl, the creator of Node.js, shares similarities with its predecessor but also introduces distinct features. Both serve as server-side runtimes, executing TypeScript and JavaScript (Node.js supports TypeScript through NPM packages). Deno is often regarded as the next generation of runtime environments, building on the experience and knowledge gained from Node.js. While Node.js has established itself in the industry, Deno, still in its early stages (four years old as of August 2024, with 45 releases from version 1.0 to 1.45), has yet to reach maturity. Deno aims to address the limitations and challenges faced by Node.js, providing a modern and secure alternative for developers. Before exploring Deno's internals, it is essential to understand its background and the motivations behind its creation. This includes examining the limitations of Node.js and how Deno addresses these drawbacks. This chapter provides a historical context and inspiration behind Deno's development, paving the way for a deeper understanding of its architecture in the subsequent chapters. [](https://choubey.gitbook.io/internals-of-deno/introduction/introduction#chapter-contents) Chapter Contents ----------------------------------------------------------------------------------------------------------------- [1.2 History of Deno](https://choubey.gitbook.io/internals-of-deno/introduction/history-of-deno) [1.3 About Deno](https://choubey.gitbook.io/internals-of-deno/introduction/about) [1.4 Releases](https://choubey.gitbook.io/internals-of-deno/introduction/releases) [1.7 What's next](https://choubey.gitbook.io/internals-of-deno/introduction/whats-next) [Previous1.0 Cover page](https://choubey.gitbook.io/internals-of-deno/introduction/chapter-cover-page) [Next1.2 History of Deno](https://choubey.gitbook.io/internals-of-deno/introduction/history-of-deno) Last updated 1 year ago --- # 1.0 Cover page | The Internals of Deno ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252FiQRizCbdcTvUZKCTSji3%252F2.png%3Falt%3Dmedia%26token%3Da92e7496-b742-4bf9-83d3-bf8fbe96f91d&width=768&dpr=4&quality=100&sign=1e3e587d&sv=2) [PreviousContents](https://choubey.gitbook.io/internals-of-deno/content) [Next1.1 Introduction](https://choubey.gitbook.io/internals-of-deno/introduction/introduction) Last updated 1 year ago --- # 1.5 The Deno Company | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/introduction/1.5-the-deno-company#the-seed-funding) The seed Funding ------------------------------------------------------------------------------------------------------------------------- On March 29, 2021, the Deno team announced the establishment of Deno Company, a significant milestone in Deno's development. The company secured a seed investment of $4.3 million, later increased to $4.9 million, from a group of investors including Dan Scholnick from Four Rivers Ventures, Guillermo from Rauch Capital, Lee Jacobs from Long Journey Ventures, the Mozilla Corporation, Shasta Ventures, and Ben Noordhuis. Deno emerges as a revitalizing force within the technological landscape, breathing fresh vitality into the existing ecosystem. Its mission is to furnish a contemporary, efficient programming system that remains aligned with browser APIs. Unlike a monolithic entity, Deno constitutes an amalgamation of technologies that possess the potential for versatile adaptation across a spectrum of requirements. Recognizing that not every scenario involving server-side JavaScript necessitates interaction with the file system, the underlying framework allows for the removal of extraneous connections. This ingenious mechanism empowers developers to forge tailored runtimes for diverse applications: think of Electron-style graphical user interfaces, Cloudflare Worker-style Serverless Functions, or even embedded scripts tailored for databases. The seed funding enables the Deno team to expand its engineering team, ensuring dedicated resources for platform development, issue resolution, bug fixing, and timely software releases. This investment underscores Deno's evolution into a robust foundation upon which others can confidently construct their innovative creations. With the necessary resources, the team is poised to systematically address issues, rectify bugs, and ensure punctual software releases, solidifying Deno's position as a revitalizing force in the technological landscape. [](https://choubey.gitbook.io/internals-of-deno/introduction/1.5-the-deno-company#the-series-a) The Series A ----------------------------------------------------------------------------------------------------------------- While the primary Deno development team focused on building the Deno runtime, a segment of the Deno team worked on a project called Deno Deploy Deno Deploy is a decentralized framework designed to execute JavaScript, TypeScript, and WebAssembly in proximity to users, at the edges of the network. This system is closely integrated with the V8 runtime, ensuring low latency and eliminating unnecessary complexity. The framework's primary objective is to provide a platform for developers to build applications that can be executed close to the user, reducing latency and improving overall performance. Deno Deploy offers several key features that make it an attractive platform for developers. These features include: * _Support for JavaScript, TypeScript, and WebAssembly_: Deno Deploy allows developers to write applications in the language of their choice, without requiring significant modifications or workarounds. * _Decentralized Architecture_: Deno Deploy's decentralized architecture enables applications to be executed at the edges of the network, closer to the user. This reduces latency and improves overall performance. * _Integration with V8 Runtime_: Deno Deploy is closely integrated with the V8 runtime, ensuring low latency and eliminating unnecessary complexity. * _Simplified Development Process_: Deno Deploy streamlines the development process, allowing developers to focus on writing code rather than managing infrastructure. * _GitHub Integration_: Deno Deploy integrates seamlessly with GitHub, enabling developers to push, review, and merge code changes efficiently. * _High Performance_: Deno Deploy is designed to provide high performance, deploying applications in under a second and serving content efficiently across the globe. **Funding and Future Development** The success of Deno Deploy led to a significant influx of funding, amounting to $21 million, through a series of investment rounds. This funding aims to transform the Deploy project into a fully-fledged product, catering to a wider range of needs and users. The Deno company plans to utilize this funding to engage top-tier engineers, establish a dependable infrastructure for businesses of all sizes, explore commercial applications of Isolate Cloud technology across various industries, and drive growth and market prospects. [Previous1.4 Releases](https://choubey.gitbook.io/internals-of-deno/introduction/releases) [Next1.6 Deno's source](https://choubey.gitbook.io/internals-of-deno/introduction/1.6-denos-source) Last updated 1 year ago --- # 1.6 Deno's source | The Internals of Deno The source code for Deno is maintained and hosted within the denoland organization. The official repository for Deno is located on GitHub, a popular web-based platform for version control and collaboration. To access the Deno source code, you can visit the following URL: [![Logo](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2Fgithub.com%2Ffluidicon.png&width=20&dpr=4&quality=100&sign=2b904219&sv=2)GitHub - denoland/deno: A modern runtime for JavaScript and TypeScript.GitHub](https://github.com/denoland/deno) ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252FLV8vujCKAF72nGUy4wHm%252FScreenshot%25202024-07-12%2520at%25202.25.53%25E2%2580%25AFPM.png%3Falt%3Dmedia%26token%3Dd5dc7a21-683e-49fe-bf72-d8495b63df9b&width=768&dpr=4&quality=100&sign=ece3d73c&sv=2) [Previous1.5 The Deno Company](https://choubey.gitbook.io/internals-of-deno/introduction/1.5-the-deno-company) [Next1.7 What's next](https://choubey.gitbook.io/internals-of-deno/introduction/whats-next) Last updated 1 year ago --- # 1.4 Releases | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/introduction/releases#first-release) First release ------------------------------------------------------------------------------------------------------- The concept of Deno was first introduced in 2018, followed by an intensive development phase that lasted approximately two years. The first official release of Deno, version 1.0.0, occurred on May 13, 2020. Although this initial release contained many bugs, it demonstrated the effectiveness and functionality of the platform. Following the successful launch, Deno adopted a regular release schedule, issuing both major and minor updates on a predetermined timetable. This iterative approach enabled Deno to evolve and improve over time, addressing issues and enhancing its capabilities with each successive release. This consistent development and release pattern have contributed to Deno's growth and maturation as a robust and reliable platform. [](https://choubey.gitbook.io/internals-of-deno/introduction/releases#subsequent-releases) Subsequent releases ------------------------------------------------------------------------------------------------------------------- The development of Deno continues at a steady and deliberate pace, with a focus on sustainability and reliability. The Deno repository on GitHub remains highly active, with frequent updates and contributions from the development team on a nearly daily basis. This consistent activity demonstrates the team's dedication to the project's ongoing advancement. The development team is committed to a meticulous and thoughtful approach to enhancements and releases, ensuring a steady and incremental evolution of Deno's capabilities. This approach prioritizes stability and quality, reflecting the team's dedication to maintaining a high standard of excellence in the project's development. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252FiKzSt5YQP4Z03tZiRCHz%252FScreenshot%25202024-07-12%2520at%25202.15.52%25E2%2580%25AFPM.png%3Falt%3Dmedia%26token%3Dbacaf2ed-dc98-460c-8b5b-b3894f23323b&width=768&dpr=4&quality=100&sign=b0fce944&sv=2) ### [](https://choubey.gitbook.io/internals-of-deno/introduction/releases#release-schedule) Release schedule Since its initial launch, Deno has undergone 45 significant updates, accompanied by numerous smaller revisions. Currently, Deno operates within the 1.x release framework. As of July 2024, discussions are underway regarding the upcoming release of Deno 2.0, expected later in the year. A comprehensive record of all releases can be found on the dedicated GitHub page: [https://github.com/denoland/deno/releases/](https://github.com/denoland/deno/releases/) . These releases, denoted by version numbers such as 1.1.0, 1.2.0, 1.3.0, 1.36.0, and 1.45.0, are issued approximately monthly. Major releases are accompanied by official blog posts, accessible at [https://deno.com/blog](https://deno.com/blog) , which detail advancements and changes. Minor releases, indicated by version numbers like 1.major.minor, occur every 2-8 days, typically following a weekly schedule unless urgent issues require a prompt response. As Deno continues to evolve, its development pace is now more rigorously managed. Each major and minor release integrates new features, solutions, and bug fixes, contributing to the platform's maturation. The scope of these updates is substantial, making it challenging to enumerate each individual change due to their abundance. The frequent releases demonstrate the dedication of the development team to maintaining a robust and reliable platform. The ongoing progress and development of Deno ensure its continued growth and improvement. [Previous1.3 About Deno](https://choubey.gitbook.io/internals-of-deno/introduction/about) [Next1.5 The Deno Company](https://choubey.gitbook.io/internals-of-deno/introduction/1.5-the-deno-company) Last updated 1 year ago --- # 1.2 History of Deno | The Internals of Deno To understand the creation of Deno, a new server-side runtime, we must first examine the circumstances that led to its development. Node.js, a well-established runtime for server-side applications, has been in use for over 13 years and has gained widespread recognition for its reliability and performance. Its popularity has grown significantly over time, earning the trust of prominent corporations such as PayPal, Netflix, Trello, NASA, Twitter, Walmart, and many others, which rely on Node.js to power fast and efficient web applications. Despite Node.js's success, the need for a new runtime emerged. We will explore the reasons behind this decision and examine the motivations that led to the creation of Deno. By understanding the context and rationale behind Deno's development, we can better appreciate its design and features. [](https://choubey.gitbook.io/internals-of-deno/introduction/history-of-deno#whos-behind-deno) Who's behind Deno? ---------------------------------------------------------------------------------------------------------------------- The Deno project is led by Ryan Dahl, the original creator of Node.js. Ryan was at the helm of Node.js for its first 3-4 years before it was transitioned to the Node Package Manager (NPM). After his involvement with Node.js, Ryan pursued other ventures, including a stint at Google. In 2018, Ryan returned to the spotlight with a notable presentation where he openly discussed his concerns and regrets regarding Node.js. He reflected on various aspects of the project that he wished he had handled differently. This presentation marked the introduction of Deno, a new runtime born out of Ryan's determination to address these regrets. The primary goal behind Deno's creation was to develop a modern, secure runtime that would rectify the shortcomings Ryan perceived in Node.js. He sought to create a runtime that would incorporate contemporary practices and offer enhanced security features, thereby mitigating the mistakes of the past. We will now examine the specific areas of regret that motivated Ryan to create Deno. [](https://choubey.gitbook.io/internals-of-deno/introduction/history-of-deno#the-regrets) The regrets ---------------------------------------------------------------------------------------------------------- Ryan Dahl's Reflections on Node.js In 2018, at the JSConf event, Ryan Dahl delivered a presentation discussing his retrospective thoughts on Node.js. This talk, available on YouTube at [https://www.youtube.com/watch?v=M3BM9TB-8yA](https://www.youtube.com/watch?v=M3BM9TB-8yA) , offers valuable insights into the motivations behind creating a new runtime. Examining the key aspects of Ryan's regrets will provide us with a deeper understanding of the initial motivations that led to the development of Deno. By analyzing these factors, we can gain a better understanding of the foundational principles that Deno was built upon. ### [](https://choubey.gitbook.io/internals-of-deno/introduction/history-of-deno#promises) Promises After the initial release of Node.js, Ryan Dahl introduced promises to the platform in June 2009. Although promises were initially integrated into the core of Node.js, they were later removed in February 2010. At that time, the use of promises for async/await operations was still relatively new, and the core Node.js framework relied heavily on callbacks, a pattern that still persists today. In modern programming, promises have become essential for supporting contemporary async/await code. However, a significant portion of the Node.js codebase still relies on callback-based structures. While asynchronous programming techniques have evolved, the original callback-oriented code remains functional and effective. Given the robust and dependable nature of the existing callback-centric foundation, it is unlikely that the established code will be completely replaced with promises or async/await constructs. The stability and reliability of Node.js applications are ensured by this foundation. ### [](https://choubey.gitbook.io/internals-of-deno/introduction/history-of-deno#open-access) Open access Both Node.js and Deno leverage Google's V8 JavaScript engine to execute JavaScript programs. However, they differ in their approaches to security and isolation. V8 operates within a secure sandboxed environment, confining JavaScript code execution and preventing direct access to sensitive system resources. In contrast, Node.js has traditionally taken a more open approach, allowing programs to access a wide range of resources available to user-level processes. This was the case for all Node.js versions prior to v20. Starting from version v20, Node.js has begun to adopt sandboxing for applications, a shift towards enhanced security and isolation. Although this feature is still experimental, it marks a significant change in Node.js' approach. The maturation of this sandboxing feature will likely require considerable time, as it undergoes refinement and enhancement. ### [](https://choubey.gitbook.io/internals-of-deno/introduction/history-of-deno#npm-or-package.json) NPM or package.json The idea behind package.json was influenced by NPM (Node Package Manager), a widely used package manager for Node.js. Initially, Ryan Dahl, the creator of Node.js, supported the use of package.json by enabling the require() function to reference the file for module resolution. As NPM evolved, it became the primary repository for various Node.js modules and expanded its scope to become one of the largest package managers across multiple programming languages. However, a significant drawback of NPM is its private ownership. Originally an independent entity, NPM was acquired by GitHub and later by Microsoft, closely tying Node.js and NPM together. Ryan Dahl regretted the limitation of only being able to source Node.js packages from a single repository, rather than diverse sources. This led to a desire for a more decentralized approach to package management. **Note:** Deno now fully supports NPM (this is required to compete in the market). ### [](https://choubey.gitbook.io/internals-of-deno/introduction/history-of-deno#node_modules) Node\_modules The module resolution algorithm in Node.js became increasingly complex over time. Initially, the concept of vendor modules showed promise, but the use of $NODE\_PATH introduced challenges. Furthermore, this approach deviated significantly from the conventions used in web browsers. Ryan Dahl, the creator of Node.js, aimed to align module resolution with established standards, but this goal was not realized within the Node.js framework. As a result, the module resolution logic in Node.js remains uniquely tailored to its specific environment, diverging from broader web standards. ### [](https://choubey.gitbook.io/internals-of-deno/introduction/history-of-deno#implicit-require) Implicit require Node.js's `require("module")` function, used without the ".js" extension, was a unique feature designed specifically for the Node.js environment. However, this approach differs from how JavaScript works in web browsers. In browser-based JavaScript, omitting the ".js" extension in a script tag's src attribute is not valid. Instead, the module loader must search various file system locations to infer the user's intended module reference, a process that can lead to ambiguity and errors. ### [](https://choubey.gitbook.io/internals-of-deno/introduction/history-of-deno#index.js) Index.js Ryan Dahl was drawn to the name `"index.js"` because of its similarity to `"index.html"`. However, this naming convention introduced unnecessary complexity into the module loading system. This complexity was further exacerbated when the 'require' function began to support package.json files, leading to unforeseen challenges in the module loading process. [](https://choubey.gitbook.io/internals-of-deno/introduction/history-of-deno#conception-of-deno) Conception of Deno ------------------------------------------------------------------------------------------------------------------------ As mentioned earlier, Ryan Dahl expressed his regrets regarding Node.js development. In his JSConf 2018 presentation [https://www.youtube.com/watch?v=M3BM9TB-8yA](https://www.youtube.com/watch?v=M3BM9TB-8yA) , he shared additional concerns. Notably, he regretted granting a private company, NPM, control over third-party modules and the Node.js ecosystem. This decision ultimately led to NPM gaining control over Node.js, which was not the intended outcome. Although some of these regrets, like NPM integration, have been addressed in Deno over time, the experience served as a valuable lesson. Deno's need to access NPM's extensive library of packages (1.4 million) drove the integration. The details of this evolution will be discussed in the next section. These moments of regret inspired Ryan to create a new runtime, designed to make better choices from the start. This runtime, built with modern technologies, prioritizes openness and avoids external control, aiming to correct the mistakes of the past. [Previous1.1 Introduction](https://choubey.gitbook.io/internals-of-deno/introduction/introduction) [Next1.3 About Deno](https://choubey.gitbook.io/internals-of-deno/introduction/about) Last updated 1 year ago --- # 1.3 About Deno | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/introduction/about#deno) Deno ---------------------------------------------------------------------------------- Deno is a user-friendly, contemporary, and highly secure runtime designed for executing JavaScript and TypeScript. Notably, Deno provides native support for TypeScript, making it seamless to work with this language. Under the hood, Deno is built using Rust, a cutting-edge systems programming language known for its performance and security features. The V8 engine, renowned for its speed, drives Deno's JavaScript execution. However, when encountering TypeScript code, Deno translates it into JavaScript using the TSC/SWC compiler duo before executing it with V8. This translation process is noteworthy, as it highlights Deno's ability to bridge the gap between TypeScript and JavaScript. Interestingly, the compilers used in this process differ in their design. The SWC compiler, written in Rust like Deno, prioritizes performance and speed, leveraging Rust's capabilities. In contrast, the TSC compiler, built using JavaScript, completes the task but may be slower compared to SWC. [](https://choubey.gitbook.io/internals-of-deno/introduction/about#unique-features-of-deno) Unique features of Deno ------------------------------------------------------------------------------------------------------------------------ Deno distinguishes itself from Node.js through several key features, many of which are direct responses to the lessons learned and challenges identified in the previous section. These distinctive attributes embody Deno's innovative approach to creating a runtime environment for JavaScript and TypeScript, setting it apart from its predecessor. ### [](https://choubey.gitbook.io/internals-of-deno/introduction/about#sandboxing) Sandboxing Deno prioritizes security as its core principle. By default, it adopts a least-privilege approach, denying access to files, networks, child processes, and the environment unless explicitly granted. Every access request must be explicitly approved, ensuring a high level of control and security. For example, file access can be restricted to specific directories or files using permissions, limiting reading and writing capabilities. Similarly, network access can be controlled to specific IPs, domains, and more. Environmental interactions are also governed by permissions, ensuring a granular level of control. Recent updates to Deno have introduced an additional security feature called "deny," which allows for the comprehensive blocking of various types of access. This denial list functions as an extra layer of security, complementing the allow list and providing a dual-layered protection approach. Deno's rigorous sandboxing technique significantly enhances its security posture. When a Deno process is initiated, it does not inherit the user's permissions, and access privileges can be configured broadly or with fine granularity. This approach creates a highly secure environment for Deno's operation, similar to running within a container without the need for actual container infrastructure. ### [](https://choubey.gitbook.io/internals-of-deno/introduction/about#typescript-support) Typescript support Deno offers built-in support for TypeScript, eliminating the need for external library installations. Deno's integrated TSC/SWC compiler handles TypeScript compilation, converting it to JavaScript for execution on the V8 engine. Note that V8 currently doesn't support TypeScript directly, so this conversion is essential. When initiating Deno, the compiler translates TypeScript code to JavaScript, except in cases involving dynamic imports. This seamless integration simplifies the setup process and ensures a smooth experience for TypeScript projects. The TSC/SWC compiler plays a crucial role in this process, enabling Deno to handle the conversion behind the scenes. This allows developers to focus on writing code without worrying about technicalities, making Deno a runtime that prioritizes developer productivity and ease of use. ### [](https://choubey.gitbook.io/internals-of-deno/introduction/about#single-executable) **Single Executable** Deno is distributed as a self-contained, bundled executable, incorporating a comprehensive suite of development tools. This single executable contains various essential components, including: 1. _Core Engine_: Executes TypeScript (TS) and JavaScript (JS) code, forming the runtime environment's foundation. 2. _Upgrade Manager_: Enables seamless updates, ensuring access to the latest features and enhancements. 3. _Formatter_: Automatically formats code for consistency and readability. 4. _Debugger/Inspector_: Facilitates efficient code debugging and exploration. 5. _Test Framework_: Allows for comprehensive and reliable testing. 6. _Linter_: Analyzes code, highlighting potential errors and deviations from best practices. 7. _Bundler_: Consolidates and packages code and dependencies for streamlined deployment. 8. _Code Coverage_: Provides insights into testing effectiveness. 9. _Additional Tools and Utilities_: Deno's executable includes various other foundational tools, catering to diverse developer needs. Notably, Deno's self-contained nature eliminates the need for additional package managers or tool installations. Upon installation, the entire environment is set up as a singular executable, with no separate dependencies to fetch and install. This streamlined approach enhances the development experience, fostering efficiency and simplicity throughout the software development lifecycle. ### [](https://choubey.gitbook.io/internals-of-deno/introduction/about#third-party-packages) Third-party packages In contrast to Node.js, which relies heavily on its private package manager, NPM, Deno adopts a different strategy. Deno does not require a package manager to host third-party packages. Instead, it supports the use of standard ES modules, which can be accessed via HTTP or locally. This approach offers flexibility in hosting locations, including GitHub, enterprise web servers, personal web servers, or the local file system. Deno caches these packages before use. Initially, Deno's creators were hesitant to support NPM, aiming to follow the Go programming language's approach of acquiring packages from diverse sources. However, as a successor to Node.js, Deno faced the challenge of losing access to the vast library of 1.4 million NPM packages. This posed a significant obstacle to gaining acceptance among developers and companies. In response to demand, Deno has since incorporated support for NPM, reconciling the need for package management with its original design principles. At the time of writing, Deno fully supports NPM. ### [](https://choubey.gitbook.io/internals-of-deno/introduction/about#top-level-await) Top-level await Deno's top-level await feature provides developers with a valuable tool for working with asynchronous code. This capability allows the use of the await keyword outside of traditional async functions, effectively creating a large async scope that influences the behavior of importing modules. These modules will pause execution before running their main code, awaiting the resolution of the Promise. While this feature may not seem revolutionary at first glance, its impact on development is significant. The await keyword can now be employed throughout the codebase, eliminating the need for wrapping code in async IIFEs (Immediately Invoked Function Expressions) solely for using await. It's important to note that this functionality is not unique to Deno, but rather is made possible by the underlying V8 engine that powers Deno. As a result, Deno can offer this feature, and its convenience has even extended beyond Deno to other runtime environments like Node.js, which has also incorporated support for top-level await in its ES modules. ### [](https://choubey.gitbook.io/internals-of-deno/introduction/about#standard-library) Standard library Deno distinguishes itself from Node.js by offering a comprehensive standard library, similar to Go's impressive standard library. This extensive library provides a range of peer-reviewed utilities for common tasks, including: * File handling * Hash generation * HTTP request management * Input/output management * MIME type handling * WebSocket connections * Logging * Date and time management * UUID generation * Cryptographic operations * Stream handling The Deno team actively maintains this library, ensuring its continued relevance and quality. Notably, Deno's standard library APIs exclusively use promises, eliminating the need for outdated callback mechanisms. This modern approach aligns Deno with contemporary asynchronous programming practices, simplifying the coding experience with consistent and predictable methods for managing asynchronous operations.This enhanced usability, combined with the robust and inclusive standard library, makes Deno an attractive choice for developers seeking to build efficient and reliable applications. [Previous1.2 History of Deno](https://choubey.gitbook.io/internals-of-deno/introduction/history-of-deno) [Next1.4 Releases](https://choubey.gitbook.io/internals-of-deno/introduction/releases) Last updated 1 year ago --- # 2.1 Architecture | The Internals of Deno This chapter marks the beginning of our in-depth exploration of Deno. Our approach will focus on providing a high-level understanding of the fundamental elements that comprise Deno's framework. **Chapter Objectives** In this chapter, we will explore Deno's architecture and its primary components. We will also conduct an in-depth examination of specific components that are integral to Deno's functionality. Please note that third-party components have extensive documentation available on their respective websites, and therefore, our discussion of these components will be limited to maintain our focus on Deno's core features and functionality. By adopting this approach, we will gain a comprehensive understanding of Deno's internal mechanics and design principles, without diverting our attention to external components that are already well-documented elsewhere. [](https://choubey.gitbook.io/internals-of-deno/architecture/architecture#chapter-contents) Chapter contents ----------------------------------------------------------------------------------------------------------------- [2.1 Architecture](https://choubey.gitbook.io/internals-of-deno/architecture/architecture) [2.2 Overall architecture](https://choubey.gitbook.io/internals-of-deno/architecture/overall-architecture) [2.4 Deno components](https://choubey.gitbook.io/internals-of-deno/architecture/core) [2.7 Rusty\_v8](https://choubey.gitbook.io/internals-of-deno/architecture/rusty_v8) [2.8 Tokio](https://choubey.gitbook.io/internals-of-deno/architecture/tokio) [2.9 V8](https://choubey.gitbook.io/internals-of-deno/architecture/v8) [Previous2.0 Cover page](https://choubey.gitbook.io/internals-of-deno/architecture/chapter-cover-page) [Next2.2 Overall architecture](https://choubey.gitbook.io/internals-of-deno/architecture/overall-architecture) Last updated 1 year ago --- # 1.7 What's next | The Internals of Deno This chapter has provided a comprehensive introduction to Deno, covering its origins, motivation, historical development, and release cycle. As outlined in the introduction to this book, our focus is on exploring Deno's internal mechanics and functionality, rather than providing sample programs. The next chapter will examine Deno's architecture and its key components in detail. Chapters 2 and 3 will provide a high-level overview, followed by a more in-depth analysis of Deno's inner workings in chapters 4, 5, and 6. This structured approach will enable a progressive understanding of Deno's functionality and design principles. By following this sequence, we will systematically uncover the intricacies of Deno's architecture and functionality, providing a thorough comprehension of this modern JavaScript runtime environment. [Previous1.6 Deno's source](https://choubey.gitbook.io/internals-of-deno/introduction/1.6-denos-source) [Next2.0 Cover page](https://choubey.gitbook.io/internals-of-deno/architecture/chapter-cover-page) Last updated 1 year ago --- # 2.6 TSC/SWC | The Internals of Deno Google's V8 engine is exclusively designed to run JavaScript code and does not support TypeScript, a superset of JavaScript that includes additional type information. To address this, TypeScript code must be translated into JavaScript through a transformation process, enabling the V8 engine to execute it. Deno, however, utilizes a combination of tools - the TypeScript Compiler (TSC) and the Super-fast Web Compiler (SWC) - to handle both TypeScript and JavaScript files. When type-checking is required, Deno employs Microsoft's TSC compiler, which converts TypeScript to JavaScript and performs type error checking. Otherwise, Deno leverages the high-performance SWC compiler for rapid transpilation. This preference is due to TSC's slower performance, being implemented in JavaScript, compared to SWC's speed and efficiency. SWC, built with Rust, is a fast TypeScript/JavaScript compiler that takes in modern files featuring async-await and generates browser-compatible JavaScript code. This transformation ensures code compatibility across various browsers with different levels of support for newer language features. Deno utilizes SWC to convert TypeScript files to JavaScript seamlessly. With its compatibility with the latest ECMAScript specifications, Deno eliminates the need to convert JavaScript files for compatibility reasons. By default, SWC selectively processes TypeScript files, leaving JavaScript files untouched unless explicitly requested to modify them. For further information on SWC, visit their official website at [https://swc.rs/](https://swc.rs/) , which provides detailed insights into the compiler's features, functionality, and optimal usage. [Previous2.5 OPs](https://choubey.gitbook.io/internals-of-deno/architecture/ops) [Next2.7 Rusty\_v8](https://choubey.gitbook.io/internals-of-deno/architecture/rusty_v8) Last updated 1 year ago --- # 2.3 Programming Languages | The Internals of Deno Before we discuss the components in detail, let's talk about the programming languages used by Deno. We'll give a brief overview of the languages used in the Deno ecosystem. This will help us understand the components better when we discuss them later. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252FLV8vujCKAF72nGUy4wHm%252FScreenshot%25202024-07-12%2520at%25202.25.53%25E2%2580%25AFPM.png%3Falt%3Dmedia%26token%3Dd5dc7a21-683e-49fe-bf72-d8495b63df9b&width=768&dpr=4&quality=100&sign=ece3d73c&sv=2) Src: GitHub [](https://choubey.gitbook.io/internals-of-deno/architecture/2.3-programming-languages#rust) Rust ------------------------------------------------------------------------------------------------------ A significant portion of Deno's codebase is written in Rust, a highly popular and secure programming language. Rust's robustness adds an extra layer of reliability to Deno's architecture. Various essential elements of Deno, such as the CLI, module graph management, runtime execution, operational functionalities, and core mechanisms, heavily utilize Rust. Rust is a versatile programming language with multiple styles and applications. It is designed to excel in overall performance, safety in handling data types, and managing tasks running concurrently. Rust ensures memory safety without relying on a garbage collector or reference counting, methods often used in other languages prioritizing memory safety. This is achieved through its "borrow checker," which closely observes the lifespan of object references throughout the program's compilation process. Rust's design incorporates functional programming concepts, including static typing, immutability, higher-order functions, and algebraic data types. These elements contribute to Rust's stronghold in the realm of systems programming, where control over hardware and memory is paramount. The history of Rust began with Graydon Hoare, who created it as a personal project while working at Mozilla Research in 2006. Mozilla officially backed the project in 2009, recognizing its potential. Since its inaugural stable release in May 2015, Rust has garnered significant adoption from major players like Amazon, Discord, Dropbox, Meta, Google, and Microsoft. In 2022, Rust became the first language, apart from C and assembly, to receive support for Linux kernel development. Rust's rapid growth has caught the attention of academia, leading to its exploration and examination in the realm of programming language research. The multifaceted nature of Rust has attracted the curious minds of researchers, setting the stage for further advancements and discoveries in the world of programming languages. [](https://choubey.gitbook.io/internals-of-deno/architecture/2.3-programming-languages#c) C++ -------------------------------------------------------------------------------------------------- While Deno's core codebase is written in Rust, the V8 engine, which Deno uses for JavaScript and WebAssembly execution, is built using C++. V8 is Google's open-source, high-performance engine for JavaScript and WebAssembly, also built with C++. This powerful engine is used not only in Chrome but also in Node.js and other software. Its main function is to interpret and execute ECMAScript and WebAssembly, essential for modern web development. V8 is compatible with multiple operating systems, including Windows 7 and later, macOS 10.12 and later, and various Linux systems using x64, IA-32, ARM, or MIPS processors. This broad compatibility allows developers to utilize V8 across different computing environments. V8's versatility lies in its ability to function both as a standalone entity and as an embedded component within any C++ application. This flexibility enables developers to customize their software solutions while leveraging V8's high-performance capabilities. Whether powering web browsers or enhancing applications, V8's impact is significant in modern software development. [](https://choubey.gitbook.io/internals-of-deno/architecture/2.3-programming-languages#javascript) JavaScript ------------------------------------------------------------------------------------------------------------------ Deno's main runtime code, which users interact with, is written in JavaScript. This code provides Deno's JavaScript APIs, which you can use in your own code. Examples of these APIs include functions like getEnv, setEnv, alert, confirm, and console logs. Additionally, this code includes APIs from the ext component, which adds more utility. This component provides a range of essential user-level functions. These include: * Buffers * File System interactions * Process management, * Signal handling * HTTP communication * User prompts * etc. This wide range of functions makes Deno versatile and capable of supporting diverse applications. Initially, Deno's runtime code was written in TypeScript. However, due to certain challenges, the decision was made to switch to pure JavaScript. This change was made to address specific issues encountered during development. [Previous2.2 Overall architecture](https://choubey.gitbook.io/internals-of-deno/architecture/overall-architecture) [Next2.4 Deno components](https://choubey.gitbook.io/internals-of-deno/architecture/core) Last updated 1 year ago --- # 2.0 Cover page | The Internals of Deno ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252FCOrldcSKPsw9HQA9tU9Y%252F3.png%3Falt%3Dmedia%26token%3D007a2957-9663-4e2f-8fd7-d883ba84be5b&width=768&dpr=4&quality=100&sign=6e1dbb3&sv=2) [Previous1.7 What's next](https://choubey.gitbook.io/internals-of-deno/introduction/whats-next) [Next2.1 Architecture](https://choubey.gitbook.io/internals-of-deno/architecture/architecture) Last updated 1 year ago --- # 2.7 Rusty_v8 | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/architecture/rusty_v8#overview) Overview --------------------------------------------------------------------------------------------- V8 and Deno have different core programming languages: V8 uses C++, while Deno uses Rust. To bridge this gap, the innovative "rusty\_v8" library was developed. This library creates efficient Rust bindings to V8's complex C++ APIs, achieving seamless integration without additional call overhead. By doing so, rusty\_v8 enables the harmonious interaction between V8's C++ foundation and Deno's Rust infrastructure. When working with APIs, rusty\_v8 harmonizes with V8's APIs through a straightforward translation of functionalities, maintaining efficiency without extra burdens. This synchronization is a testament to the library's ability to foster compatibility and efficiency. The creation of rusty\_v8 demonstrates the successful integration of languages and technologies, showcasing the potential for dynamic interaction between different programming languages. This connection highlights the compatibility of C++ and Rust, emphasizing the commitment to efficiency and seamless integration within the Deno ecosystem. [](https://choubey.gitbook.io/internals-of-deno/architecture/rusty_v8#example) Example ------------------------------------------------------------------------------------------- Let's examine a few code examples to better understand how rusty\_v8 works. Analyzing its practical uses will help us comprehend rusty\_v8 more effectively. A specific aspect to investigate is the way rusty\_v8 implements Strings. Copy pub fn length(&self) -> usize { unsafe { v8__String__Length(self) as usize } } The String class in Rust includes a function that calculates the length of a string. When this "length" function is called, it activates the "v8\_\_String\_\_Length" code, which is written in C++. This code determines the length of the v8 string data type. There is a direct and simple relationship between the function and the code, with a one-to-one correspondence. This means that the function and code are closely linked, providing a clear connection between the Rust String class and v8's string data type. Copy int v8__String__Length(const v8::String& self) { return self.Length(); } } Let's explore another example within the rusty\_v8 module, specifically the 'parse' function: Copy pub fn parse<'s>( scope: &mut HandleScope<'s>, json_string: Local<'_, String>, ) -> Option> { unsafe { scope .cast_local(|sd| v8__JSON__Parse(sd.get_current_context(), &*json_string)) } } Rusty\_v8 has a parse function that matches the v8\_\_JSON\_\_Parse function. The parse function, written in C++, acts as a connector to access the corresponding function in the v8 library. This connection enables Rusty\_v8 to utilize v8's capabilities for parsing JSON data. Copy const v8::Value* v8__JSON__Parse(const v8::Context& context, const v8::String& json_string) { return maybe_local_to_ptr( v8::JSON::Parse(ptr_to_local(&context), ptr_to_local(&json_string))); } Rusty\_v8 generally matches the v8 API in most cases. For more information on rusty\_v8, visit their GitHub page at [https://github.com/denoland/rusty\_v8](https://github.com/denoland/rusty_v8) . Now, let's discuss Tokio, a tool that enables Deno to perform tasks asynchronously. This means Deno can handle tasks efficiently without getting stuck on one task, improving its overall performance and responsiveness. [Previous2.6 TSC/SWC](https://choubey.gitbook.io/internals-of-deno/architecture/tsc) [Next2.8 Tokio](https://choubey.gitbook.io/internals-of-deno/architecture/tokio) Last updated 1 year ago --- # 2.2 Overall architecture | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/architecture/overall-architecture#the-architecture-of-deno) The architecture of Deno ----------------------------------------------------------------------------------------------------------------------------------------- Similar to other runtime environments like Node.js, Bun, Go, and Python, Deno operates as a compact monolith within a single process. This design approach is common among major runtime environments, leveraging multi-threading capabilities to efficiently handle computationally intensive tasks. Notably, Go distinguishes itself by providing robust support for green threads, enhancing performance in specific scenarios. Deno's foundation uniquely spans multiple programming languages, with core components built using Rust, TypeScript, and JavaScript. Interestingly, Deno's primary runtime, initially written in TypeScript, has been rewritten in pure JavaScript, without ES modules, showcasing Deno's adaptability and dynamic development process. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252FLV8vujCKAF72nGUy4wHm%252FScreenshot%25202024-07-12%2520at%25202.25.53%25E2%2580%25AFPM.png%3Falt%3Dmedia%26token%3Dd5dc7a21-683e-49fe-bf72-d8495b63df9b&width=768&dpr=4&quality=100&sign=ece3d73c&sv=2) Deno's intricate operations rely on seven pivotal components, which synergize to execute diverse tasks. Beyond these core components, three essential third-party libraries enrich the ecosystem, enabling JavaScript program execution while maintaining asynchronous internal mechanisms. This balance of native and third-party components enables Deno to seamlessly run JavaScript programs while upholding its asynchronous nature. In total, Deno's architecture comprises approximately 10 components, categorized into seven core and three third-party blocks. These blocks form the essential building blocks of Deno's functionality and features, underscoring its robust and adaptable design. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/overall-architecture#architecture-diagram) Architecture Diagram The following block diagram illustrates the key building blocks that constitute Deno's native architecture. This diagram provides a comprehensive overview of the main components developed and maintained by the Deno team. Please note that this diagram exclusively focuses on Deno's native components and does not include any elements contributed by third-party sources. This distinction is important, as it highlights the core components that form the foundation of Deno's runtime environment, separate from external libraries and dependencies. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252FAlY5b1Pc5xmuD3BGl1fx%252Fdeno%2520overall%2520arch.png%3Falt%3Dmedia%26token%3Dc1ddf31b-be47-4b0d-95e7-0fbef9ac2ed0&width=768&dpr=4&quality=100&sign=39df2e48&sv=2) Before we proceed to a detailed examination of Deno's architecture, let us first take a brief look at the principal components that comprise this runtime environment. While we will explore each of these components in greater depth in subsequent sections, this initial overview will provide a foundational understanding of Deno's functionality and the role each component plays in its overall framework. This introduction will set the stage for a more comprehensive analysis of Deno's inner workings. [](https://choubey.gitbook.io/internals-of-deno/architecture/overall-architecture#major-components) Major components ------------------------------------------------------------------------------------------------------------------------- ### [](https://choubey.gitbook.io/internals-of-deno/architecture/overall-architecture#cli) CLI In addition to relying on essential third-party components like Tokio and V8, Deno's user-facing functionality is largely integrated into the Command-Line Interface (CLI). The CLI serves as a central hub, fulfilling two critical roles: orchestration and execution. By connecting various components, the CLI plays a vital role in facilitating user interaction with Deno's core features and functionality. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/overall-architecture#runtime) Runtime The runtime component is a crucial aspect of Deno's architecture, encompassing a range of essential elements. These include: * Deno's runtime code, written in JavaScript * Fundamental operations at a lower level * An inspector for debugging purposes * Performance metrics * Key functionalities, such as: * Main worker * Web worker * Permissions control This component plays a central role in executing Deno's core functionality, managing worker processes, and enforcing security permissions, while also providing critical tools for debugging and performance measurement. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/overall-architecture#ext) Ext The ext component is a collection of compact, modular blocks that provide a range of web APIs. These modules are designed to be self-contained and flexible, offering a variety of web-related functionalities that can be easily integrated into Deno's runtime environment. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/overall-architecture#core) Core The Core component plays a vital role in Deno's architecture, serving as a bridge between the Rust and JavaScript layers. On one hand, it collaborates with the Rust components, including the CLI and the runtime, to facilitate seamless interaction. On the other hand, it provides a range of services that can be easily accessed from the Deno runtime using JavaScript. While the CLI orchestrates the execution, the Core component provides critical functionality for running TypeScript/JavaScript code and enabling interoperability between JavaScript and Rust. Specifically, the Core implements essential features such as: * V8 bindings * Flags * Modules * JsRuntime * Zero overhead bindings * Shared queue The primary purpose of the Core component is to provide low-level functionality that enables efficient communication between the Rust and JavaScript layers. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/overall-architecture#graph) **Graph** The Graph component provides module graph services to the CLI component. The module graph is a visual representation of the relationships between various modules in a Deno project, illustrating their dependencies and interconnections. By offering a comprehensive view of module relationships, the Graph component enables the CLI to efficiently manage and resolve dependencies. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/overall-architecture#npm) NPM The NPM component serves as an interface between Deno and the Node Package Manager (NPM) ecosystem, providing a set of APIs specifically designed for interacting with NPM modules. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/overall-architecture#cache) Cache The Cache component is a vital part of Deno's architecture, responsible for managing and maintaining the module cache within the runtime environment. This cache contains modules sourced from diverse locations, including the NPM repository, as well as other external sources. By efficiently managing the module cache, the Cache component ensures rapid access to frequently used modules, reduces loading times, and enhances overall performance. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/overall-architecture#rusty_v8) **Rusty\_v8** Although not depicted in the diagram, the Rusty\_v8 component is a crucial element in the Deno project. It provides high-quality Rust bindings to V8's C++ API, effectively serving as a bridge between the Rust programming language and V8's core functionalities. By offering a reliable and efficient interface, Rusty\_v8 enables seamless communication between Rust and V8, facilitating the integration of V8's powerful engine with Rust's memory safety guarantees. [](https://choubey.gitbook.io/internals-of-deno/architecture/overall-architecture#third-party-components) Third party components ------------------------------------------------------------------------------------------------------------------------------------- Although Deno's core components are essential for its operation, they are insufficient on their own to execute tasks. To attain its full functionality, Deno relies on critical external components that act as the backbone of the Deno runtime. These external components, including Tokio and V8, among others, play a vital role in enabling Deno's to run JS code. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252FDmrMX5Kk4D9zOMLhV4eH%252Fdeno%2520arch%2520third%2520party.png%3Falt%3Dmedia%26token%3D94faacbd-67ac-4ed1-a520-4ccd99557636&width=768&dpr=4&quality=100&sign=4fdb5067&sv=2) ### [](https://choubey.gitbook.io/internals-of-deno/architecture/overall-architecture#tokio) Tokio Tokio, a crucial third-party component, plays a central role in Deno's architecture, serving as the foundation upon which Deno's asynchronous capabilities are built. The harmonious integration of Tokio empowers Deno to unlock its full asynchronous potential, enabling the efficient execution of concurrent tasks. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/overall-architecture#v8) V8 JavaScript programs rely on the presence of a JavaScript engine to execute, and V8 is a prominent example of such an engine. Developed by Google, V8 is an open-source, high-performance engine specifically designed for executing JavaScript and WebAssembly code. Built using the C++ programming language, V8 plays a vital role in powering JavaScript execution, enabling the runtime environment to interpret and execute JavaScript code efficiently. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/overall-architecture#typescript-compiler) Typescript compiler The TypeScript compiler is a crucial third-party component in Deno, though not depicted in the diagram. As previously mentioned, V8 solely processes JavaScript code. However, Deno supports both TypeScript (TS) and JavaScript (JS). To allow TypeScript, Deno uses the TypeScript compiler to transform TypeScript code into JavaScript before presenting it to V8 for execution. This process enables Deno to have built-in support for TypeScript applications. \-- This section has provided a comprehensive overview of Deno's architecture, highlighting the key components that comprise its structure. We have briefly examined the roles and relationships between these components, gaining a foundational understanding of Deno's inner workings. In the next section, we will examine each component in more detail, analyzing their functions and characteristics. [Previous2.1 Architecture](https://choubey.gitbook.io/internals-of-deno/architecture/architecture) [Next2.3 Programming Languages](https://choubey.gitbook.io/internals-of-deno/architecture/2.3-programming-languages) Last updated 1 year ago --- # 2.5 OPs | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/architecture/ops#introduction) Introduction ------------------------------------------------------------------------------------------------ Operations (OPs) extend Deno's capabilities beyond the ECMAScript specification. Unlike OPs, the V8 engine operates within a limited environment that strictly follows ECMAScript guidelines, unable to perform tasks like reading files, managing sockets, and handling timers on its own. These tasks require external APIs to function. OPs bridge this gap, expanding Deno's functionality beyond V8's limitations. Let's consider a simple example: extracting an environment variable in Deno. To achieve this, an OP interacts with the underlying operating system, retrieves the variable's value (or perform the desired operation), and makes it accessible to your JavaScript or TypeScript code. By providing access to system resources, OPs enable Deno to interact with its environment beyond ECMAScript's scope. OPs play a crucial role in enhancing Deno's capabilities, allowing it to perform tasks that would otherwise be impossible within the V8 engine's constrained environment. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/ops#types-of-ops) Types of OPs Operations (OPs) can be categorized into two main types: synchronous ops and asynchronous ops. _Synchronous OPs_: * Run continuously from start to finish without interruption. * When executed, the script's execution is paused until the operation is complete and a result is obtained. * This means that synchronous ops block the main thread until they finish processing. _Asynchronous OPs_: * Scheduled to produce a result at a later time. * Results become available later and are processed without blocking the main thread. * Unlike synchronous OPs, asynchronous OPs do not halt program execution, allowing for non-blocking behavior. Synchronous operations (ops) block the current thread's progress until the desired result is achieved. In contrast, asynchronous ops allow the program to continue running without waiting for the result. To implement asynchronous ops, Deno uses the Tokio runtime, which employs asynchronous green threads. These threads enable the execution of asynchronous ops without impeding the overall program flow. We will explore Tokio's workings in more detail later. Note that some operations can be both synchronous and asynchronous, depending on their nature. Others are exclusively synchronous ops. The classification of an operation depends on its specific characteristics. For example, consider the "getRandomValues()" function in the crypto module. This function is only available as a synchronous op through "op\_get\_random\_values()." Since this function involves intensive CPU computations, there is no need to provide an asynchronous version. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/ops#list-of-ops) List of OPs Let's review the list of services provided by Ops. Note that this list is not exhaustive, but rather a selection of some of the services offered:: * Crypto * Fetch * HTTP client * FS ops * open * seek * mkdir * chmod * stat * read dir * etc. (the list if quite long) * Net * accept * connect * shutdown * listen * etc. * OS * exit * env * set/get env * hostname * load average * memory info * CPU info * Permissions * query * revoke * request * Open plugin * Process * run * kill * Runtime * compile * transpile * Signal * Bind * Unbind * Poll * Timer * Start * Stop * Now * Sleep sync * TLS * start * connect * listen * accept * Worker * Create a worker * Terminate worker * Get message * Post message * close * WebSocket * Create * Send * Close * Next event The list of operations is quite long, and a notable pattern emerges upon closer inspection. Most operations have a direct connection to the functions provided by the JavaScript component of the runtime, with many forming a one-to-one correspondence. This relationship can be seen in examples such as the "kill()" function in the Process module, which directly aligns with the "op\_kill()" operation. Similarly, the "openSync()" function used for file operations has a corresponding "op\_open\_sync()" operation. This parallel between JavaScript functions and their associated operations highlights the strong link between Deno's runtime services and its JavaScript foundation. [](https://choubey.gitbook.io/internals-of-deno/architecture/ops#example) Example -------------------------------------------------------------------------------------- Let's consider the Deno.env.get() function in Deno as a representative example. This function is a simple and uncomplicated API within the Deno programming environment. When we refer to it as a "sync" function, we mean that the JavaScript thread - the sequence of instructions being executed - will temporarily pause its execution until the desired outcome is obtained from a lower-level operation. In other words, the JavaScript thread will wait until the operation is complete before continuing to execute. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/ops#user-program) User program Retrieving the value of an environment variable in Deno is a simple process. Let's demonstrate how to retrieve the value of the "HOME" variable with a concise code example: Copy Deno.env.get('HOME'); ### [](https://choubey.gitbook.io/internals-of-deno/architecture/ops#deno-code) Deno code The "env" object provides several useful APIs, including "get", "set", "delete", and "toObject". Upon closer inspection, it becomes clear that the Deno.env.get method is equivalent to a function called "getEnv". Copy Deno.env { get: [Function: getEnv], toObject: [Function: toObject], set: [Function: setEnv], has: [Function: has], delete: [Function: deleteEnv] } The getEnv functions is part of the OS service: Copy function getEnv(key) { return ops.op_get_env(key) ?? undefined; } The getEnv() function is relatively simple, serving as a wrapper for the lower-level op\_get\_env() operation. This operation, implemented in Rust, performs the underlying work. While exploring how Deno's JavaScript environment triggers this low-level Rust operation (referred to as "OP") can be fascinating, we'll save that discussion for later. For now, let's proceed to examine the actual code behind op\_get\_env, which is written in Rust. By doing so, we'll gain insight into its internal mechanics and functionality, helping us better understand the underlying mechanics that drive this aspect of Deno's functionality. Copy fn op_get_env( state: &mut OpState, key: String, ) -> Result, AnyError> { let skip_permission_check = NODE_ENV_VAR_ALLOWLIST.contains(&key); if !skip_permission_check { state.borrow_mut::().check_env(&key)?; } if key.is_empty() { return Err(type_error("Key is an empty string.")); } if key.contains(&['=', '\0'] as &[char]) { return Err(type_error(format!( "Key contains invalid characters: {key:?}" ))); } let r = match env::var(key) { Err(env::VarError::NotPresent) => None, v => Some(v?), }; Ok(r) } The Rust code is relatively simple, performing a series of validations in a single step. Then, it uses the convenient std::env crate to access the environment variable. The result is then returned as a value in the response. Interestingly, this response travels from the Rust code to the JavaScript code, a connection that will be thoroughly explored in the upcoming chapters of this book. This transition from Rust to JavaScript is a crucial aspect of Deno's functionality, and understanding its intricacies is essential for appreciating the overall architecture of Deno. \-- We have now covered the fundamentals of ops. In Chapter 6, we will further explore the intricacies of ops, gaining a thorough understanding of the bidirectional transition between JavaScript and Rust. This language switching mechanism is essential to the Deno environment, and in the next chapter, we will dissect its step-by-step mechanics, revealing the underlying dynamics that facilitate seamless communication between the two programming languages. [Previous2.4 Deno components](https://choubey.gitbook.io/internals-of-deno/architecture/core) [Next2.6 TSC/SWC](https://choubey.gitbook.io/internals-of-deno/architecture/tsc) Last updated 1 year ago --- # 2.10 What's next | The Internals of Deno We have completed our detailed analysis of Deno's architecture and its components. Our examination began with a broad overview and progressed to a detailed analysis of each element, providing us with a thorough understanding of Deno's fundamental structure. In the next chapter, we will thoroughly examine Deno's threading model. This examination is essential, as omitting the threading model's details would leave our discussion of Deno's architecture incomplete. By examining the threading model, we will gain a deeper understanding of Deno's internal mechanics, further enhancing our understanding of its architectural framework. [Previous2.9 V8](https://choubey.gitbook.io/internals-of-deno/architecture/v8) [Next3.0 Cover page](https://choubey.gitbook.io/internals-of-deno/threading-model/threading-model) Last updated 1 year ago --- # 3.1 Threading model | The Internals of Deno When a runtime like Node.js, Deno, or Bun operates, it uses a specific threading model. This model defines the threads within the process and explains their roles. Understanding the threading model is essential when combined with the system's overall structure. Deno's threading model is unique, using two types of threads: OS-aware and OS-unaware. This approach sets Deno apart and affects its behavior. In this chapter, we will explore the default threads created by Deno. We will also examine web workers and asynchronous operations, which impact the threading model. By the end of the chapter, you will have a thorough understanding of these threads, their interactions, and their significance in Deno's threading framework. [](https://choubey.gitbook.io/internals-of-deno/threading-model/3.1-threading-model#chapter-contents) Chapter contents --------------------------------------------------------------------------------------------------------------------------- [3.2 Default threading model](https://choubey.gitbook.io/internals-of-deno/threading-model/default-threads) [3.3 Asynchronous green threads](https://choubey.gitbook.io/internals-of-deno/threading-model/tokio-threads) [Previous3.0 Cover page](https://choubey.gitbook.io/internals-of-deno/threading-model/threading-model) [Next3.2 Default threading model](https://choubey.gitbook.io/internals-of-deno/threading-model/default-threads) Last updated 1 year ago --- # 3.4 What's next | The Internals of Deno n this chapter, we explored Deno's threading model. We discussed the default threading approach and how Deno provides ways to perform tasks asynchronously through its APIs or tasks. This concludes the theoretical part of the book. Next, we will move on to practical examples, showing how Deno works in practice. The next chapter will discuss the relationship between V8 and Deno. While V8 handles JavaScript code execution, it may not cover all the functions Deno needs. Therefore, we need to understand how to interact between these two components. Then, we will explore how Deno loads and executes TypeScript/JavaScript code in detail. This will be an in-depth journey across chapters 4, 5, and 6. Get ready to learn about Deno's inner workings. [Previous3.3 Asynchronous green threads](https://choubey.gitbook.io/internals-of-deno/threading-model/tokio-threads) [Next4.0 Cover page](https://choubey.gitbook.io/internals-of-deno/bridge/chapter-cover-page) Last updated 1 year ago --- # 4.4 What's next | The Internals of Deno We have briefly discussed the bridge, but our discussion is not over. We will revisit this concept again in chapter 6 when we explore ops. In the next chapter, we will focus on a simple "hello world" program. We will guide you through each essential step, starting from the beginning and going through to the final execution. This will give you a solid understanding of the process. [Previous4.3 Encode and decode](https://choubey.gitbook.io/internals-of-deno/bridge/4.4-encode-and-decode) [Next5.0 Cover page](https://choubey.gitbook.io/internals-of-deno/foundations/chapter-cover-page) Last updated 1 year ago --- # 2.4 Deno components | The Internals of Deno In the previous section, we discussed Deno's overall structure at a high level, covering both core and third-party components. Now, let's focus on Deno's core components, excluding third-party elements for the time being. When we refer to Deno components in this section, we mean the essential parts of Deno itself, excluding third-party libraries. Deno's primary function is straightforward: it executes TypeScript and JavaScript programs. Deno handles various tasks, including: * Providing the main program structure * Managing sub-commands * Orchestrating the event loop * Offering a debugger inspector * Acting as a linter * Fetching files * Resolving modules * Providing a standard library * Presenting core APIs Deno takes care of everything outside Tokio and v8's responsibilities, particularly v8. While v8 runs JavaScript efficiently within a sandboxed environment, it relies on Deno for tasks like file management, networking, input-output operations, timers, and more. This collaboration between Deno and v8 creates a comprehensive runtime environment for executing programs. We previously overviewed Deno's components at a high level. Now, we'll explore them in detail, referencing the same diagram: ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252FAlY5b1Pc5xmuD3BGl1fx%252Fdeno%2520overall%2520arch.png%3Falt%3Dmedia%26token%3Dc1ddf31b-be47-4b0d-95e7-0fbef9ac2ed0&width=768&dpr=4&quality=100&sign=39df2e48&sv=2) [](https://choubey.gitbook.io/internals-of-deno/architecture/core#cli) CLI ------------------------------------------------------------------------------- As discussed earlier, the CLI component plays a vital role in Deno. However, it's important to note that V8, with its extensive codebase of around a million lines, is the largest component, surpassing CLI in size. The CLI serves two key purposes: orchestration and service provision, making it central to the Deno ecosystem. As an orchestrator, the CLI coordinates the activities of essential services like ext, runtime, graph, cache, file fetching, core functionalities, ops, tokio, and the V8 engine. By harmonizing their interactions, the CLI ensures smooth operation and efficient collaboration among Deno's components. Additionally, the CLI provides essential services to other Deno components. Its dual role as orchestrator and service provider underscores its importance in the Deno framework, enabling right communication, smooth execution, and robust performance among Deno's components. The CLI also provides the Deno executable, containing the primary (or main) program that implements various subcommands and user-oriented APIs. While the CLI doesn't cover all APIs, it encompasses a significant portion. Essentially, the CLI serves as the gateway to Deno's functionality, providing tools for executing Deno-powered tasks and interacting with its capabilities. The following list outlines the functionalities provided by the CLI. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/core#main-program) Main program The main program in Deno is the entry point for all operations. When the "deno" command is invoked, it triggers the execution process. Note that this marks the beginning of Deno's operations, but not the immediate start of the program's functionality. Before the program can run, several essential tasks need to be completed. We will explore these tasks in detail in chapters 4, 5, and 6. By understanding these chapters, we will gain a comprehensive understanding of the processes that occur before the program's actual runtime. This understanding will provide valuable insights into the fundamental aspects of Deno's functioning. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/core#sub-commands) Sub-commands Deno offers a range of development utilities through its sub-commands, despite appearing as a single executable. The 'run' and 'test' sub-commands are the most prominent and frequently used by developers. 'Run' executes JavaScript or TypeScript programs, while 'test' initiates unit tests to ensure code integrity. Deno provides various sub-commands for different development aspects, including: * run: Executes JavaScript or TypeScript programs * bench: Conducts benchmarking * bundle: Merges modules and dependencies into a single file * cache: Caches dependencies for performance enhancement * check: Performs type-checking on dependencies * compile: Compiles scripts into self-contained executables (UNSTABLE) * completions: Generates shell completion scripts * coverage: Produces coverage reports for code analysis * doc: Displays module documentation * eval: Executes scripts for on-the-fly evaluation * fmt: Automatically formats source files for consistent coding styles * init: Initializes new projects * info: Provides cache or source file-related information * install: Installs scripts as executables * uninstall: Removes installed scripts * lsp: Starts the language server for enhanced code editing * lint: Performs source code linting for quality enhancement * repl: Sets up a Read-Eval-Print Loop environment for interactive scripting * task: Executes tasks defined in configuration files * test: Launches tests for code validation * types: Outputs runtime TypeScript declarations * upgrade: Upgrades the Deno executable to specified versions * vendor: Imports remote modules into local directories * help: Provides guidance and information on subcommands ### [](https://choubey.gitbook.io/internals-of-deno/architecture/core#file-fetcher) File fetcher The file fetcher plays a crucial role in the Deno ecosystem. Unlike platforms with dedicated package managers, Deno takes a flexible approach. It enables modules to be accessed through HTTP, local sources, or NPM repositories. The file fetcher's main responsibility is to retrieve module files, regardless of their location. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/core#flags) Flags Deno relies heavily on flags to enable its sandboxing feature. These flags are essential for enforcing permissions that maintain Deno's robust sandboxing system. The flags are created using command-line arguments. Sandboxing is a critical component of Deno's security framework, designed to limit and control the actions a Deno script can perform. The flags act as gatekeepers, allowing or blocking specific operations and interactions within the script's runtime environment. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/core#module-loader) Module loader The module loader in Deno plays a vital role in handling ES modules. It provides various functionalities to resolve, load, and prepare modules for use. This includes: * Fetching modules from their sources * Compiling them as needed * Caching them for optimized performance The module loader manages the entire process, ensuring efficient and effective handling of ES modules. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/core#file-watcher) File watcher The file watcher monitors the main source code and its dependencies, triggering a Deno process restart when it detects changes. This ensures the Deno runtime stays current with the latest modifications, facilitating a smoother & quicker development experience. When working on a Deno project, the file watcher observes your code changes and alerts the system if it detects modifications in the main code or dependent files. This alert triggers an automatic restart of the Deno process, ensuring it runs with the most up-to-date code. [](https://choubey.gitbook.io/internals-of-deno/architecture/core#runtime) Runtime --------------------------------------------------------------------------------------- The runtime is the core of Deno, comprising its essential operations and features. It combines Deno's core capabilities written in JavaScript with foundational operations coded in Rust, enabling efficient and reliable performance. The runtime framework utilizes "workers" - specialized components that handle specific tasks in parallel, enhancing Deno's multitasking capabilities. Workers play a vital role in optimizing Deno's functionality, whether it's executing scripts, managing I/O operations, or orchestrating tasks. Given its importance, the runtime is housed within the main Deno repository. The runtime consists of multiple interconnected services, categorized as: * **Workers**: Separate execution contexts for concurrent code execution, enhancing performance and efficiency. * **Permissions**: Manages code execution access, ensuring security and control by only granting explicit permissions. * **Metrics**: Collects and provides performance and behavior insights, aiding developers in optimization. * **Ops**: Facilitates interaction with the underlying operating system, enabling system-level operations. * **JavaScript Runtime**: The core where JavaScript code is executed, enabling program execution and functionality realization. Deno's runtime combines its services to provide a secure, efficient, and developer-friendly environment for executing JavaScript and TypeScript code. This combination ensures Deno works smoothly, supporting modern development practices and various applications. The runtime serves as the interface for accessing the Deno API in both TypeScript and JavaScript environments. It includes all core features necessary for executing programs within Deno. While Deno's CLI and core services are primarily implemented in Rust, the runtime implements these services in both Rust and JavaScript. Deno's runtime code is written entirely in JavaScript, without using ECMAScript modules. The runtime provides a comprehensive Deno API for user programs and implements low-level Rust operations, playing a crucial role in the Deno ecosystem. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/core#workers) Workers Workers play a vital role in Deno's runtime environment, similar to Node.js. Deno executes the main program in the primary thread, but can also create additional web worker threads if needed. Each worker, including the main worker, has its own runtime and event loop. There are two distinct types of workers: * **Main Worker**: As implied by its name, the main worker is responsible for running the primary program. It operates within the primary thread itself, without requiring a separate thread for execution. This main worker is automatically established when the program runs. * **Web Worker**: The web worker is an additional type of worker that can be initiated by an application. Usually, these additional workers are employed to handle tasks that demand substantial CPU resources, potentially causing the main event loop to be blocked. For tasks that revolve around intensive input/output operations, a separate worker isn't necessary since Deno effectively manages asynchronous I/O tasks within the primary thread. These web workers follow the conventional standard of web workers found in web browsers. In summary, workers are pivotal in the functioning of Deno's runtime environment. They provide a means to offload resource-intensive operations from the main thread, allowing the program to continue running smoothly. While the main worker handles the main program execution, web workers are an advantageous tool to tackle heavy computational tasks without hampering the overall responsiveness of the application. This enables developers to build robust and scalable applications without compromising performance. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/core#permissions) Permissions Permissions play a vital role in creating a secure sandbox environment in Deno. These permissions cover a wide range of actions, including reading and writing files, accessing the network, managing the environment, spawning processes, utilizing plugins, and more. The importance of permissions is particularly significant in Deno and distinguishes it from other platforms like Node.js. Deno's approach to permissions is a notable differentiator. It allows users to grant or deny specific abilities to their scripts, enhancing security. This approach contrasts with traditional systems, where scripts often have unrestricted access to resources, potentially leading to vulnerabilities. Off lately, Node.js has also recognized the importance of permissions. Over time, it has been incorporating a permission-like system in its releases, acknowledging the security benefits it brings. This shift demonstrates the growing recognition of the importance of permissions in ensuring secure sandbox environments. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/core#js-part-of-deno-runtime) JS part of Deno runtime The JS Runtime in Deno is a collection of code that operates within the user's JavaScript environment. This crucial component forms the foundation of Deno's capabilities, all written in pure JavaScript. The following functionalities are provided by the JS runtime: 1. **Build Information:** This encompasses details like the target platform, architecture, operating system, vendor, and environment. Such information assists in optimizing Deno's behavior for different setups. 2. **Colors:** Deno's runtime allows you to work with colorization, enhancing the visual experience of output in the terminal. 3. **Errors (non-ECMAScript):** While not a part of the official ECMAScript specification, Deno's runtime introduces its own error handling mechanisms to improve code robustness. 4. **Version Information:** Accessing information about the current Deno version helps users stay updated and aware of the features available. 5. **Console Utilities:** The runtime provides tools for efficient interaction with the console, aiding in debugging and logging. 6. **Dispatch:** This involves sending operations to Rust, the underlying programming language Deno is built with, and processing the responses efficiently. 7. **Timers:** The runtime supports various timer functionalities such as timeout and interval, enabling scheduling and coordination in applications. 8. **Workers:** Deno's runtime comprehends the intricacies of the web worker lifecycle and facilitates communication between parent and child workers. 9. **I/O Operations:** Input and output operations like copying, reading, and writing files are handled smoothly. 10. **Buffer Management:** Efficient handling and manipulation of buffers, used for various data operations. 11. **Websockets:** Support for websockets enables real-time, bidirectional communication between clients and servers. 12. **File Operations:** Deno's runtime manages standard I/O streams like stdout and stdin, as well as operations like creating, opening, seeking, and manipulating files. 13. **File System Operations:** The runtime encompasses a plethora of file system tasks like changing directories, modifying permissions and ownership, copying files, reading directories, and more. 14. **Metrics:** Facilities for gathering various metrics to monitor and optimize Deno's performance. 15. **Networking (Net):** Deno's runtime facilitates network-related tasks like connecting, listening, and working with datagrams. 16. **Operating System Information:** Accessing essential information about the operating system, including release, memory, CPU, and load averages. 17. **TypeScript Compiler:** A TypeScript compiler is available within the user space, enabling seamless TypeScript development. 18. **File Watcher:** Deno's runtime includes a file watcher, which can monitor changes in files and trigger appropriate actions. 19. **Permissions Management:** This involves querying, revoking, and requesting permissions required for specific actions. 20. **Process Control:** The runtime handles processes, including running and killing them. 21. **TLS (Transport Layer Security):** Facilities for starting, listening, and connecting with secure TLS connections. 22. **User Prompts:** Interaction with users is facilitated through prompts like alerts, confirmations, and input queries. The JS Runtime API offers a wide range of functionalities, enabling developers to build robust and feature-rich applications. Notably, the JS runtime automates various complex operations, such as managing interactions with Deno, thereby simplifying the development process and providing a seamless user experience. This automation eliminates the need for developers to have in-depth knowledge of low-level operations, making it easier to build applications. The JS Runtime API plays a crucial role in abstracting away underlying complexities, allowing developers to focus on building applications without worrying about intricate details. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/core#ops) OPs Operations (OPs) refer to functionalities that extend beyond the ECMAScript specification. The V8 engine operates within a confined sandbox, strictly adhering to ECMAScript standards. However, the runtime environment manages additional services like networking, file input/output, timers, and more. Exploring the world of OPs is a significant topic that warrants a dedicated section. In the next segment, we will thoroughly examine OPs, illuminating their intricacies and importance within Deno's functionality. This upcoming section will provide a detailed understanding of OPs, their role in Deno, and their impact on the runtime environment. [](https://choubey.gitbook.io/internals-of-deno/architecture/core#ext) EXT ------------------------------------------------------------------------------- "Ext" refers to external APIs, a vital component of Deno's functionality. These external APIs consist of modules, each with a specific purpose. Two key components make up these modules: 1. **JS APIs accessible from user space**: These are JavaScript APIs designed to be called directly from your code. They provide a bridge between your application's logic and Deno's capabilities. You can use these JS APIs to interact with various functionalities offered by Deno, simplifying the process of utilizing Deno's features in your applications. 2. **OPs supporting the JS APIs**: Operations (OPs) support the JS APIs, executing the operations requested through the JS APIs. They are the behind-the-scenes workers that carry out tasks triggered by your code. In summary, Deno's "Ext" modules combine accessible JS APIs and underlying OPs, providing a seamless and effective way to interact with external functionalities while maintaining a clear division of responsibilities for better code organization and execution. This combination enables efficient and organized interaction with external capabilities, making it easier to build robust applications. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252FKNSRIQQmE4RxdHqxxo86%252Fdeon%2520ext%2520arch.png%3Falt%3Dmedia%26token%3D2ee61463-0098-4807-895d-12e4e12afbb6&width=768&dpr=4&quality=100&sign=81aa9dd0&sv=2) The "ext" component in Deno comprises a range of modules, each serving a distinct purpose. These modules include "broadcast\_channel", "cache", "console", "crypto", "fetch", "ffi", "fs", "http", "io", "kv", "napi", "net", "node", "tls", "url", "web", "webidl", "websocket", and "webstorage". Each module corresponds to a specific set of web APIs, providing various capabilities for that particular web API. A notable feature of the "ext" component is its extensibility, allowing developers to add new modules without modifying the existing codebase. This flexibility enables developers to enhance Deno's functionality by introducing custom modules tailored to their specific needs. In essence, the "ext" component serves as a comprehensive toolset, providing convenient access to various web-related functionalities. As technology advances and new web APIs emerge, the extensibility of Deno's "ext" component ensures adaptability and accommodation, fostering a platform for continuous growth and innovation. This modular approach makes sure that new web APIs do not touch the core Deno code. [](https://choubey.gitbook.io/internals-of-deno/architecture/core#core) Core --------------------------------------------------------------------------------- The Deno core component is a fundamental element within the Deno framework, playing a crucial role in its operation. Notably, this essential component is located outside of Deno's primary repository, in its own dedicated repository. The code that powers this core functionality is housed in this separate repository, highlighting the importance of the Deno core component as a standalone entity. [![Logo](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2Fgithub.com%2Ffluidicon.png&width=20&dpr=4&quality=100&sign=2b904219&sv=2)GitHub - denoland/deno\_core: The core engine at the heart of DenoGitHub](https://github.com/denoland/deno_core) The core component is the central hub of the Deno runtime, analogous to the heart of a living organism. This foundational framework provides essential functionalities that enable the smooth operation of Deno's runtime code. These functionalities include managing JavaScript APIs, executing JavaScript code segments, orchestrating an event loop for asynchronous operations, interacting with the V8 engine for efficient code execution, facilitating various operations (known as ops), and more. Notably, the core component is the first module loaded when Deno initializes its runtime environment. As the base of the Deno ecosystem, it sets the stage for the entire runtime's functionality. Additionally, the core component is responsible for loading external modules, which extend Deno's capabilities beyond its core functionalities. These modules enhance Deno's versatility, introducing new tools and features that cater to diverse programming needs. The following list highlights key functionalities provided by Deno's core framework: * Interfacing with V8: Enabling interaction with the V8 engine for efficient code execution * Data exchange: Facilitating data exchange between V8, Rust, and JavaScript * Executing operations: Executing both synchronous and asynchronous operations from JavaScript in Rust and returning the results * Console logging: Providing console logging capabilities for debugging and monitoring purposes * Maintaining OPs metrics: Tracking and managing metrics for operations (OPs) * Event loop: Orchestrating an event loop for smooth asynchronous operations * Low-level read and write APIs: Offering low-level APIs for reading and writing data [](https://choubey.gitbook.io/internals-of-deno/architecture/core#graph) Graph ----------------------------------------------------------------------------------- The graph component, as its name suggests, provides essential services to the Deno CLI for building a module graph. This graph is a map of how various modules depend on each other. The graph component is a Rust crate containing the fundamental code for constructing the module graph, aligning with the Deno CLI's module resolution logic. Additionally, it offers a web assembly interface to the generated code, allowing the constructed logic to be accessed and utilized from JavaScript or TypeScript environments. This feature expands the module graph's functionality beyond the Deno CLI, enabling its use in diverse contexts. The graph component starts with the main file or main module, loading all dependencies recursively using a simple graph data structure. This structure keeps track of visited modules, avoiding repeated reloading and recompilation. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252FYL9IKFl2PfSvswQnLimO%252Fjs%2520module%2520graph%2520overview.png%3Falt%3Dmedia%26token%3Db151006b-0701-406b-bb54-c18a19b0ea51&width=768&dpr=4&quality=100&sign=69f60bf0&sv=2) As we work through a simple program with dependencies, we will gain a deeper understanding of the graph component's inner workings. This exploration will give us a thorough understanding of how it works and why it's important. [](https://choubey.gitbook.io/internals-of-deno/architecture/core#npm) NPM ------------------------------------------------------------------------------- The NPM component, as its name suggests, is responsible for managing packages obtained from NPM. This involves finding and gathering the necessary packages, ensuring they are properly installed, and maintaining them over time. The NPM component in Deno serves as a guide, navigating the vast NPM repository, understanding package metadata, and more. However, to keep this book simple, we will not explore the details of NPM packages when discussing Deno's inner workings. [](https://choubey.gitbook.io/internals-of-deno/architecture/core#cache) Cache ----------------------------------------------------------------------------------- The cache component's code is located in two places: within Deno's CLI code and in a separate crate for external use. Initially developed for Deno CLI, its utility extended to other contexts, allowing consistent cache access. Like Deno CLI, other components like deno\_graph, deno\_doc, dnt, and emit can interact with and populate the cache using the same methods. The cache system stores downloaded packages and modules on your computer's disk, acting as a storage space. The main storage location is the DENO\_DIR/cache folder. Here's how it works: * Deno checks the cache for a needed module. * If found, Deno uses the cached version instead of redownloading. * If not found, Deno downloads the module and stores a copy in the cache for future use. In simple terms, the cache system is a convenient storage area for JavaScript modules, saving them for quick access and avoiding repeated downloads. We'll explore the cache concept further as we work through a program with dependencies. \-- We have now discussed the components that are native to Deno, not relying on third-party sources. Next, we will examine the components that come from external sources. However, before we do that, we will take a closer look at the Operations (OPs) to gain a better understanding of their role in Deno's architecture. [Previous2.3 Programming Languages](https://choubey.gitbook.io/internals-of-deno/architecture/2.3-programming-languages) [Next2.5 OPs](https://choubey.gitbook.io/internals-of-deno/architecture/ops) Last updated 1 year ago --- # 3.0 Cover page | The Internals of Deno ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252F8TV7znl14t45d3xaqY45%252F4.png%3Falt%3Dmedia%26token%3D48addef2-e43d-4709-9d38-c4d52952a44c&width=768&dpr=4&quality=100&sign=f35e7b2d&sv=2) [Previous2.10 What's next](https://choubey.gitbook.io/internals-of-deno/architecture/2.7-whats-next) [Next3.1 Threading model](https://choubey.gitbook.io/internals-of-deno/threading-model/3.1-threading-model) Last updated 1 year ago --- # 4.2 Print | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/bridge/4.2-print#overview) Overview ---------------------------------------------------------------------------------------- Let's begin by exploring one of the most straightforward and quite frequently used external reference in Deno: **printing data**. According to the ECMAScript specification, there isn't a predefined function known as console.log. The availability of this function depends on the particular implementation, so it's up to the user to handle it. The V8 engine does offer some logging capabilities, but these are primarily focused on the inner workings of the core engine itself. Unfortunately, V8 does not extend its support to user-level logging, including the widely used console.log function. The widely used `console.log` function is implemented externally in Deno, meaning that Deno itself provides this functionality. This function allows you to display messages and information in the terminal or console while you are running your Deno programs. [](https://choubey.gitbook.io/internals-of-deno/bridge/4.2-print#function) Function ---------------------------------------------------------------------------------------- ### [](https://choubey.gitbook.io/internals-of-deno/bridge/4.2-print#registration) Registration The registration of the print function happens at the startup: Copy pub fn op_print(#[string] msg: &str, is_err: bool) -> Result<(), Error> { The op\_print function stands as an inherent operation within Deno's framework. Similar to other operations (ops), this function is enlisted as an external reference. ### [](https://choubey.gitbook.io/internals-of-deno/bridge/4.2-print#js-space) JS space Let's see how console.log is implemented in JS space. The function console.log comes from the console class: Copy const windowOrWorkerGlobalScope = { // -- CODE OMITTED -- console: util.nonEnumerable( new console.Console((msg, level) => core.print(msg, level > 1)), ) // -- CODE OMITTED -- } Let's take a look at how console.log is implemented in Deno. It's worth noting that the printFunc used here is essentially the same as core.print. Copy log = (...args) => { this.#printFunc( inspectArgs(args, { ...getConsoleInspectOptions(), indentLevel: this.indentLevel, }) + "\n", 1, ); }; The core.print simply invokes op\_print API: Copy print: (msg, isErr) => ops.op_print(msg, isErr), ### [](https://choubey.gitbook.io/internals-of-deno/bridge/4.2-print#rust-space) Rust space The op\_print function is a basic tool in Deno that uses Rust's standard APIs to print data to the console. By using Rust's std library, this function provides the simplest way to display information to the user through the console. Copy pub fn op_print(#[string] msg: &str, is_err: bool) -> Result<(), Error> { if is_err { stderr().write_all(msg.as_bytes())?; stderr().flush().unwrap(); } else { stdout().write_all(msg.as_bytes())?; stdout().flush().unwrap(); } Ok(()) } Here, we have a straightforward function at play. Its primary task is to take some data as input and transform it into bytes. Once this transformation is complete, the function proceeds to send these bytes to the standard output or standard error, depending on the nature of the data. [Previous4.1 The bridge](https://choubey.gitbook.io/internals-of-deno/bridge/4.1-the-bridge) [Next4.3 Encode and decode](https://choubey.gitbook.io/internals-of-deno/bridge/4.4-encode-and-decode) Last updated 1 year ago --- # 2.9 V8 | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/architecture/v8#overview) Overview --------------------------------------------------------------------------------------- V8 is a fundamental component of Deno's architecture, playing a crucial role in the execution of JavaScript code. While Tokio is also essential, V8's presence is indispensable, and its absence would significantly impair Deno's functionality. V8, developed by Google, is a high-performance JavaScript and WebAssembly engine, released as open-source software. Written in C++, it has evolved into a powerful tool, extending its capabilities beyond Deno to Chrome and Node.js. As a new addition to Deno, V8 continues to empower this innovative runtime. V8 provides a supportive environment for ECMAScript and WebAssembly, allowing them to thrive. Its compatibility is extensive, supporting various operating systems, including Windows 7 and later, macOS 10.12 and higher, and Linux systems with x64, IA-32, ARM, or MIPS processors. V8 can function as a standalone engine or integrate with C++ applications, significantly enhancing their capabilities. V8 assumes a multifaceted role, encompassing various critical functions. It initiates its processes by orchestrating the compilation and execution of JavaScript source code, thereby creating virtual environments. Additionally, V8 takes on the responsibility of managing memory allocation for a diverse range of objects, including the graceful removal of unnecessary objects through garbage collection, a unique strength that distinguishes it from other technologies. The precision of V8's garbage collector is a crucial factor in its exceptional performance. V8 also bridges the gap between C++ and JavaScript, enabling seamless interactions. With remarkable finesse, V8 allows C++ applications to share their objects and functions with JavaScript code, fostering harmonious collaboration. This interaction remains under the control of the application, which can selectively expose objects and functions to the JavaScript domain. To explore the complex features of V8, one can utilize its comprehensive public API. A wealth of information and resources is available at [https://v8.dev/](https://v8.dev/) , inviting individuals to investigate and discover the full potential of this powerful engine. [](https://choubey.gitbook.io/internals-of-deno/architecture/v8#key-concepts) Key concepts ----------------------------------------------------------------------------------------------- V8, the core of Deno's runtime, is a vast and complex component, consisting of approximately one million lines of code. This powerful engine features a extensive public API, offering numerous functionalities and capabilities. Due to its complexity, navigating V8 in its entirety can be challenging, so we will focus on key concepts to provide a foundational understanding. To demonstrate V8's importance, consider a simple "hello world" program, which showcases the engine's significance. (ref: [https://chromium.googlesource.com/v8/v8/+/refs/tags/11.8.34/samples/hello-world.cc](https://chromium.googlesource.com/v8/v8/+/refs/tags/11.8.34/samples/hello-world.cc) ) Copy // Copyright 2015 the V8 project authors. All rights reserved. // Use of this source code is governed by a BSD-style license that can be // found in the LICENSE file. #include #include #include #include "include/libplatform/libplatform.h" #include "include/v8-context.h" #include "include/v8-initialization.h" #include "include/v8-isolate.h" #include "include/v8-local-handle.h" #include "include/v8-primitive.h" #include "include/v8-script.h" int main(int argc, char* argv[]) { // Initialize V8. v8::V8::InitializeICUDefaultLocation(argv[0]); v8::V8::InitializeExternalStartupData(argv[0]); std::unique_ptr platform = v8::platform::NewDefaultPlatform(); v8::V8::InitializePlatform(platform.get()); v8::V8::Initialize(); // Create a new Isolate and make it the current one. v8::Isolate::CreateParams create_params; create_params.array_buffer_allocator = v8::ArrayBuffer::Allocator::NewDefaultAllocator(); v8::Isolate* isolate = v8::Isolate::New(create_params); { v8::Isolate::Scope isolate_scope(isolate); // Create a stack-allocated handle scope. v8::HandleScope handle_scope(isolate); // Create a new context. v8::Local context = v8::Context::New(isolate); // Enter the context for compiling and running the hello world script. v8::Context::Scope context_scope(context); { // Create a string containing the JavaScript source code. v8::Local source = v8::String::NewFromUtf8Literal(isolate, "'Hello' + ', World!'"); // Compile the source code. v8::Local script = v8::Script::Compile(context, source).ToLocalChecked(); // Run the script to get the result. v8::Local result = script->Run(context).ToLocalChecked(); // Convert the result to an UTF8 string and print it. v8::String::Utf8Value utf8(isolate, result); printf("%s\n", *utf8); } { // Use the JavaScript API to generate a WebAssembly module. // // |bytes| contains the binary format for the following module: // // (func (export "add") (param i32 i32) (result i32) // get_local 0 // get_local 1 // i32.add) // const char csource[] = R"( let bytes = new Uint8Array([\ 0x00, 0x61, 0x73, 0x6d, 0x01, 0x00, 0x00, 0x00, 0x01, 0x07, 0x01,\ 0x60, 0x02, 0x7f, 0x7f, 0x01, 0x7f, 0x03, 0x02, 0x01, 0x00, 0x07,\ 0x07, 0x01, 0x03, 0x61, 0x64, 0x64, 0x00, 0x00, 0x0a, 0x09, 0x01,\ 0x07, 0x00, 0x20, 0x00, 0x20, 0x01, 0x6a, 0x0b\ ]); let module = new WebAssembly.Module(bytes); let instance = new WebAssembly.Instance(module); instance.exports.add(3, 4); )"; // Create a string containing the JavaScript source code. v8::Local source = v8::String::NewFromUtf8Literal(isolate, csource); // Compile the source code. v8::Local script = v8::Script::Compile(context, source).ToLocalChecked(); // Run the script to get the result. v8::Local result = script->Run(context).ToLocalChecked(); // Convert the result to a uint32 and print it. uint32_t number = result->Uint32Value(context).ToChecked(); printf("3 + 4 = %u\n", number); } } // Dispose the isolate and tear down V8. isolate->Dispose(); v8::V8::Dispose(); v8::V8::DisposePlatform(); delete create_params.array_buffer_allocator; return 0; } This straightforward program is designed to display the phrase "hello world." Although it appears simple, the hello world program offers valuable insights into several key V8 concepts. Let's embark on a step-by-step exploration of each concept. This code's sole purpose is to output the iconic greeting "hello world." Despite its seeming simplicity, the hello world program serves as a gateway to a range of crucial V8 concepts. Let's proceed to examine each concept individually, delving into each one step by step. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/v8#isolate) Isolate In V8, an "isolate" represents a separate instance of the engine, responsible for executing JavaScript code. Each isolate operates independently, with its own environment, data, and variables. It's essential to note that objects or elements from one isolate should not be used in another. When V8 is initialized, it automatically creates a default isolate and sets it as the current context. This default isolate serves as the starting point for V8's activities. However, the programmer (embedder) can create additional isolates, which can run simultaneously, enabling parallel code execution on multiple threads. Note that only one thread can interact with an isolate at a time. Think of isolates as self-contained virtual machines (VMs), each with a dedicated heap for storing data during runtime. Initializing an isolate is the first step, setting the stage for subsequent actions. By managing isolates effectively, developers can create efficient and organized environments for running JavaScript code. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/v8#context) Context In V8, a "context" represents a unique execution environment that enables multiple, unrelated JavaScript applications to run within a single V8 instance. To specify a context for executing JS code, a user program must provide the necessary information. Once established, a context can be entered and exited as needed. Imagine executing JavaScript operations in context A, then seamlessly transitioning to context B, replacing A as the active environment. When exiting context B, context A is reinstated, ensuring a smooth transition. V8 introduced contexts to create isolated JavaScript environments for elements like windows and iframes in web browsers, preventing interference between their JavaScript activities. The utility of contexts becomes apparent when isolation is crucial for smooth, independent functioning of components within a larger application or system. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/v8#compile-and-run) Compile and run The process proceeds with the compilation and execution of JavaScript (JS) code. You may wonder why compilation is necessary, as JS is typically interpreted. However, V8 takes it a step further by incorporating compilation for enhanced performance. This additional step may seem unnecessary, but it's a crucial aspect of V8's functionality. But why compile JS code? Initially, JS is interpreted, but V8's compilation phase boosts execution speed. The secret to V8's speed lies in Just in Time (JIT) compilation. Here's how it works: JS code is compiled into native machine code via JIT. During execution, this machine code is dynamically analyzed and recompiled for optimal performance. This process ensures that the code is executed efficiently, resulting in faster execution times. This dynamic approach significantly enhances speed, much like a car's finely-tuned engine constantly adjusting for optimal performance. In V8, continuous machine code analysis and optimization enable swift JS program execution. The end result is a significant boost in speed, making V8 a powerful engine for executing JavaScript code. \-- This section has provided a foundational understanding of V8, but a comprehensive examination of its inner workings requires a more in-depth approach. The topic of V8 is complex and nuanced, necessitating a thorough and detailed analysis to fully grasp its underlying mechanics. A brief overview is insufficient to capture the full scope of V8's complexity, and a more extensive investigation is necessary to gain a complete understanding of its intricate details. [Previous2.8 Tokio](https://choubey.gitbook.io/internals-of-deno/architecture/tokio) [Next2.10 What's next](https://choubey.gitbook.io/internals-of-deno/architecture/2.7-whats-next) Last updated 1 year ago --- # 4.0 Cover page | The Internals of Deno ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252F4vi9St4FZCfYJflZj0Me%252F5.png%3Falt%3Dmedia%26token%3D6179c7df-2073-47ea-851f-77a6b331372f&width=768&dpr=4&quality=100&sign=4f99d8e7&sv=2) [Previous3.4 What's next](https://choubey.gitbook.io/internals-of-deno/threading-model/3.4-whats-next) [Next4.1 The bridge](https://choubey.gitbook.io/internals-of-deno/bridge/4.1-the-bridge) Last updated 1 year ago --- # 3.2 Default threading model | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/threading-model/default-threads#overview) Overview ------------------------------------------------------------------------------------------------------- In JavaScript, there is a notable concept: _single-threaded execution_. JavaScript is known for running on a single thread, which handles all tasks. Think of this thread as a dedicated worker responsible for everything within the program's operation. Node.js popularized this approach, calling it the "single-threaded event loop." This term describes how the single thread efficiently manages and cycles through various events and actions. Despite being a single thread, it keeps things moving by effectively handling its tasks. This unique approach has become a characteristic of JavaScript, and its efficiency is notable. [](https://choubey.gitbook.io/internals-of-deno/threading-model/default-threads#single-threaded-execution) Single-threaded execution ----------------------------------------------------------------------------------------------------------------------------------------- This single thread is responsible for managing JavaScript code and handling various tasks. To better understand this concept, consider a simple JavaScript HTTP server that echoes back what it receives from clients. The sole thread handles multiple responsibilities, including: * Running JavaScript code * Monitoring the socket for new TCP connections * Managing multiple incoming requests from the socket simultaneously * Creating request and response objects * Parsing JSON data * Assigning a unique request-id to each request * Converting responses to JSON format * Sending JSON-encoded responses over the TCP connection * Closing the socket It's essential to note that despite its small footprint, this single thread plays a vital role in efficiently managing these complex tasks. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MPpo7M-qfVHz6n-Nb3h%252F-MPpoNcoxhYpcCZjZGrT%252FJS%2520single%2520thread.png%3Falt%3Dmedia%26token%3D949b30b8-c841-4859-b393-22f278ba5f21&width=768&dpr=4&quality=100&sign=f5c93a71&sv=2) A single thread manages all concurrent tasks, creating a competitive environment where tasks vie for processing time. Referring to the diagram above, you'll see all requests being processed within this solitary thread. It's a competitive struggle for these requests to be included in the thread's schedule. In contrast, programming languages like Java often employ a different approach. They use a manager with a fixed-sized thread pool, where incoming requests are received and delegated to available threads for processing. The manager assigns a free thread to handle each request. If all threads are occupied, the manager queues the request until a thread becomes available. When comparing the single thread approach with the thread pool approach, it may seem like JavaScript is inefficient. However, this is not entirely accurate. Although JavaScript executes in a single thread, it has a clever way of managing this limitation. This thread handles incoming requests from multiple clients, and its efficient management enables it to cope with the workload effectively. The key to its efficiency lies in its use of asynchronous processing and callbacks, which allows it to manage multiple tasks simultaneously. JavaScript uses asynchronous programming, which means it can start a task, move on to another task while the first one is processing in the background, and then return to handle the results when ready. This approach allows JavaScript to handle multiple tasks concurrently without being limited by a single thread. While it may not be the fastest, JavaScript's approach enables effective concurrency management. Does this mean only one thread is operating within the Deno process? The answer is no. We will explore this further shortly. [](https://choubey.gitbook.io/internals-of-deno/threading-model/default-threads#additional-threads) Additional threads --------------------------------------------------------------------------------------------------------------------------- Deno, like other JavaScript runtimes, operates in a single-threaded environment. This means one thread handles all tasks, as discussed earlier. However, Deno can create additional application threads when needed, known as web workers. Web workers enable Deno to perform concurrent tasks efficiently. Think of separate workers handling different tasks, like data processing or user interactions, independently. These workers operate separately from the main thread, optimizing system resource usage and performance. Each web worker has its own isolated execution context, preventing potential conflicts in a multi-threaded scenario. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MPpoWwbvXgZEFF2wMzD%252F-MPppAMu9Qz4TzshNjpT%252Fdeno%2520worker%2520types.png%3Falt%3Dmedia%26token%3Dbcb72940-14f1-4009-b365-d452db23a374&width=768&dpr=4&quality=100&sign=1159e1cd&sv=2) Web workers operate independently but maintain a connection with their parent thread, enabling bidirectional message exchange. Apart from this communication channel, web workers have complete autonomy. Each web worker is assigned a fresh V8 snapshot, serving as a unique starting point for execution. Additionally, each worker has its own event loop or Tokio runtime, allowing separate task management and enhancing efficiency. The significance of web workers lies in their ability to effectively manage CPU-intensive tasks, which could otherwise overwhelm the main thread. Delegating these tasks to web workers ensures the main thread can continue its operations without disruption, enhancing overall performance and responsiveness. This separation of duties optimizes computational resource utilization, making web workers a valuable tool. By leveraging web workers, developers can build more efficient and scalable applications. [](https://choubey.gitbook.io/internals-of-deno/threading-model/default-threads#denos-default-threading-model) Deno's default threading model -------------------------------------------------------------------------------------------------------------------------------------------------- Deno executes JavaScript programs in a single loop, similar to other JavaScript runtimes. At first, it may seem like Deno operates with only one thread. However, the reality is more complex. Deno runs JavaScript code using a single thread, but it functions as a multithreaded process underneath. This dual behavior may seem contradictory at first, but it's a crucial aspect of Deno's architecture. Let's explore this in more detail. When you run a JavaScript program in Deno, it enters a single-threaded environment, executing sequentially, step by step. Deno then leverages its multithreaded nature to enhance efficiency and resource management. The Deno process uses two types of threads: the primary thread for executing JavaScript code and additional V8 threads for garbage collection (GC) purposes. Deno uses additional threads from the V8 engine by default. The V8 engine executes JavaScript code in the main thread's context. To boost JavaScript execution speed, V8 uses extra threads for CPU-intensive tasks like garbage collection and ahead-of-time compilation. While it's possible to configure V8 to run on a single thread, this would significantly impact performance. With only one thread, resource-intensive tasks would overload the main thread, hindering overall performance and delaying program execution. This would limit the main thread's availability for running user programs, potentially causing disruptions. Using a single-threaded V8 configuration could have severe consequences. It could lead to slow responsiveness, longer execution times, and a poor user experience. Therefore, V8's current use of multiple threads is crucial for efficient and smooth JavaScript execution within the Deno runtime environment. When you start a Deno process with its default settings, it uses 8 threads by default. This applies when no web workers are present. Web workers, as we know, are specialized scripts that run in the background, performing tasks without blocking the main program's execution. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MOKfYuI08nbszx4mI7I%252F-MOKgMepZb5HzgwldT16%252Fdeno%2520default%2520threading%2520model.png%3Falt%3Dmedia%26token%3D2f442412-0ae0-4941-afcd-03ba41f87cfb&width=768&dpr=4&quality=100&sign=e7757fc&sv=2) Deno's thread allocation is organized as follows: * **Main Worker Thread (Main Thread)**: This is the central thread in every Deno application, responsible for coordinating tasks and facilitating communication between program parts. * **V8 Threads**: Deno uses seven additional V8 threads to enhance performance and responsiveness. These threads work together to execute JavaScript code efficiently and manage runtime operations. By leveraging multiple threads, Deno improves task execution and resource utilization. ### [](https://choubey.gitbook.io/internals-of-deno/threading-model/default-threads#the-main-thread) **The main thread** This section focuses on the primary thread, which is under the ownership and supervision of the Tokio Runtime. This particular thread is responsible for overseeing the entire operation. Below, you'll find the stack trace for the main thread, offering insights into its sequence of actions and functions. This stack trace provides a clear view of how tasks are executed and controlled within this central thread. Copy 2731 Thread_8461543 DispatchQueue_1: com.apple.main-thread (serial) + 2731 start (in libdyld.dylib) + 1 [0x7fff6dd53cc9] + 2731 main (in deno) + 418 [0x108ecfada] + 2731 std::sys_common::backtrace::__rust_begin_short_backtrace::h4e8d5235f9254db6 (in deno) + 10 [0x108d368d1] + 2731 deno::main::h7d1b5a97f8aef853 (in deno) + 10706 [0x108ec8676] + 2731 tokio::runtime::Runtime::block_on::h9f5c6c3dddbd431f (in deno) + 1768 [0x108d9fa5c] + 2731 _$LT$tokio..park..either..Either$LT$A$C$B$GT$$u20$as$u20$tokio..park..Park$GT$::park::h123a854f0de101c3 (in deno) + 204 [0x109439078] + 2731 _$LT$tokio..park..either..Either$LT$A$C$B$GT$$u20$as$u20$tokio..park..Park$GT$::park_timeout::h6834b5b57b34a394 (in deno) + 78 [0x109439308] + 2731 tokio::io::driver::Driver::turn::h1e669b6b05307d9f (in deno) + 72 [0x1094394df] + 2731 mio::poll::Poll::poll::hbd211acf9552cbd4 (in deno) + 763 [0x109148978] + 2731 kevent (in libsystem_kernel.dylib) + 10 [0x7fff6de99766] As evident from the stack trace, the primary thread is under the management of the tokio runtime. This thread is responsible for executing tokio's event loop, a crucial mechanism that drives the flow of operations. In this setup, the tokio runtime takes charge of coordinating tasks and ensuring efficient execution. ### [](https://choubey.gitbook.io/internals-of-deno/threading-model/default-threads#v8s-worker-threads) V8's worker threads Deno has seven v8 worker threads by default. These threads handle essential tasks like garbage collection, runtime optimizations, and Just-In-Time (JIT) compilation. However, they do not directly execute JavaScript code. To understand their role in Deno's operation, let's explore what these worker threads do when your Deno application runs. They handle behind-the-scenes tasks, such as memory cleanup through garbage collection, which improves code efficiency. They also contribute to runtime optimizations, ensuring smooth and fast code execution. Additionally, they perform Just-In-Time compilation, translating high-level JavaScript code into lower-level machine code for faster execution. The stack trace of a worker thread provides a snapshot of its activity. The stack trace, similar for all worker threads, offers insight into the processes and functions managed by these threads. Copy 2731 Thread_8461546: V8 DefaultWorke + 2731 thread_start (in libsystem_pthread.dylib) + 15 [0x7fff6df53b8b] + 2731 _pthread_start (in libsystem_pthread.dylib) + 148 [0x7fff6df58109] + 2731 v8::base::ThreadEntry(void*) (in deno) + 87 [0x10948c547] + 2731 v8::platform::DefaultWorkerThreadsTaskRunner::WorkerThread::Run() (in deno) + 31 [0x109491e6f] + 2731 v8::platform::DelayedTaskQueue::GetNext() (in deno) + 708 [0x109492644] + 2731 _pthread_cond_wait (in libsystem_pthread.dylib) + 698 [0x7fff6df58425] + 2731 __psynch_cvwait (in libsystem_kernel.dylib) + 10 [0x7fff6de97882] [](https://choubey.gitbook.io/internals-of-deno/threading-model/default-threads#denos-threading-model-with-web-workers) Deno's threading model with web workers -------------------------------------------------------------------------------------------------------------------------------------------------------------------- If the application has generated a web worker, the allocation of threads functions in the following manner: ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MOKhhS_lOpwCtMLaS6G%252F-MONYjXfEtyYLIQikmVY%252Fdeno%2520threading%2520with%2520webworkers.png%3Falt%3Dmedia%26token%3D7b91fde9-1d24-4371-bea4-1afb4dc55c58&width=768&dpr=4&quality=100&sign=f7add712&sv=2) The situation becomes more interesting when a web worker is added. The total thread count increases to 9, distributed as follows: * 1 main worker thread (also known as the main thread) * 1 web worker thread * 7 V8 threads This adds up to 9 threads in total. Notably, the number of V8 worker threads remains the same even when an additional web worker is introduced. However, it's important to note that users can increase the number of V8 worker threads if needed, depending on their specific use case requirements. Now, let's examine the stack trace of this new worker thread: Copy 2718 Thread_8525355: deno-worker-0 2718 thread_start (in libsystem_pthread.dylib) + 15 [0x7fff6df53b8b] 2718 _pthread_start (in libsystem_pthread.dylib) + 148 [0x7fff6df58109] 2718 std::sys::unix::thread::Thread::new::thread_start::hd4805e9612a32deb (in deno) + 45 [0x10a2e1b9d] 2718 core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h7e30e5a55a4af9fa (in deno) + 116 [0x109d60e85] 2718 std::sys_common::backtrace::__rust_begin_short_backtrace::h30b57608d94f7821 (in deno) + 2856 [0x109d507e6] 2718 tokio::runtime::Runtime::block_on::h9f5c6c3dddbd431f (in deno) + 1768 [0x109dbaa5c] 2718 _$LT$tokio..park..either..Either$LT$A$C$B$GT$$u20$as$u20$tokio..park..Park$GT$::park::h123a854f0de101c3 (in deno) + 204 [0x10a454078] 2718 _$LT$tokio..park..either..Either$LT$A$C$B$GT$$u20$as$u20$tokio..park..Park$GT$::park_timeout::h6834b5b57b34a394 (in deno) + 78 [0x10a454308] 2718 tokio::io::driver::Driver::turn::h1e669b6b05307d9f (in deno) + 72 [0x10a4544df] 2718 mio::poll::Poll::poll::hbd211acf9552cbd4 (in deno) + 763 [0x10a163978] 2718 kevent (in libsystem_kernel.dylib) + 10 [0x7fff6de99766] The web worker's stack trace is similar to the main thread's stack trace, with the only difference being the starting point within Deno. In the main thread, the thread starts in deno::main, while in the web worker, it starts in core::ops. The core::ops is where the web worker is created. Now, let's look at an example that shows how threads are distributed when an application creates five web worker threads: ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MONZx8Au7ne4ZGiAAYZ%252F-MON_LXE5HHJHPKqsFsl%252Fdeno%2520threading%2520with%25205%2520webworkers.png%3Falt%3Dmedia%26token%3D5c0af859-1385-4603-85ca-55753e18b532&width=768&dpr=4&quality=100&sign=638549c7&sv=2) The pattern is clear. When using five web workers, the system uses a total of 13 threads, distributed as follows: * 1 primary worker thread (main thread) * 5 web worker threads * 7 V8 threads This adds up to 13 threads working together to execute tasks and processes in the system. [](https://choubey.gitbook.io/internals-of-deno/threading-model/default-threads#more-on-threads) More on threads --------------------------------------------------------------------------------------------------------------------- In the previous section, we explored Deno's threading model, with and without web workers. However, Deno uses additional threads beyond these. The threads we've discussed so far are OS-aware static threads, created at the program's start. Note that applications can create web workers at runtime, but this can be costly. Moreover, Deno may introduce new threads in the future to handle asynchronous operations. These dynamic threads, or green threads, are created through tokio. It's essential to understand that these threads operate differently from the previous ones, as they are managed by tokio, not the operating system's scheduler. [Previous3.1 Threading model](https://choubey.gitbook.io/internals-of-deno/threading-model/3.1-threading-model) [Next3.3 Asynchronous green threads](https://choubey.gitbook.io/internals-of-deno/threading-model/tokio-threads) Last updated 1 year ago --- # 5.1 Hello world program | The Internals of Deno We have discussed the theoretical aspects of Deno, including its introduction, architecture, threading model, and the Deno-v8 bridge. Now, we will explore how a program works within the Deno environment. We will start with a simple program that prints a message, like "hello world," to the console. Why start with a simple program? We want to focus on the basic elements of Deno. Understanding these elements is essential for grasping how Deno works. By analyzing Deno's code, we can see how even the simplest program is executed. Once we understand these basics, we can move on to more complex programs. This chapter contains a lot of important information, so take your time to understand it thoroughly. Don't rush through it. This chapter, along with the next one, is the core of this book. To get the most out of it, we recommend reading this chapter twice before moving on to the next content. This will help you learn more effectively. [](https://choubey.gitbook.io/internals-of-deno/foundations/hello-world#chapter-contents) Chapter contents --------------------------------------------------------------------------------------------------------------- [5.2 Basic hello world](https://choubey.gitbook.io/internals-of-deno/foundations/basic-hello-world) [5.3 Main program of Deno](https://choubey.gitbook.io/internals-of-deno/foundations/main-program) [5.4 Module Specifier](https://choubey.gitbook.io/internals-of-deno/foundations/resolve_url_or_path) [5.5 CLI Factory](https://choubey.gitbook.io/internals-of-deno/foundations/program-state) [5.6 Permissions](https://choubey.gitbook.io/internals-of-deno/foundations/permissions) [5.7 Main Worker](https://choubey.gitbook.io/internals-of-deno/foundations/mainworker) [5.8 JS Runtime](https://choubey.gitbook.io/internals-of-deno/foundations/jsruntime) [5.9 Run main module](https://choubey.gitbook.io/internals-of-deno/foundations/execute_module) [5.10 Load module](https://choubey.gitbook.io/internals-of-deno/foundations/load-module) [5.11 Recursive module loading](https://choubey.gitbook.io/internals-of-deno/foundations/4.11-recursive-module-loading-and-module-graphs) [5.12 Module graphs](https://choubey.gitbook.io/internals-of-deno/foundations/4.12-module-graphs) [5.13 File fetching](https://choubey.gitbook.io/internals-of-deno/foundations/4.13-file-fetcher) [5.14 Transpile](https://choubey.gitbook.io/internals-of-deno/foundations/4.13-check-and-or-transpilation) [5.15 Register / compile module](https://choubey.gitbook.io/internals-of-deno/foundations/register-and-instantation) [5.16 Instantiate module](https://choubey.gitbook.io/internals-of-deno/foundations/instantiate-module) [5.17 Evaluate module](https://choubey.gitbook.io/internals-of-deno/foundations/evaluate-module) [Previous5.0 Cover page](https://choubey.gitbook.io/internals-of-deno/foundations/chapter-cover-page) [Next5.2 Basic hello world](https://choubey.gitbook.io/internals-of-deno/foundations/basic-hello-world) Last updated 1 year ago --- # 4.1 The bridge | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/bridge/4.1-the-bridge#overview) Overview --------------------------------------------------------------------------------------------- The Deno runtime uses Google's v8 engine to execute JavaScript code that it loads. However, Deno's responsibilities don't end there. It can't simply let v8 handle everything. The v8 engine only supports the features outlined in the ECMAScript specification. Any functionality beyond this specification is the responsibility of the v8 engine's user, not Deno. For example, the console.log function isn't part of the ECMAScript specification. Therefore, whoever uses the v8 engine must implement this function themselves. Similarly, tasks like making HTTP calls to servers aren't within the scope of the ECMAScript specification. So, the user of the v8 engine is responsible for implementing such functionality. Many operations fall outside the scope of the ECMAScript specification, including file management, process management, network communications, and system-level tasks. Anything beyond the bounds of the specification requires the user to provide support within the v8 engine. In summary, it's important to understand that the v8 engine relies on the user for assistance with anything beyond the ECMAScript specification. Deno's role continues even after code is loaded into v8, facilitating the implementation of additional functionalities, making the system functional and versatile. [](https://choubey.gitbook.io/internals-of-deno/bridge/4.1-the-bridge#v8-less-than-greater-than-deno) V8 <-> Deno ---------------------------------------------------------------------------------------------------------------------- The JavaScript application runs using the v8 engine. Deno's primary function is to offer assistance for all the aspects that exist beyond this specific engine's capabilities. Imagine the connection between the v8 user (which is Deno in this case) and the v8 engine as a bridge. This analogy arises from the distinctions between v8 and Deno. This bridge functions in both directions, resembling a two-way passage. It is important to note that there exists a predetermined way to traverse this bridge while ensuring a smooth round trip. Just like a bridge, Deno acts as a facilitator, allowing the JavaScript code to access functionalities beyond the v8 engine's confines and return seamlessly. This interaction forms an essential part of Deno's role in enhancing the capabilities of the JavaScript application. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MPqQnDbx0GeoAeKv2qX%252F-MPqSOFBAPNZGAvVvbxK%252Fdeno%2520v8%2520bridge.png%3Falt%3Dmedia%26token%3D5232f7eb-58de-4cbe-96d9-c7be8edaa221&width=768&dpr=4&quality=100&sign=40de098c&sv=2) In programming, crossing the bridge is like making a function call, but with a difference. This special connection is called a "bridge" because v8 and Deno are different in nature. It's important to understand that Deno and v8 are separate entities with their own identities. To explain further, Deno is a standalone software written in Rust, while v8 is a separate entity written in C++. ### [](https://choubey.gitbook.io/internals-of-deno/bridge/4.1-the-bridge#handling-un-known-functions) Handling (un)known functions Using external functions follows a straightforward logic. Whenever the V8 engine encounters something that lies beyond the boundaries of the ECMAScript specification, it takes specific actions: ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MPqTIunczFYht2iIaWj%252F-MPqUL1KbSulmoXHnghz%252Fdeno%2520v8%2520external%2520ref%2520steps.png%3Falt%3Dmedia%26token%3D0740d174-ffc2-43ae-bcb6-292d654fe190&width=768&dpr=4&quality=100&sign=19c8c7d0&sv=2) The V8 algorithm outlines the steps it takes when encountering a function call. Here's a breakdown in simpler terms: When a function is called: 1. First, the algorithm looks at the name of the function in its internal data records. 2. If the algorithm recognizes the function (meaning it's a function that follows the rules of ECMAScript and is supported by V8), it proceeds to execute that function. 3. But if the function isn't recognized by the algorithm, it moves to the next step. 4. In this case, if the user has given V8 information about an external function in advance, the algorithm temporarily stops its current process. 5. It then calls the external function provided by the user. 6. The algorithm waits until a response is received from this external function. 7. Once the response is ready, the algorithm resumes its work using the obtained response. 8. If no external function is registered or if the response doesn't come through, the algorithm raises an error to indicate the issue. These external functions that the algorithm can use are known as external references. Beforehand, these references must be added to the V8's knowledge. This ensures that V8 is aware of these external functions and can use them as needed during its execution. This registration step is important for smooth interaction between the algorithm and external code. [](https://choubey.gitbook.io/internals-of-deno/bridge/4.1-the-bridge#registration-of-external-references) Registration of external references --------------------------------------------------------------------------------------------------------------------------------------------------- During its initialization process, Deno systematically records all external references with the v8 engine. This crucial step guarantees that v8 has these references properly set up whenever the fundamental Deno runtime is activated. Below, you will find a compilation of some of the external references that Deno registers, contributing to the seamless functioning of the runtime. * Built-in OPs (all the OPs are the basic built-in OPs) * Extension OPs (These OPs are provided by extensions) * import\_meta\_resolve * catch\_dynamic\_import\_promise\_error * wasm\_async\_resolve\_promise\_callback * host\_import\_module\_dynamically\_callback * host\_initialize\_import\_meta\_object\_callback * import\_meta\_resolve * empty\_fn * catch\_dynamic\_import\_promise\_error * etc. Below is a code snippet that illustrates how external functions are registered with the V8 engine: Copy references.push(v8::ExternalReference { function: call_console.map_fn_to(), }); references.push(v8::ExternalReference { function: import_meta_resolve.map_fn_to(), }); references.push(v8::ExternalReference { function: catch_dynamic_import_promise_error.map_fn_to(), }); references.push(v8::ExternalReference { function: empty_fn.map_fn_to(), }); ### [](https://choubey.gitbook.io/internals-of-deno/bridge/4.1-the-bridge#ops) Ops In Deno, aside from just registering functions like the promise reject callback and module resolve callback, there's another noteworthy function it performs. Deno takes the step of registering all the OPs (short for operations) with V8, the underlying JavaScript engine it uses. Copy for ctx in ops { let ctx_ptr = ctx as *const OpCtx as _; references.push(v8::ExternalReference { pointer: ctx_ptr }); references.push(v8::ExternalReference { function: ctx.decl.v8_fn_ptr, }); if let Some(fast_fn) = &ctx.decl.fast_fn { references.push(v8::ExternalReference { pointer: fast_fn.function as _, }); references.push(v8::ExternalReference { pointer: ctx.fast_fn_c_info.unwrap().as_ptr() as _, }); } } \-- Let's now explore some frequently utilized external references in the upcoming sections. [Previous4.0 Cover page](https://choubey.gitbook.io/internals-of-deno/bridge/chapter-cover-page) [Next4.2 Print](https://choubey.gitbook.io/internals-of-deno/bridge/4.2-print) Last updated 1 year ago --- # 3.3 Asynchronous green threads | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/threading-model/tokio-threads#overview) Overview ----------------------------------------------------------------------------------------------------- Deno uses threads in multiple ways: OS-aware static threads, OS-handled static threads, and asynchronous green threads for executing asynchronous operations. Understanding the differences between static threads and green threads is crucial. Static threads are like regular threads and are managed by the operating system. The OS controls their entire lifecycle, from creation to termination, and schedules them for execution. This means that the operating system is responsible for initiating and terminating static threads, and it also handles their scheduling. Asynchronous green threads, on the other hand, are managed by the tokio runtime as tokio tasks. The tokio runtime schedules these threads, and the operating system is not aware of their existence. This makes tokio tasks very efficient, as they are not subject to the overhead of the operating system's scheduling mechanisms. The combination of Deno's threading mechanisms, including static and green threads, allows it to manage asynchronous tasks efficiently and deliver optimal performance. [](https://choubey.gitbook.io/internals-of-deno/threading-model/tokio-threads#tokio-tasks) Tokio tasks ----------------------------------------------------------------------------------------------------------- ### [](https://choubey.gitbook.io/internals-of-deno/threading-model/tokio-threads#task-or-green-thread) Task or green thread A task, also known as a green thread, represents a small and efficient unit of work that does not block the entire process. It is similar to a tiny worker that carries out a task without stopping everything else. Imagine it as a mini version of a computer task. In the world of programming, tasks are similar to OS threads, which are separate paths of execution. However, they are managed differently. While OS threads are scheduled by the operating system's scheduler, tasks are managed by the Tokio runtime. The Tokio runtime acts as a supervisor, ensuring each task gets its turn without waiting. This means that tasks are executed efficiently and without blocking the entire process. The term "green threads" is a nickname for this setup. It is important to understand that green threads are not actual threads but rather a way to manage tasks efficiently. Now, let's explore what makes Tokio tasks special: ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MPprF_X8fMVIS4HPgcK%252F-MPprXmEyuP7J_fCu0tO%252Fdeno%2520tokio%2520tasks%2520props.png%3Falt%3Dmedia%26token%3D1f5307ec-c850-4416-803d-66a78f94760b&width=768&dpr=4&quality=100&sign=6a3766c7&sv=2) #### [](https://choubey.gitbook.io/internals-of-deno/threading-model/tokio-threads#lightweight) Lightweight Tokio Tasks have the characteristic of being lightweight. The Tokio runtime is responsible for scheduling tasks, rather than relying on the operating system. This approach offers a significant advantage: creating new tasks or switching between existing ones does not require a context switch, resulting in minimal additional processing overhead. The cost of creating, executing, and completing multiple tasks is remarkably low, especially when compared to traditional OS threads. As a result, Tokio Tasks demonstrate exceptional efficiency in their performance. #### [](https://choubey.gitbook.io/internals-of-deno/threading-model/tokio-threads#fast-scheduling) Fast Scheduling Tokio Tasks use a cooperative scheduling approach, unlike many operating systems that use preemptive multitasking. In preemptive multitasking, threads are interrupted after a set time. In contrast, Tokio Tasks use cooperative multitasking, where a task runs until it voluntarily gives up control. This allows the Tokio runtime's scheduler to smoothly switch to the next task. In Tokio, tasks are not forced to pause until they choose to yield control. This difference in scheduling methods is crucial for Tokio's efficient and predictable concurrency management. By prioritizing cooperation over interruption, Tokio Tasks enhance the stability and responsiveness of asynchronous applications. #### [](https://choubey.gitbook.io/internals-of-deno/threading-model/tokio-threads#non-blocking) Non-blocking Tokio Tasks operate in a non-blocking manner. Normally, when an operating system thread engages in I/O activities or needs to coordinate with another thread, it becomes obstructed, allowing the operating system to arrange for the execution of a different thread. In situations where a task cannot proceed with its execution, it must instead relinquish control, enabling the Tokio runtime to arrange for the execution of a different task. It's generally advisable for tasks to refrain from carrying out system calls or other activities that could lead to the blocking of a thread. Doing so would impede the execution of other tasks that share the same thread. Keep in mind that these tasks don't correspond to actual threads; they're akin to virtual threads that operate within the scope of the main thread. Consequently, if a virtual thread encounters a blockage, it effectively halts the entire process. Rather than facing such a scenario, tasks should employ APIs designed to facilitate the execution of blocking operations within an asynchronous context. Tokio provides a range of equivalent asynchronous APIs that can be utilized for this purpose. This approach ensures the efficient execution of tasks and prevents bottlenecks that could arise from thread blocking. ### [](https://choubey.gitbook.io/internals-of-deno/threading-model/tokio-threads#spawning-tasks) Spawning tasks Creating a tokio task or a green thread requires a process called spawning. This involves initiating the task or thread so that it can run concurrently with other tasks. There exist three distinct methods for spawning a tokio task, each serving its purpose in managing asynchronous operations. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MPpsGDfqW1MVtPhH7Qj%252F-MPptrliZEZ7memchMzc%252Fdeno%2520tokio%2520task%2520spawn%2520ways.png%3Falt%3Dmedia%26token%3De8672a55-6479-42f1-8973-ea6dc5174503&width=768&dpr=4&quality=100&sign=d0df7e58&sv=2) #### [](https://choubey.gitbook.io/internals-of-deno/threading-model/tokio-threads#task-spawn) **task::spawn** The task::spawn function serves as an asynchronous counterpart to the thread::spawn function found in the standard library. Its purpose is to handle async blocks or other futures, initiating the creation of a fresh task to execute those tasks simultaneously. By spawning a task, you enable it to operate concurrently alongside other tasks. This newly spawned task might execute on the ongoing thread or be directed to a separate thread for execution. The precise execution scenario hinges on the current configuration of the runtime environment. It's important to note that there exists no assurance regarding the completion of a spawned task. In the event of a runtime shutdown, all ongoing tasks are abruptly terminated, regardless of their individual lifecycles. Copy use tokio::task; task::spawn(async { // do async work }); The Tokio `spawn()` function is utilized to execute a set of asynchronous instructions enclosed within an async block. This function provides back a "join handle," which essentially allows us to await the completion of the task that was spawned. To better understand this concept, let's check a concrete example provided by the Tokio documentation: Copy use tokio::task; #[tokio::main] async fn main() { let join_handle = tokio::spawn(async { // Asynchronous operations to be performed here println!("Task has been spawned!"); }); // Waiting for the spawned task to complete join_handle.await.unwrap(); } In this example, the `tokio::spawn()` function is used to create a new asynchronous task. Inside the provided async block, various asynchronous operations can be executed. As soon as the task is spawned, the code continues running without waiting for the spawned task to finish. To ensure that the main program waits for the spawned task to complete, the `join_handle` is used with the `await` keyword. This implies that the program will pause at this point and wait until the spawned task is done executing before continuing further. In summary, the Tokio `spawn()` function facilitates the concurrent execution of asynchronous tasks, allowing us to efficiently manage and synchronize multiple asynchronous operations within our Rust programs. #### [](https://choubey.gitbook.io/internals-of-deno/threading-model/tokio-threads#task-spawn_blocking) **task::spawn\_blocking** This presents a unique method for creating Tokio tasks that involve blocking the ongoing process. When a task within Tokio involves a process that blocks its execution, it can disrupt the entire flow, especially if that task shares the same thread with other tasks. In such cases, the entire thread can become stalled, preventing any other tasks from making progress. To address this issue, there's a function known as `task::spawn_blocking`. This function works similarly to the `task::spawn` function we discussed earlier, but with a key difference. Instead of initiating a non-blocking future within the Tokio runtime, `task::spawn_blocking` initiates a blocking function on a distinct thread pool dedicated to handling such blocking tasks. It's crucial to understand that using a blocking call or engaging in extensive computations without yielding control is generally discouraged. Such practices can impede the executor's ability to drive other futures forward. However, with the approach provided by `task::spawn_blocking`, these concerns are mitigated. When you apply a closure using this method, it's executed on a separate thread pool designated for managing these kinds of blocking tasks. This arrangement ensures that the main futures executor remains unaffected and continues to operate smoothly. As a result, the execution of your program remains efficient and responsive. Copy use tokio::task; task::spawn_blocking(|| { // do sync work }); Blocking tasks are executed within a specialized thread pool designed to handle tasks that require synchronous operations or involve extensive computations. These tasks play a significant role in performing work that can't proceed concurrently, such as tasks that depend on previous steps or tasks that demand significant processing power. Here is an example from tokio documentation: Copy use tokio::task; let res = task::spawn_blocking(move || { // do some compute-heavy work or call synchronous code "done computing" }).await?; assert_eq!(res, "done computing"); #### [](https://choubey.gitbook.io/internals-of-deno/threading-model/tokio-threads#task-block_in_place) **task::block\_in\_place** The block\_in\_place function in Tokio plays a unique role in converting the current worker thread into a blocking thread. This conversion efficiently relocates other tasks executing on the same thread to a different worker thread, enhancing performance by reducing context switches. Unlike other functions, block\_in\_place has the distinctive ability to transform the current thread into a specialized blocking mode. Tokio's scheduler manages the migration of remaining tasks away from this green thread, optimizing system operation. Copy use tokio::task; task::block_in_place(|| { // do sync work }); [](https://choubey.gitbook.io/internals-of-deno/threading-model/tokio-threads#denos-usage-of-tasks) Deno's usage of tasks ------------------------------------------------------------------------------------------------------------------------------ The main purpose of Deno is to execute code in an asynchronous manner. Deno accomplishes this goal through two key methods: * _Tokio's Asynchronous Functions_: Deno includes a set of asynchronous functions from the Tokio library. These functions mirror the behavior of traditional synchronous functions that would normally block the program's flow. * _Utilizing Tokio Tasks_: In situations where Tokio doesn't offer predefined asynchronous functions, Deno uses tasks. These tasks enable Deno to execute functions asynchronously, even if they're not directly provided by the Tokio library. This asynchronous execution is crucial because it allows Deno to efficiently manage multiple tasks simultaneously, enhancing the overall performance and responsiveness of applications. By leveraging Tokio's asynchronous capabilities and using tasks when necessary, Deno provides a versatile environment for developing applications that can handle complex and concurrent tasks without sacrificing efficiency. Deno operates in a distinct manner compared to task::spawn, which generates a fresh task. This choice is possibly motivated by the pursuit of enhanced performance. Whenever a basic spawn is executed, a novel green thread is brought forth, which might result in a small extra load when carrying out uncomplicated asynchronous code. In lieu of the conventional spawn approach, Deno uses a strategy called spawn\_blocking to manage asynchronous operations. However, this method is predominantly used for tasks associated with the file system. For alternative asynchronous activities, Deno directly capitalizes on the available asynchronous APIs, which are attainable either through tokio or other external libraries. To reinforce our understanding, let's get into an illustrative example of each of these methodologies as employed within the Deno environment. ### [](https://choubey.gitbook.io/internals-of-deno/threading-model/tokio-threads#async-op-without-a-task) Async op without a task To start, let's look at an example of an asynchronous operation that doesn't need to create a task explicitly. Deno handles async functions seamlessly, making task creation unnecessary. For instance, handling an asynchronous TCP connection typically involves a blocking procedure. However, Deno uses a low-level operation called "op\_accept" to receive TCP connections asynchronously. This operation efficiently handles incoming TCP connections. Deno's approach to accepting TCP connections uses the "op\_accept" operation. Normally, this would require complex handling to avoid blocking, but Deno's design avoids these complexities. Other frameworks like Tokio already support asynchronous TCP and UDP, making it easy for Deno to use the existing infrastructure without needing special interventions. This approach makes Deno's asynchronous programming more efficient and scalable. Deno's innovative design and seamless integration of async functions make it a popular choice for developers. Copy async fn op_accept( state: Rc>, rid: ResourceId, ) -> Result { let listener = state.borrow().resource_table.get::(rid)?; let stream = listener.accept().await?; let rid = state.borrow_mut().resource_table.add(stream); Ok(rid) } // TcpListener struct TcpListener { inner: tokio::net::TcpListener, } As previously mentioned, Deno makes direct use of asynchronous functions provided by Tokio. ### [](https://choubey.gitbook.io/internals-of-deno/threading-model/tokio-threads#async-op-with-blocking-task) Async op with blocking task Next, let's look at an example where Deno converts synchronous operations to asynchronous tasks. A good example is file system operations. In Deno, all file system operations are asynchronous and start with a blocking call. Here are some examples of asynchronous file system operations: * Opening files * Seeking within files * Getting file status information (Fstat) * Creating directories (Mkdir) * Changing file permissions (Chmod) * Changing file ownership (Chown) * Copying files * Reading directories This list shows the range of asynchronous file system operations. It's important to note that only asynchronous file system operations are executed as blocking tasks. This means they are likely to be handled by Tokio's dedicated thread pool, making them more efficient. To illustrate further, let's compare the synchronous and asynchronous implementation of a basic operation like 'opening a file': Copy fn open_sync( &self, path: &Path, options: OpenOptions, ) -> FsResult> { let opts = open_options(options); let std_file = opts.open(path)?; Ok(Rc::new(StdFileResourceInner::file(std_file))) } async fn open_async( &self, path: PathBuf, options: OpenOptions, ) -> FsResult> { let opts = open_options(options); let std_file = spawn_blocking(move || opts.open(path)).await??; Ok(Rc::new(StdFileResourceInner::file(std_file))) } These two functions have many similarities. They both call the same function, opts.open(path). However, there is a significant difference between them. In the synchronous version, the call to set the file time blocks the current thread until opts.open(path) finishes its task. In contrast, the asynchronous version starts a blocking task that handles the call to set the file time. This approach avoids blocking the thread directly, as the task runs in a dedicated thread pool managed by Tokio. After starting the task with spawn\_blocking, the code uses the await instruction to wait for the task to finish before continuing with the next steps. Copy // Sync let std_file = opts.open(path)?; // Async let std_file = spawn_blocking(move || opts.open(path)).await??; \-- This section covered the asynchronous abilities provided by Tokio using its built-in APIs or tasks. This involves understanding how Tokio helps handle tasks that can run independently without blocking other processes. [Previous3.2 Default threading model](https://choubey.gitbook.io/internals-of-deno/threading-model/default-threads) [Next3.4 What's next](https://choubey.gitbook.io/internals-of-deno/threading-model/3.4-whats-next) Last updated 1 year ago --- # 5.2 Basic hello world | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/foundations/basic-hello-world#typescript-code) Typescript code ------------------------------------------------------------------------------------------------------------------- Let's begin by exploring the most straightforward hello world program, written using Typescript. This program could also be referred to as a console print program, as it outputs information to the console. Throughout this discussion, we'll use the terms "hello world" and "console print" interchangeably. There are two key motivations for using Typescript over Javascript within the context of Deno: 1. Deno inherently supports Typescript, making it the preferred choice for developing user-level programs. While Deno does provide support for Javascript, the core essence of using Deno could be undermined if Typescript isn't used. 2. Creating a program in Typescript allows us to observe the process of converting it into Javascript. It's important to recall that the V8 engine, which powers Deno, can only execute Javascript code. Now, let's look at the simplest console print program written in TypeScript. This program is slightly different from a typical console log program. We made this variation to show how TypeScript works and how it is converted to JavaScript. By examining this program, we can gain valuable insight into how Deno works with different programming languages. The file name is `helloLog.ts`. The location of the source file is: _/Users/mayankc/Work/source/denoExamples/helloLog.ts_ The purpose of indicating the path of the source file is to demonstrate how it is retrieved during the compilation of the program. This provides insight into the process of fetching the necessary files. Let's take a look at the content present within the "helloLog.ts" source file: Copy // File helloLog.ts function printNumber(input: number) { console.log(input); } function printString(input: string) { console.log(input); } printNumber(1); printString("One"); The program is quite straightforward. It consists of just two functions: one for displaying numbers and the other for showing strings. It might not seem like much, but these functions are enough to understand the very basics of Deno. [](https://choubey.gitbook.io/internals-of-deno/foundations/basic-hello-world#javascript-code) Javascript code ------------------------------------------------------------------------------------------------------------------- As we discussed earlier, Deno comes with built-in support for Typescript. However, it's important to note that the V8 engine, which powers Deno, is designed to execute Javascript code exclusively. This means that before code can run in V8, any Typescript code must be transformed into Javascript. Deno takes on the task of converting Typescript to Javascript prior to loading and running it within the V8 engine. To achieve this conversion, Deno relies on a Rust-based tool called the SWC compiler. This compiler is responsible for translating Typescript code into its Javascript counterpart, ensuring that it becomes compatible with the V8 engine's execution environment. This translation process is a pivotal step in the Deno ecosystem, enabling developers to seamlessly work with Typescript while still benefiting from the performance and capabilities of the V8 engine. Additionally, when there's a need to perform type-checking during the startup phase, Deno leverages Microsoft's TSC compiler. This step is crucial in ensuring that the Typescript code adheres to the specified type rules and structures, helping catch potential errors early in the development process. By incorporating TSC into its workflow, Deno provides developers with a comprehensive solution for managing Typescript code from conversion to type validation. The following is the output of the converted input file: Copy // File helloLog.ts function printNumber(input) { console.log(input); } function printString(input) { console.log(input); } printNumber(1); printString("One"); //# sourceMappingURL=data:application/json;base64,eyJ2ZXJzaW9uIjozLCJzb3VyY2VzIjpbImZpbGU6Ly8vVXNlcnMvbWF5YW5rYy9Xb3JrL3NvdXJjZS9kZW5vRXhhbXBsZXMvaGVsbG9Mb2cudHMiXSwic291cmNlc0NvbnRlbnQiOlsiLy8gRmlsZSBoZWxsb0xvZy50c1xuXG5mdW5jdGlvbiBwcmludE51bWJlcihpbnB1dDogbnVtYmVyKSB7XG4gIGNvbnNvbGUubG9nKGlucHV0KTtcbn1cblxuZnVuY3Rpb24gcHJpbnRTdHJpbmcoaW5wdXQ6IHN0cmluZykge1xuICBjb25zb2xlLmxvZyhpbnB1dCk7XG59XG5cbnByaW50TnVtYmVyKDEpO1xucHJpbnRTdHJpbmcoXCJPbmVcIik7XG4iXSwibmFtZXMiOltdLCJtYXBwaW5ncyI6IkFBQUEsbUJBQW1CO0FBRW5CLFNBQVMsWUFBWSxLQUFhO0lBQ2hDLFFBQVEsSUFBSTtBQUNkO0FBRUEsU0FBUyxZQUFZLEtBQWE7SUFDaEMsUUFBUSxJQUFJO0FBQ2Q7QUFFQSxZQUFZO0FBQ1osWUFBWSJ9 Converting the code was a simple process. During compilation, the argument types were checked and aligned, and then the specific type details were removed to create clean JavaScript code. Note that there is a comment at the end about "sourceMappingURL." This comment is for Deno's internal use and helps it function. This program is simple because it doesn't have imports or other operations. We will discuss imports and operations in more detail in the next chapter. Although this program is basic, it helps us understand the fundamental concepts. Let's run it now: Copy > deno run helloLog.ts 1 One Now, let's start our journey by examining the main program of Deno. This will give us a solid foundation to build on as we move forward. [Previous5.1 Hello world program](https://choubey.gitbook.io/internals-of-deno/foundations/hello-world) [Next5.3 Main program of Deno](https://choubey.gitbook.io/internals-of-deno/foundations/main-program) Last updated 1 year ago --- # 4.3 Encode and decode | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/bridge/4.4-encode-and-decode#overview) Overview ---------------------------------------------------------------------------------------------------- Encode and Decode represent a fundamental pair of functions within Deno's framework, serving as integral OPs (operations) and APIs (application programming interfaces). These functions hold substantial significance, facilitating the seamless transformation of V8 strings into V8 buffers and reciprocally. Notably, it's essential to grasp that a V8 string deviates slightly from a JS (JavaScript) string due to its construction as a C++ data structure. Although most external references pertain to infrastructural tasks, certain exceptions such as 'print' and 'get promise details' stand out. Evident from their nomenclature, the 'encode' function assumes the role of translating a string into a sequence of bytes, effectively encapsulating the information within a binary format. Conversely, the 'decode' function specializes in the inverse process, adeptly converting bytes back into a human-readable string format. Both of these functions interact with V8 data structures, further emphasizing their underlying mechanics. It's intriguing to highlight that the implementation of the encode and decode functions is rooted in Rust. Despite this Rust foundation, these functions seamlessly interact with V8 data structures, harmonizing the low-level Rust components with the intricacies of V8's C++ data structure. This harmonious interplay is pivotal in ensuring the efficient and accurate translation between V8 strings and buffers. Both encode and decode are implemented in Deno core. [](https://choubey.gitbook.io/internals-of-deno/bridge/4.4-encode-and-decode#functions) Functions ------------------------------------------------------------------------------------------------------ ### [](https://choubey.gitbook.io/internals-of-deno/bridge/4.4-encode-and-decode#registration) Registration The process of registering the foundational OP functions takes place during startup, similar to how all other OPs are registered. Copy pub fn op_encode<'a>( scope: &mut v8::HandleScope<'a>, text: v8::Local<'a, v8::Value>, ) -> Result, Error> pub fn op_decode<'a>( scope: &mut v8::HandleScope<'a>, #[buffer] zero_copy: &[u8], ) -> Result, Error> ### [](https://choubey.gitbook.io/internals-of-deno/bridge/4.4-encode-and-decode#js-space) JS Space The encode function is invoked from multiple locations within the Deno codebase. This utilization extends to the JavaScript context as well. Copy // fetch request body encoding reqBody = typeof reqBody === "string" ? core.encode(reqBody) : reqBody; // form data boundary encoding this.boundaryChars = core.encode(this.boundary); // Text encoder encode(input = "") { webidl.assertBranded(this, TextEncoderPrototype); // The WebIDL type of `input` is `USVString`, but `core.encode` already // converts lone surrogates to the replacement character. input = webidl.converters.DOMString( input, "Failed to execute 'encode' on 'TextEncoder'", "Argument 1", ); return core.encode(input); } Likewise, the core.decode functions are invoked from various points within the codebase. A few illustrative instances include: Copy // prompt return core.decode(new Uint8Array(buf)); // fetch response return core.decode(buffer); // reading file as text return core.decode(buffer); ### [](https://choubey.gitbook.io/internals-of-deno/bridge/4.4-encode-and-decode#rust-space) Rust space #### [](https://choubey.gitbook.io/internals-of-deno/bridge/4.4-encode-and-decode#encode) Encode Let's take a look at the code snippet from the OP that presents the 'encode' function. Copy pub fn op_encode<'a>( scope: &mut v8::HandleScope<'a>, text: v8::Local<'a, v8::Value>, ) -> Result, Error> { let text = v8::Local::::try_from(text) .map_err(|_| type_error("Invalid argument"))?; let text_str = serde_v8::to_utf8(text, scope); let bytes = text_str.into_bytes(); let len = bytes.len(); let backing_store = v8::ArrayBuffer::new_backing_store_from_vec(bytes).make_shared(); let buffer = v8::ArrayBuffer::with_backing_store(scope, &backing_store); let u8array = v8::Uint8Array::new(scope, buffer, 0, len).unwrap(); Ok(u8array) } The function is big, but it carries out a simple task: * Convert string to bytes * Set the return value (rv) so that v8 can proceed after the encode returns Here is an example of an object and the encoded bytes: Copy // OBJECT TO ENCODE { path: "/var/tmp/test.log", len: 0, promiseId: 2 } // ENCODED BYTES [\ 123, 34, 112, 97, 116, 104, 34, 58, 34, 47,\ 118, 97, 114, 47, 116, 109, 112, 47, 116, 101,\ 115, 116, 46, 108, 111, 103, 34, 44, 34, 108,\ 101, 110, 34, 58, 48, 44, 34, 112, 114, 111,\ 109, 105, 115, 101, 73, 100, 34, 58, 50, 125\ ] #### [](https://choubey.gitbook.io/internals-of-deno/bridge/4.4-encode-and-decode#decode) Decode The code for the decode function is also equally simple: Copy pub fn op_decode<'a>( scope: &mut v8::HandleScope<'a>, #[buffer] zero_copy: &[u8], ) -> Result, Error> { let buf = &zero_copy; match v8::String::new_from_utf8(scope, buf, v8::NewStringType::Normal) { Some(text) => Ok(text), None => Err(range_error("string too long")), } } * Get buffer * Convert buffer to string * Set return value so that v8 can proceed after the decode returns Here is an example of bytes and the decode object: Copy // BYTES [\ 123, 34, 111, 107, 34, 58,\ 123, 125, 44, 34, 112, 114,\ 111, 109, 105, 115, 101, 73,\ 100, 34, 58, 50, 125\ ] // DECODED OBJECT { ok: {}, promiseId: 2 } [Previous4.2 Print](https://choubey.gitbook.io/internals-of-deno/bridge/4.2-print) [Next4.4 What's next](https://choubey.gitbook.io/internals-of-deno/bridge/4.6-whats-next) Last updated 1 year ago --- # 5.0 Cover page | The Internals of Deno ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252FoaOVELEQy2fgFoJO6LXE%252F6.png%3Falt%3Dmedia%26token%3Dd9d089f7-4ae1-4096-9ebb-77819a7bf1b0&width=768&dpr=4&quality=100&sign=422944e4&sv=2) [Previous4.4 What's next](https://choubey.gitbook.io/internals-of-deno/bridge/4.6-whats-next) [Next5.1 Hello world program](https://choubey.gitbook.io/internals-of-deno/foundations/hello-world) Last updated 1 year ago --- # 2.8 Tokio | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/architecture/tokio#overview) Overview ------------------------------------------------------------------------------------------ Tokio, a vital third-party library, plays a central role in Deno's framework. Deno relies heavily on asynchronous programming to minimize callbacks. To achieve this, Deno utilizes Tokio, which converts synchronous functions into asynchronous ones. Tokio is a runtime environment for building reliable, efficient, and asynchronous applications in Rust. It provides an event-driven, non-blocking platform for creating asynchronous applications in Rust. Deno benefits from Tokio's non-blocking nature, aligning with its internal architecture. As an asynchronous runtime, Tokio provides developers with the essential tools to build network-oriented applications. Its design enables developers to target a wide range of systems, from large servers with multiple cores to compact embedded devices. The functionality provided by Tokio is extensive and versatile. Here are some of the notable features it offers: * **fs:** This module encompasses utility methods and adapter types tailored for input/output operations involving files and standard streams (such as Stdin, Stdout, and Stderr). It also facilitates filesystem manipulation. * **io:** Serving as the asynchronous counterpart to std::io, this module handles input/output tasks asynchronously. * **net:** Within this module, you'll find TCP, UDP, and Unix networking types. These types, similar to those present in the standard library, enable the implementation of diverse networking protocols. * **process:** This module presents asynchronous versions of functions responsible for creating processes. * **runtime:** The core of Tokio involves an I/O event loop, aptly referred to as the driver. This component manages I/O resources and distributes corresponding I/O events to tasks reliant on them. Furthermore, Tokio includes a scheduler designed to execute tasks that harness these I/O resources, along with a timer mechanism for scheduling tasks to commence after specific time intervals. * **task:** Encompassing asynchronous green-threads, this module aids in managing concurrent tasks. * **and many more:** Tokio's offerings extend beyond the aforementioned modules, encompassing a diverse array of functionalities. Tokio's core functionality is built around asynchronicity, a unifying theme that runs throughout its features. It provides asynchronous versions of traditional synchronous functions, ensuring non-blocking operations. For instance, Tokio wraps OS system calls in asynchronous functions, adhering to its asynchronous programming approach. This integration harmonizes Deno's async-oriented design with Tokio's non-blocking capabilities, leading to a synergistic relationship that enhances overall performance. Now, let's explore several benefits that arise from utilizing Tokio within the context of Deno programming: ### [](https://choubey.gitbook.io/internals-of-deno/architecture/tokio#reliable) Reliable Tokio's API prioritizes safety across three key areas: memory, threads, and preventing misuse. This multi-faceted approach helps prevent common programming errors, including unbounded queues, buffer overflows, and task starvation. Tokio's API ensures memory safety by carefully managing memory usage, reducing the likelihood of memory-related bugs and crashes. It also enables thread safety, allowing concurrent program execution without conflicts or race conditions. This means that multiple tasks can run simultaneously without interfering with each other's data, reducing unpredictable behavior and debugging issues. Additionally, Tokio's API is designed to prevent misuse by providing guidelines and structures that guide developers away from potentially harmful practices. By combining these safety measures, Tokio enables the creation of efficient, responsive, robust, and reliable applications, acting as a safety net to catch potential problems early in development and contributing to a more stable software ecosystem. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/tokio#fast) Fast Built with Rust, Tokio features a advanced multi-threaded scheduler that utilizes a work-stealing technique. This technology enables applications to handle a large volume of requests efficiently, processing up to hundreds of thousands per second. Notably, this performance is achieved with minimal additional processing overhead. Tokio's architecture empowers developers to create high-performance applications that scale efficiently and provide responsive user experiences. As a result, Tokio is a crucial tool for developers to build powerful and efficient applications that manage significant workloads effectively. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/tokio#easy) Easy Using async/await makes building asynchronous applications much easier. With Tokio's ecosystem, which offers a dynamic environment and helpful tools, developing applications becomes very straightforward and effortless. Tokio provides various utilities and resources that work well with async/await, helping with tasks like asynchronous I/O operations, timers, and concurrent execution. With Tokio's support, you can focus on your application's logic and features, rather than worrying about the complexities of asynchronous programming. This allows you to develop applications more efficiently and effectively, without getting bogged down in low-level details. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/tokio#flexible) Flexible Tokio recognizes that server applications and embedded devices have different requirements. While Tokio provides preset configurations that work well out of the box, it also offers tools to customize and optimize for specific use cases. Server applications require efficient handling of a high volume of incoming requests, necessitating high performance and responsiveness. In contrast, embedded devices have limited resources, such as memory and processing power, requiring careful resource management to ensure optimal operation. Tokio accommodates these diverse needs, enabling developers to tailor their applications accordingly. \-- Tokio is the foundation of Deno's asynchronous functionality, enabling Deno to run entirely asynchronously. While this book focuses on Deno, it's important to recognize Tokio's significance in the Deno ecosystem. Although we'll discuss Deno's features in this book, our coverage of Tokio will be brief to maintain our focus. If you want to learn more about Tokio's capabilities, you can find comprehensive documentation at [https://docs.rs/tokio/0.3.5/tokio/index.html](https://docs.rs/tokio/0.3.5/tokio/index.html) . [](https://choubey.gitbook.io/internals-of-deno/architecture/tokio#functionalities) Functionalities -------------------------------------------------------------------------------------------------------- ### [](https://choubey.gitbook.io/internals-of-deno/architecture/tokio#fs) FS This Tokio module provides essential tools and flexible structures for handling input/output operations with files and standard streams like Stdin, Stdout, and Stderr. It also helps manage the filesystem. These tools are designed for use within the Tokio runtime environment. The async fs module offers a collection of asynchronous functions that efficiently handle various filesystem tasks. These functions are designed for asynchronous programming, ensuring non-blocking execution and preventing thread halting. The following asynchronous functions are available: 1. **Copy File:** This function facilitates the copying of a file from one location to another. 2. **Create Directory:** This function allows you to create a new directory, aiding in organizing your filesystem. 3. **Read Directory:** With this function, you can access the contents of a directory, enabling you to gather information about the files it contains. 4. **Remove Directory:** When you need to delete a directory and its contents, this function comes to your rescue. 5. **Read File:** Utilize this function to read the contents of a file asynchronously, avoiding any disruption to the program's flow. 6. **Write File:** If you wish to write data to a file, this function ensures that the process occurs smoothly within the async environment. 7. **Remove File:** This function handles the deletion of a specific file in an asynchronous manner. 8. **Rename File:** When the need arises to rename a file, this function provides a seamless solution. 9. **Set Permissions:** With this function, you can set the permissions for a file or directory according to your requirements. The functions mentioned above are asynchronous, meaning they are designed to operate without interrupting the program's execution. This allows other tasks to continue running concurrently without being blocked. To help illustrate their usage, here's a simple code example that demonstrates how to use these async filesystem functions within a Tokio runtime: Copy use tokio::fs; #[tokio::main] async fn main() -> Result<(), Box> { // Example usage of async filesystem functions let content = fs::read_to_string("input.txt").await?; println!("Content of the file: {}", content); fs::write("output.txt", "Hello, Tokio!").await?; Ok(()) } Here is another one: Copy use tokio::fs; use std::io; #[tokio::main] async fn main() -> io::Result<()> { fs::create_dir("/some/dir").await?; Ok(()) } The code snippet reads the "/some/dir" directory asynchronously, without blocking the main thread. The use of 'await' at the end of 'create\_dir' indicates that it's an asynchronous operation. By studying and experimenting with code like this, you can gain hands-on understanding of how async filesystem functions work within the Tokio ecosystem, deepening your understanding of their capabilities and benefits. Here's another example of an async fs operation: Copy use tokio::fs; fs::write("foo.txt", b"Hello world!").await?; The code writes "Hello world" to a file named foo.txt asynchronously. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/tokio#io) IO This Tokio module serves as an asynchronous version of std::io, introducing two key traits: AsyncRead and AsyncWrite. These traits are the asynchronous equivalents of the Read and Write traits in the standard library. The async fs module provides various asynchronous functions, including: * Buffer copying * Standard error (stderr) management * Standard output (stdout) handling * Standard input (stdin) handling * And others Let's explore an example of writing content to standard output (stdout), demonstrating the practical application of these concepts: Copy use tokio::io::{self, AsyncWriteExt}; #[tokio::main] async fn main() -> io::Result<()> { let mut stdout = io::stdout(); stdout.write_all(b"Hello world!").await?; Ok(()) } The write\_all function enables asynchronous writing of "Hello World" to the standard output (stdout), without blocking the main thread. Next, let's examine an example that demonstrates buffer usage: Copy use tokio::io; let mut reader: &[u8] = b"hello"; let mut writer: Vec = vec![]; io::copy_buf(&mut reader, &mut writer).await?; assert_eq!(b"hello", &writer[..]); The copy\_buf interface allows for asynchronous copying from a reader to a writer. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/tokio#net) Net This Tokio module includes essential networking components, such as TCP, UDP, and Unix networking types. These components are crucial for developing various networking protocols, including those used in internet communication, local network communication, and more. The 'net' module contains three sub-modules, each focused on a specific networking aspect: * **TCP**: Ensures reliable, ordered, and error-checked data delivery between devices, making it a fundamental building block of internet communication. * **UDP**: Offers faster but less reliable communication, suitable for tasks where speed is crucial and occasional data loss is acceptable. * **Unix Domain Sockets**: Enables local inter-process communication (IPC) on the same machine, providing a means for processes to communicate locally. These sub-modules provide developers with tools to create and manage connections, establish communication channels, and design applications that prioritize speed, efficiency, and reliability. An example of an asynchronous TCP client demonstrates how to use Tokio's tools to establish a TCP connection efficiently, enabling non-blocking communication between the client and server and enhancing overall responsiveness and scalability in networking interactions. Copy use tokio::net::TcpSocket; use std::io; #[tokio::main] async fn main() -> io::Result<()> { let addr = "127.0.0.1:8080".parse().unwrap(); let socket = TcpSocket::new_v4()?; let stream = socket.connect(addr).await?; Ok(()) } The connect system call operates in a way that blocks other processes until its task is complete, hindering the progress of other tasks. In contrast, the connect() function provided by Tokio follows a non-blocking approach, enabling efficient and concurrent processing. When you use connect(), it swiftly concludes and provides a result as soon as a connection is established, without delaying your main program thread. This ensures that your program continues executing other tasks without interruption, improving overall performance and responsiveness. To illustrate this concept, consider the following example of a UDP server, which demonstrates the benefits of Tokio's non-blocking connect() function: Copy use tokio::net::UdpSocket; use std::io; #[tokio::main] async fn main() -> io::Result<()> { let sock = UdpSocket::bind("0.0.0.0:8080").await?; let mut buf = [0; 1024]; loop { let (len, addr) = sock.recv_from(&mut buf).await?; println!("{:?} bytes received from {:?}", len, addr); let len = sock.send_to(&buf[..len], addr).await?; println!("{:?} bytes sent", len); } } In this context, three essential system calls are utilized: bind, recv\_from, and send\_to. It's important to recognize that all three of these system calls function in a non-blocking mode, enabling efficient and concurrent processing. This means that these system calls will not obstruct other processes, allowing for seamless execution and improved performance. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/tokio#process) Process The Tokio module introduces a Command structure, inspired by the std::process::Command type in the standard library, but with enhanced asynchronous capabilities for process creation. The module offers asynchronous variations of spawn, status, output, and related functions, which yield types compatible with futures and seamlessly integrate with Tokio's features. This module provides a range of asynchronous features, including: * **Child Process Creation**: Asynchronous spawning of child processes for efficient multitasking. * **Communication with Child Processes**: Asynchronous interaction with child processes, managing inputs and outputs (stdin, stdout, stderr) with ease. The Command structure proves valuable for executing commands in various scenarios, such as: Copy let mut cmd = Command::new("example_command"); cmd.arg("arg1").arg("arg2"); // Asynchronously spawn the child process let child = cmd.spawn().await.expect("Failed to spawn the process"); // Communicate with the child process's stdin, stdout, and stderr asynchronously let child_stdin = child.stdin().expect("Failed to open stdin"); let child_stdout = child.stdout().expect("Failed to open stdout"); let child_stderr = child.stderr().expect("Failed to open stderr"); // ... Further interactions with the child process ... The Command module in Tokio provides a robust asynchronous foundation for efficiently managing child processes and their communication. This enables effective process management and seamless interaction with child processes. Consider the following example, which demonstrates the module's capabilities: Copy use tokio::process::Command; #[tokio::main] async fn main() -> Result<(), Box> { // The usage is similar as with the standard library's `Command` type let mut child = Command::new("echo") .arg("hello") .arg("world") .spawn() .expect("failed to spawn"); // Await until the command completes let status = child.wait().await?; println!("the command exited with: {}", status); Ok(()) } ### [](https://choubey.gitbook.io/internals-of-deno/architecture/tokio#runtime) Runtime The Tokio runtime is a vital component in the Deno ecosystem, providing essential services for asynchronous program execution. Unlike typical Rust programs, asynchronous applications require runtime support for efficient operation. The Tokio runtime offers the following vital services: * **I/O Event Loop (Driver)**: A dynamic loop managing I/O resources and dispatching events to dependent tasks for seamless execution. * **Scheduler (Task Executor)**: A dedicated scheduler efficiently orchestrating tasks relying on I/O resources, ensuring organized execution and optimal performance. * **Timer (Temporal Precision)**: A time-sensitive component enabling precise task scheduling at predefined intervals. Tokio's comprehensive Runtime integrates these services into a unified entity, allowing for initiation, conclusion, and customization as a cohesive whole. Notably, manual runtime creation is not always necessary, as the tokio::main attribute macro automatically generates a Runtime, simplifying the process. This convenience feature eliminates the need for manual configuration, making it easier to utilize the Tokio runtime. In Deno, the Tokio runtime is easily accessible via the tokio::main attribute, streamlining asynchronous operations. While understanding the Tokio runtime may seem complex initially, its core concept is to provide unwavering support for asynchronous tasks, serving as the backbone for efficient operation and outcome management. By leveraging the Tokio runtime, developers can create scalable and efficient asynchronous applications with ease. ### [](https://choubey.gitbook.io/internals-of-deno/architecture/tokio#task) Task Tokio tasks are asynchronous green-threads, representing a lightweight and non-obstructive unit of execution. Similar to operating system threads, tasks execute independently, but unlike OS threads, they are managed by the Tokio runtime instead of the OS scheduler. This concept is commonly known as green threads, a term that highlights their resource-efficient and eco-friendly nature. For a more in-depth exploration of green threads, you can visit this informative resource: [https://en.wikipedia.org/wiki/Green\_threads](https://en.wikipedia.org/wiki/Green_threads) . The realm of tasks offers a range of essential functionalities, including: 1. **Task Spawning:** Initiating a new task's lifecycle. 2. **Non-Blocking Tasks:** Tasks that execute without causing obstructions. 3. **Blocking Mode Task Spawning:** Creating tasks in a mode that allows for potential blocks. 4. **Selective Blocking Mode:** Instances where utilizing a blocking mode is warranted. 5. **Block-in-Place:** Temporarily halting the ongoing operations of the present thread at a specific juncture. The underpinning of tasks rests upon their exceptional efficiency, attributable to several factors: 1. **Lightweight Nature:** Tasks are notably gentle on system resources. 2. **Cooperative Scheduling:** Task scheduling is orchestrated in a cooperative manner. 3. **Non-Blocking Essence:** The inherent design of tasks ensures they operate without inducing blocks. To help you understand these concepts better, let's consider some examples of task spawning in both blocking and non-blocking modes, as well as a demonstration of the block-in-place concept. By reviewing these examples, you will gain a better understanding of the flexibility and capabilities that Tokio tasks offer in Deno applications. Copy use tokio::task; // NON-BLOCKING let join = task::spawn(async { // ... "hello world!" }); // ... // Await the result of the spawned task. let result = join.await?; assert_eq!(result, "hello world!"); // BLOCKING let join = task::spawn_blocking(|| { // do some compute-heavy work or call synchronous code "blocking completed" }); let result = join.await?; assert_eq!(result, "blocking completed"); // BLOCK IN PLACE let result = task::block_in_place(|| { // do some compute-heavy work or call synchronous code "blocking completed" }); assert_eq!(result, "blocking completed"); [](https://choubey.gitbook.io/internals-of-deno/architecture/tokio#denos-usage) Deno's usage ------------------------------------------------------------------------------------------------- Deno utilizes Tokio in two main ways: 1. **Creating green threads for asynchronous operations**: Deno uses Tokio to create green threads, specialized threads that handle asynchronous operations efficiently. These threads enable concurrent task execution without the complexities of traditional multithreading, enhancing Deno's ability to manage multiple asynchronous tasks smoothly. 2. **Extending Tokio's async capabilities to user space**: Deno provides users with access to Tokio's asynchronous functions, allowing developers to leverage Tokio's asynchronous runtime capabilities, such as efficient event handling and I/O operations, in their applications. Deno also integrates with the Tokio runtime and other asynchronous modules. The following code snippet from Deno's codebase illustrates this integration: The highlighted `op_connect` function demonstrates how Deno utilizes Tokio's capabilities to manage asynchronous operations efficiently, particularly in establishing TCP connections. This integration enables Deno to provide a robust and performant environment for handling various network-related tasks. Copy pub async fn op_net_connect_tcp( state: Rc>, addr: IpAddr, ) -> Result<(ResourceId, IpAddr, IpAddr), AnyError> where NP: NetPermissions + 'static, { { let mut state_ = state.borrow_mut(); state_ .borrow_mut::() .check_net(&(&addr.hostname, Some(addr.port)), "Deno.connect()")?; } let addr = resolve_addr(&addr.hostname, addr.port) .await? .next() .ok_or_else(|| generic_error("No resolved address found"))?; let tcp_stream = TcpStream::connect(&addr).await?; let local_addr = tcp_stream.local_addr()?; let remote_addr = tcp_stream.peer_addr()?; let mut state_ = state.borrow_mut(); let rid = state_ .resource_table .add(TcpStreamResource::new(tcp_stream.into_split())); Ok((rid, IpAddr::from(local_addr), IpAddr::from(remote_addr))) } The `op_net_connect` function is an internal operation used within Deno's codebase. It enables the establishment of a TCP connection to a specified address. This functionality is crucial in scenarios involving WebSocket operations, which rely on the underlying TCP protocol for their operation. Copy let tcp_stream = TcpStream::connect(&addr).await?; \-- In summary, Tokio is a complex and multifaceted library with numerous features. Its efficiency is remarkable, enabling exceptional functionality. This is just a brief introduction to its capabilities. For further exploration, extensive resources are available at [https://tokio.rs/](https://tokio.rs/) and [https://docs.rs/tokio/latest/tokio/](https://docs.rs/tokio/latest/tokio/) (currently version 1.38.0). Next, we have the V8 engine, a crucial third-party library that plays a vital role in executing JavaScript code. Despite its complexity, its importance cannot be overstated. The V8 engine serves as the foundation for JavaScript execution, highlighting its essential role in this context. [Previous2.7 Rusty\_v8](https://choubey.gitbook.io/internals-of-deno/architecture/rusty_v8) [Next2.9 V8](https://choubey.gitbook.io/internals-of-deno/architecture/v8) Last updated 1 year ago --- # 5.5 CLI Factory | The Internals of Deno In addition to creating a module specifier, the "run" command in Deno also gives rise to what's known as CLIFactory, generated via a CLIFactory. These CLIFactory encompass various settings and objects used widely across the program. These valuable CLIFactory stem from the very options provided through the command line when Deno is invoked. This essentially means that the way you configure and set up Deno through its command line has a direct impact on the makeup of CLIFactory and subsequently influences the behavior of the program. The CLIFactory act as a central hub for crucial configurations and resources that Deno requires to function effectively. Copy let factory = CliFactory::from_flags(flags).await?; [](https://choubey.gitbook.io/internals-of-deno/foundations/program-state#attributes) Attributes ----------------------------------------------------------------------------------------------------- The CLIFactory is a crucial component in the Deno ecosystem. It comes into play after generating the module specifier for the main module. The CLIFactory has several important elements that contribute to Deno's functionality in different ways. Let's examine these essential aspects: * **Deno Directory**: This particular facet pertains to the working directory of Deno. Within this directory, all the downloaded modules find their home and undergo the compilation process. It serves as a central repository for Deno's operations. * **Caches**: Deno manages distinct types of caches, and the CLIFactory helps keep track of their paths. These caches encompass various categories such as Dependency, Gen (generation), and HTTP cache. The Dependency cache stores vital dependencies, the Gen cache handles generated code, and the HTTP cache maintains data fetched from the web. * **File Fetcher**: A versatile tool, the file fetcher, proves invaluable for procuring files, whether they are located locally or reside on remote servers. This function underpins Deno's capability to seamlessly access and work with diverse sources. * **HTTP Client**: An integral part of CLIFactory, the HTTP Client empowers Deno to interact with web resources. This feature facilitates tasks such as making HTTP requests and handling responses from various online services. * **Certificate Store**: Security is of paramount importance, and the CLIFactory include a certificate store. This repository safeguards the certificates necessary for securing connections, assuring the authenticity and integrity of data exchanges. * **Module Resolver**: The module resolver within CLIFactory tackles the intricate task of determining module dependencies. It navigates the network of interconnected modules, ensuring that the correct modules are fetched and utilized. * **Module Graph Builder**: This component aids in constructing the module graph, which outlines the relationships and dependencies among different modules. The module graph builder ensures the organized and systematic interaction between various components. * **NPM Cache and Resolver**: Deno interfaces with the popular NPM package manager, and the CLIFactory oversees the NPM cache and resolver. This facilitates seamless integration and utilization of NPM packages within Deno projects. * **Package.json Support and Installer**: CLIFactory encompasses support for the widely used package.json configuration format. It also includes an installer that streamlines the process of adding and managing dependencies through package.json. * **Type Checker**: Type checking is a crucial step in maintaining code quality. CLIFactory includes a type checker that assists in detecting type-related issues, promoting reliable and well-structured code. * **Common JS Resolver**: In JavaScript, CommonJS modules are important. The CLIFactory has a resolver that helps integrate CommonJS modules into Deno projects smoothly. In summation, the CLIFactory offer an array of indispensable services, collectively contributing to the seamless functioning of Deno. By overseeing diverse operations, from caching to module resolution, Deno's CLIFactory ensure a robust and comprehensive environment for developers to create and deploy their projects effectively. Below is the comprehensive list of services that the CLIFactory offers: Copy struct CliFactoryServices { deno_dir_provider: Deferred>, caches: Deferred>, file_fetcher: Deferred>, global_http_cache: Deferred>, http_cache: Deferred>, http_client: Deferred>, emit_cache: Deferred, emitter: Deferred>, fs: Deferred>, graph_container: Deferred>, lockfile: Deferred>>>, maybe_import_map: Deferred>>, maybe_inspector_server: Deferred>>, root_cert_store_provider: Deferred>, blob_store: Deferred>, parsed_source_cache: Deferred>, resolver: Deferred>, maybe_file_watcher_reporter: Deferred>, module_graph_builder: Deferred>, module_load_preparer: Deferred>, node_code_translator: Deferred>, node_resolver: Deferred>, npm_api: Deferred>, npm_cache: Deferred>, npm_resolver: Deferred>, npm_resolution: Deferred>, package_json_deps_provider: Deferred>, package_json_deps_installer: Deferred>, text_only_progress_bar: Deferred, type_checker: Deferred>, cjs_resolutions: Deferred>, } ### [](https://choubey.gitbook.io/internals-of-deno/foundations/program-state#internal-directories) Internal directories It would be beneficial to understand the internal directory paths that Deno utilizes for its operations. Below, you will find the primary directory paths on MacOS where Deno operates: Copy root dir: /Users/mayankc/Library/Caches/deno deps dir: /Users/mayankc/Library/Caches/deno/deps gen dir: /Users/mayankc/Library/Caches/deno/gen Below, you will find the directory paths designated for storing HTTP and HTTPS cache. These specific paths are situated within the 'deps' directory. The 'deps' directory, where these cache paths are located, serves as a storage hub for various dependencies and cached content. When Deno fetches external resources, it intelligently stores them in this directory, categorizing them based on their protocols (HTTP or HTTPS) to ensure organized and speedy retrieval when needed. Copy http cache: /Users/mayankc/Library/Caches/deno/deps/http https cache: /Users/mayankc/Library/Caches/deno/deps/https In Deno, a cache system for local files is not implemented. This is because the process of accessing local files is much more cost-effective than retrieving remote files over the network. As a result, Deno does not allocate resources to cache local files. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/program-state#filefetcher) FileFetcher The FileFetcher tool provides a useful function for obtaining a file and saving it within a storage area called the cache. This feature is versatile, capable of handling both local files stored on your device and remote files accessed through the internet. Interestingly, even local files are treated like fetched files, which essentially means they're read and handled in a similar manner. The FileFetcher entity consists of several key components, each contributing to its functionality: 1. **FileCache**: This component manages the storage of fetched files within a cache. It ensures efficient retrieval when needed. 2. **HttpCache**: Designed specifically for files acquired via HTTP requests, this element efficiently stores and handles such files within the cache. 3. **Cache Settings**: These settings define how the cache behaves. You can opt to use cached files, trigger a full reload of all files, and even configure other behaviors. 4. **Http Client**: This part of FileFetcher serves as a wrapper for the asynchronous `reqwest::Client` functionality. It streamlines the process of making HTTP requests and handling responses. FileFetcher offers support for three different URI schemes, which determine the type of resource being accessed: * **http**: Used for standard HTTP resources. * **https**: Similar to HTTP, but this scheme indicates a secure and encrypted connection. * **file**: Intended for local files present on the same device. When we discuss module loading, we will thoroughly examine the FileFetcher's role in efficiently obtaining and managing different types of files, both local and remote. We will see how it contributes to Deno's functionality. [](https://choubey.gitbook.io/internals-of-deno/foundations/program-state#functionality) Functionality ----------------------------------------------------------------------------------------------------------- In addition to its attributes, the CLIFactory provides another very important functionality that is closely associated with ES modules: The Module Preparer in the CLIFactory is very important. It handles fetching, loading, and compiling modules. When working with Deno and its CLIFactory, this function ensures modules are efficiently handled and prepared for use. It simplifies the process of gathering required modules, loading them into the application, and compiling them for seamless integration into your codebase. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/program-state#prepare-module) Prepare Module The process of getting a module ready for use in the JavaScript runtime is known as "module preparation" in Deno. This involves a series of steps, including fetching, loading, and compiling the module along with all its required components. This entire process ensures that the module and its dependencies are properly set up and ready to be utilized within the v8 engine. Let's look into some of the key activities that take place during the preparation of a module: 1. **Initiating the Module Graph:** All the ECMAScript (ES) modules are organized and managed within a structure called the "module graph." This graph keeps track of how various modules relate to each other. 2. **Inclusion of the Main Module:** The primary module, also known as the "main module," is integrated into the module graph. This marks the starting point of the entire preparation process. 3. **Recursive Dependency Exploration:** In a step-by-step manner, the system explores each module's dependencies by delving into their import statements. This recursive traversal builds up the module graph further. 4. **Fetching and Loading Dependencies:** The preparation process involves fetching and loading all the dependencies that a module relies upon. This ensures that the module has access to the necessary resources it needs to function. 5. **Compilation Process:** One of the vital stages is the compilation of the entire module graph. This step involves translating the code written in a high-level language (like JavaScript or TypeScript) into a format that can be readily executed by the v8 engine. As you can see, the preparation of a module in Deno is a multi-faceted process that ensures all required pieces are in place for smooth execution. We'll dive deeper into these steps when we discuss the specifics of module loading later on. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/program-state#fetch-compiled-module) Fetch compiled module This code snippet performs the task of retrieving a compiled module by providing its specifier. It uses the file fetcher to gather the compiled source corresponding to the specifier. We will look deeper into the details of this process when we discuss module loading in subsequent sections. [Previous5.4 Module Specifier](https://choubey.gitbook.io/internals-of-deno/foundations/resolve_url_or_path) [Next5.6 Permissions](https://choubey.gitbook.io/internals-of-deno/foundations/permissions) Last updated 1 year ago --- # 5.4 Module Specifier | The Internals of Deno The initial task for executing a program involves creating a module specifier designated for the main module. In Deno, the main module serves as the initial piece of code, file, or program that is provided as input. This core module is also referred to as the root module or the main module. Copy let main_module = cli_options.resolve_main_module()?; [](https://choubey.gitbook.io/internals-of-deno/foundations/resolve_url_or_path#overview) Overview ------------------------------------------------------------------------------------------------------- The concept of a "Module Specifier" permeates throughout Deno's codebase. Essentially, the role of a Module Specifier in Deno is to transform an input path into a URL format. This format can take on various forms, such as "file," "http," or "https" URLs. This URL format is crucial because it helps Deno accurately locate and access the required modules, whether they are local files or resources from the web. The code to resolve the main module is as follows: Copy DenoSubcommand::Run(run_flags) => { if run_flags.is_stdin() { std::env::current_dir() .context("Unable to get CWD") .and_then(|cwd| { resolve_url_or_path("./$deno$stdin.ts", &cwd) .map_err(AnyError::from) }) } else if run_flags.watch.is_some() { resolve_url_or_path(&run_flags.script, self.initial_cwd()) .map_err(AnyError::from) } else if NpmPackageReqReference::from_str(&run_flags.script).is_ok() { ModuleSpecifier::parse(&run_flags.script).map_err(AnyError::from) } else { resolve_url_or_path(&run_flags.script, self.initial_cwd()) .map_err(AnyError::from) } } The main interface is called "resolve\_url\_or\_path," and it's also used by various other subcommands. This interface combines two distinct functionalities: resolving URLs and resolving file paths. These functionalities are designed to take the input, which could be either a URL or a file path, and convert it into a format that adheres to a Uniform Resource Identifier (URI) scheme. This applies to different types of files, whether they are stored locally or accessed remotely. As a result of using this interface, you obtain an object known as the "ModuleSpecifier," which encapsulates the resolved information about the input. [](https://choubey.gitbook.io/internals-of-deno/foundations/resolve_url_or_path#functionality) Functionality ----------------------------------------------------------------------------------------------------------------- As we discussed previously, the primary API called resolve\_url\_or\_path carries out two main functions: * resolve\_url * resolve\_path Just as the names imply, the initial function is used for resolving a URL, whereas the latter one is employed to resolve a local file path. This means that the first function helps in figuring out the details of a web address, while the second function assists in determining the specifics of a file's location on your device. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MOTRXlhctvPYsax4ndS%252F-MOTTC4THnO7c69bcTlq%252Fmodule%2520specifier.png%3Falt%3Dmedia%26token%3D797c8c5a-9ccd-4890-8df6-315c69181b9b&width=768&dpr=4&quality=100&sign=d13428fc&sv=2) ### [](https://choubey.gitbook.io/internals-of-deno/foundations/resolve_url_or_path#resolve_url_or_path) resolve\_url\_or\_path Here is the source of the function resolve\_url\_or\_path: Copy pub fn resolve_url_or_path( specifier: &str, current_dir: &Path, ) -> Result { if specifier_has_uri_scheme(specifier) { resolve_url(specifier) } else { resolve_path(specifier, current_dir) } } The code is quite straightforward. * If input already has a URI scheme * resolve as URL * Otherwise, * resolve as a path (for local files) Let's take an example to understand this better. Copy > deno run helloLog.ts // RESOLVED PATH -- file:///Users/mayankc/Work/source/deno-vs-nodejs/helloLog.ts // -- > deno run https://raw.githubusercontent.com/mayankchoubey/deno-vs-nodejs/master/helloLog.ts // RESOLVED PATH -- https://raw.githubusercontent.com/mayankchoubey/deno-vs-nodejs/master/helloLog.ts The source code for the `resolve_url` function is also straightforward. It's not overly complex and can be easily understood. Copy pub fn resolve_url( url_str: &str, ) -> Result { Url::parse(url_str).map_err(ModuleResolutionError::InvalidUrl) } Since the input is already in the form of a URL, we proceed to parse the URL mainly to validate its correctness. Now, let's look into the source of the resolve\_path function, which is quite straightforward: Copy pub fn resolve_path( path_str: &str, current_dir: &Path, ) -> Result { let path = current_dir.join(path_str); let path = normalize_path(path); Url::from_file_path(&path) .map_err(|()| ModuleResolutionError::InvalidPath(path)) } In resolving the path, it takes three steps: * Append current directory path to input path * Example: `/Users/mayankc/Work/source/denoExamples/` + `helloLog.ts` * The path is normalized * Example: `/Users/mayankc/Work/source/denoExamples/` + `../../ABC.ts` gets normalized to `/Users/mayankc/Work/ABC.ts` * Build a file URI scheme from the path * Example: `/Users/mayankc/Work/source/denoExamples/helloLog.ts` gets converted to `file:///Users/mayankc/Work/source/denoExamples/helloLog.ts` ### [](https://choubey.gitbook.io/internals-of-deno/foundations/resolve_url_or_path#resolve_import) resolve\_import Up until now, we have examined how Deno handles the determination of the main module. This module refers to the primary file specified when using the 'deno run' command. But what about imports? It's important to note that imports also undergo a comparable resolution process. While we're discussing this subject, let's look into the concept of import resolution. It's worth mentioning that we will explore imports more extensively in Chapter 6 of this book. This will provide us with a deeper understanding of how Deno manages and resolves imported modules. Copy pub fn resolve_import( specifier: &str, base: &str, ) -> Result { let url = match Url::parse(specifier) { // 1. Apply the URL parser to specifier. // If the result is not failure, return he result. Ok(url) => url, // 2. If specifier does not start with the character U+002F SOLIDUS (/), // the two-character sequence U+002E FULL STOP, U+002F SOLIDUS (./), // or the three-character sequence U+002E FULL STOP, U+002E FULL STOP, // U+002F SOLIDUS (../), return failure. Err(ParseError::RelativeUrlWithoutBase) if !(specifier.starts_with('/') || specifier.starts_with("./") || specifier.starts_with("../")) => { let maybe_referrer = if base.is_empty() { None } else { Some(base.to_string()) }; return Err(ImportPrefixMissing(specifier.to_string(), maybe_referrer)); } // 3. Return the result of applying the URL parser to specifier with base // URL as the base URL. Err(ParseError::RelativeUrlWithoutBase) => { let base = Url::parse(base).map_err(InvalidBaseUrl)?; base.join(specifier).map_err(InvalidUrl)? } // If parsing the specifier as a URL failed for a different reason than // it being relative, always return the original error. We don't want to // return `ImportPrefixMissing` or `InvalidBaseUrl` if the real // problem lies somewhere else. Err(err) => return Err(InvalidUrl(err)), }; Ok(url) } [Previous5.3 Main program of Deno](https://choubey.gitbook.io/internals-of-deno/foundations/main-program) [Next5.5 CLI Factory](https://choubey.gitbook.io/internals-of-deno/foundations/program-state) Last updated 1 year ago --- # 5.18 What's next | The Internals of Deno And that concludes our overview of how a program operates within the Deno environment. We've journeyed from the starting point to the finish line, covering the essential steps along the way. This initial example marked our first foray into Deno, and while it may have been straightforward, its simplicity proved incredibly valuable in grasping the fundamental concepts. By sidestepping the intricacies of more complex user programs, we were able to focus on building a solid foundation of understanding. Now that we've grasped the fundamentals, we're ready to move forward. Next, we'll explore a simple program that introduces an asynchronous event triggered by a timer. In the next chapter of this book, we'll examine a program that incorporates various imports and operations. This will further enhance our understanding of Deno's capabilities and demonstrate how to effectively leverage its features. [Previous5.17 Evaluate module](https://choubey.gitbook.io/internals-of-deno/foundations/evaluate-module) [Next6.0 Cover page](https://choubey.gitbook.io/internals-of-deno/import-and-ops/chapter-cover-page) Last updated 1 year ago --- # 5.9 Run main module | The Internals of Deno Up to this point, we have successfully completed the initialization process for different components within Deno. Our main goal in initiating Deno was to run a basic "hello world" program. However, up until now, Deno has not yet engaged with our actual code—except for organizing the module specifier. The present moment marks the juncture where we get deep into the steps of arranging, loading, and ultimately running our code. Let's take a look at the code responsible for executing the main module: Copy let exit_code = worker.run().await?; Recall that the main worker was already created for the main module: Copy let mut worker = worker_factory .create_main_worker(main_module, permissions) .await?; [](https://choubey.gitbook.io/internals-of-deno/foundations/execute_module#overview) Overview -------------------------------------------------------------------------------------------------- The awaited moment has arrived with the introduction of the Run() function in Deno. This remarkable function serves as the catalyst for executing our code. Although it might seem simple with only two steps, the impact of these steps is profound and far-reaching. Let's explore the complexities of these two fundamental steps: * **Preload module** -- The first step entails the preloading of the module. This involves a meticulous process of recursively fetching not only the module itself, but also its interconnected dependencies. The web of connections is delicately woven as each dependent module is brought into the fold. -- Once the fetching is complete, the loaded modules are seamlessly integrated into the v8 engine, where their contents are prepared for execution. * **Instantiate the main module** -- The second step involves the instantiation of the main module. This module, representing the heart of our code, is meticulously prepared for execution. -- The module undergoes a rigorous evaluation process, where its contents are scrutinized and prepared for interpretation. Within the v8 engine, the module's instructions are comprehended and organized. -- With everything in place, the module is set to be executed within the v8 engine. The culmination of this execution triggers a crucial waiting period, during which the module's evaluation result is anticipated. * **Run event loop** -- The completion of these preparatory steps paves the way for the commencement of the event loop. This is the dynamic core of Deno's execution, where various asynchronous tasks are managed and orchestrated. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MOxnEo0uuhnCT-18u9k%252F-MOxnmlkQJftiJCbkPk7%252Fdeno%2520execute%2520module.png%3Falt%3Dmedia%26token%3D1b2da8e2-bf68-453a-9943-7000ae493fa9&width=768&dpr=4&quality=100&sign=40aa1eb7&sv=2) No matter how intricate the application might be, the beginning always takes place within the main module. This process occurs in a recursive manner, starting from the core, which is the main module itself. Since this operation involves asynchronous actions, the 'await' keyword is used at its conclusion. Naturally, the execution of the 'run' function takes place within the main worker. This is a component of the central processing thread. Now, let's see how the 'run' function is implemented: Copy pub async fn run(&mut self) -> Result { let mut maybe_coverage_collector = self.maybe_setup_coverage_collector().await?; log::debug!("main_module {}", self.main_module); if self.is_main_cjs { deno_node::load_cjs_module( &mut self.worker.js_runtime, &self.main_module.to_file_path().unwrap().to_string_lossy(), true, self.shared.options.inspect_brk, )?; } else { self.execute_main_module_possibly_with_npm().await?; } self.worker.dispatch_load_event(located_script_name!())?; loop { self .worker .run_event_loop(maybe_coverage_collector.is_none()) .await?; if !self .worker .dispatch_beforeunload_event(located_script_name!())? { break; } } self.worker.dispatch_unload_event(located_script_name!())?; if let Some(coverage_collector) = maybe_coverage_collector.as_mut() { self .worker .with_event_loop(coverage_collector.stop_collecting().boxed_local()) .await?; } Ok(self.worker.exit_code()) } Module loading is a crucial aspect of Deno, with all related code running asynchronously. We'll dive into a detailed examination of module loading, uncovering its inner mechanics as they relate to our code, specifically the main module. Our journey begins with the initial phase, "preload", which is embedded in the execute\_main\_module\_possibly\_with\_npm function. [](https://choubey.gitbook.io/internals-of-deno/foundations/execute_module#preload-module) Preload module -------------------------------------------------------------------------------------------------------------- The preload module is used to load and instantiate any ES module. This isn't limited to the main module. It is applicable to any ES module. The code of preload\_module is very simple: Copy pub async fn execute_main_module_possibly_with_npm( &mut self, ) -> Result<(), AnyError> { let id = self.worker.preload_main_module(&self.main_module).await?; self.evaluate_module_possibly_with_npm(id).await } pub async fn preload_main_module( &mut self, module_specifier: &ModuleSpecifier, ) -> Result { self .js_runtime .load_main_module(module_specifier, None) .await } The function "preload\_main\_module" essentially delegates the task to the JavaScript runtime's "load\_main\_module" function. The process of loading the main module takes place in a sequence of four steps: 1. **Loading the Main Module:** Initially, the main module is loaded into the runtime environment. 2. **Recursive Dependency Loading:** The process then involves poking into the main module and subsequently loading all the dependencies it requires. This step is carried out recursively, meaning that dependencies of dependencies are also loaded as needed. 3. **Main Module Instantiation:** After the dependencies are loaded, the main module is instantiated. This means that the code within the main module is executed and any initializations or setups are performed. 4. **Dependency Resolution:** Once the main module is instantiated, the process moves on to resolve any dependencies. This involves making sure that all the required components are available and connected properly. The outcome of the "load\_module" function is an identification number for the module. This identification number needs to be immediately used when evaluating the module. This sequential procedure ensures that the main module and its dependencies are loaded, prepared, and connected appropriately within the Deno runtime environment. [](https://choubey.gitbook.io/internals-of-deno/foundations/execute_module#evaluate-module) Evaluate module ---------------------------------------------------------------------------------------------------------------- Evaluation is essentially the same as execution. This is the term used by V8, the JavaScript engine, and it's also the term Deno adopts. When we talk about evaluation, we're essentially asking V8 to carry out the instructions contained in the code. In Deno, it's the way we prompt V8 to execute the module that has been created. However, it's important to note that the outcome of this evaluation doesn't show up right away. Instead, it appears as something called a "future." This future represents the result that we expect from the evaluation process. It's like a placeholder that will eventually be filled with the outcome of the evaluation. Only when the module evaluation is completely done, this future gets resolved – meaning it gets filled in with the actual result we were waiting for. This process ensures that we can keep track of the progress and completion of the evaluation while allowing us to work with the results effectively once they're ready. The job of event loop is to keep track of the evaluation result. [](https://choubey.gitbook.io/internals-of-deno/foundations/execute_module#receive-evaluation-result) Receive evaluation result ------------------------------------------------------------------------------------------------------------------------------------ The process of evaluation in Deno involves asynchronous calls that reach their conclusions through various pathways. These pathways are as follows: 1. **Completion due to Tasks Done:** * When all tasks within the main module have been executed. * This includes resolving any pending operations (ops). * Also, this covers the completion of dynamic import processing. 2. **Unhandled Exception:** * If an unhandled exception occurs during the execution. When there are no pending tasks remaining and no dynamic imports to process, the program's evaluation comes to a conclusion, signifying that the program has finished its intended operations. In the context of our uncomplicated example, it's anticipated that the evaluation process will promptly come to an end since there are no asynchronous operations or never-ending event listeners in the program. We will dig deeper into the intricacies of how this evaluation process operates a little later in the text, gaining a better understanding of its functioning. \-- Now that we have a general understanding of the process, let's explore the critical stages of loading and evaluation in more depth. First, we'll examine the intricacies of the loading process. Next, we'll analyze the loading procedure of a simple "hello world" program, which lacks imports and consists of a single main module. This straightforward example allows us to understand the fundamental principles more easily. In the next chapter, we'll explore a program with multiple imports, which adds complexity to the loading mechanism. This will provide a more comprehensive understanding of Deno's capabilities. [Previous5.8 JS Runtime](https://choubey.gitbook.io/internals-of-deno/foundations/jsruntime) [Next5.10 Load module](https://choubey.gitbook.io/internals-of-deno/foundations/load-module) Last updated 1 year ago --- # 5.10 Load module | The Internals of Deno This subsection holds significant importance, perhaps being the most pivotal one within the text. In the upcoming paragraphs, as well as those that follow, the discussion revolves around the manner in which Deno undertakes the loading of modules into the V8 engine. Up until this juncture, Deno's efforts have predominantly been directed towards our specific program by transforming the given path into a module specifier and accomplishing a substantial portion of its initialization tasks. [](https://choubey.gitbook.io/internals-of-deno/foundations/load-module#overview) Overview ----------------------------------------------------------------------------------------------- The `load_main_module` function in JsRuntime is a significant piece of code, notable not for its length, but for its diverse functionality. In the following discussion, we will explore key aspects of this function. The module loading process follows a recursive pattern, continuing until both the main module and its dependent submodules are successfully loaded. The sequence of actions involved in module loading occurs through a sequence of distinct steps: ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MOxOLdyoiiNL4aqH2zW%252F-MOxPBk4q68Zidbl2LZW%252Fdeno%2520load%2520module.png%3Falt%3Dmedia%26token%3D37258f57-55f1-4e15-ba6f-397348919c67&width=768&dpr=4&quality=100&sign=4ed50205&sv=2) We have repeatedly traversed these steps multiple times within the recent preceding sections. We begin by considering the root module and initiating its loading. Following this, we load all associated dependencies. Afterward, both the root module and its dependencies are brought into existence. This phase involves the instantiation of the root module along with its dependent modules. As we conclude this loading process, the main module, along with its dependencies, finds instantiation within the V8 engine. This iterative process has been extensively covered in the preceding sections, providing a comprehensive understanding of the sequential actions involved. [](https://choubey.gitbook.io/internals-of-deno/foundations/load-module#steps-in-loading) Steps in loading --------------------------------------------------------------------------------------------------------------- The process of loading a module in Deno occurs through a sequence of clear steps. Let's take a closer look at each of these steps to better understand how it all works: 1. _Acquiring the Module Loader_: To begin, Deno retrieves the module loader, which is responsible for managing and orchestrating the module loading process. 2. _Loading and Registering Modules and Dependencies_: The next step involves a recursive procedure where both the main module and all its associated dependencies are loaded and registered. Deno meticulously fetches each module, making sure all the interdependencies are accounted for and properly integrated. 3. _Generating the Module Graph_: As a result of the comprehensive recursive loading process, something referred to as a "module graph" is formed. This graph essentially represents the intricate web of connections between different modules and their relationships. 4. _Creating Instances of the Main Module_: After successfully establishing the module graph, Deno proceeds to create instances of the main module. This is achieved using the unique identifier of the root (main) module as a reference. The underlying code that drives the loading of the main module within the JavaScript runtime is encapsulated in the function named `load_main_module`. This function embodies the logic that orchestrates the entire process we've just outlined. Copy pub async fn load_main_module( &self, isolate: &mut v8::Isolate, specifier: &ModuleSpecifier, code: Option, ) -> Result { let module_map_rc = self.0.module_map(); if let Some(code) = code { let specifier = specifier.as_str().to_owned().into(); let scope = &mut self.handle_scope(isolate); // true for main module module_map_rc .borrow_mut() .new_es_module(scope, true, specifier, code, false) .map_err(|e| match e { ModuleError::Exception(exception) => { let exception = v8::Local::new(scope, exception); exception_to_err_result::<()>(scope, exception, false).unwrap_err() } ModuleError::Other(error) => error, })?; } let mut load = ModuleMap::load_main(module_map_rc.clone(), &specifier).await?; while let Some(load_result) = load.next().await { let (request, info) = load_result?; let scope = &mut self.handle_scope(isolate); load.register_and_recurse(scope, &request, info).map_err( |e| match e { ModuleError::Exception(exception) => { let exception = v8::Local::new(scope, exception); exception_to_err_result::<()>(scope, exception, false).unwrap_err() } ModuleError::Other(error) => error, }, )?; } let root_id = load.root_module_id.expect("Root module should be loaded"); self.instantiate_module(isolate, root_id).map_err(|e| { let scope = &mut self.handle_scope(isolate); let exception = v8::Local::new(scope, e); exception_to_err_result::<()>(scope, exception, false).unwrap_err() })?; Ok(root_id) } Now, let's take a quick look at the steps involved in this process. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/load-module#recursively-load-module) Recursively load module Recursive module loading might seem intricate, but it's a systematic process that unfolds in the following manner: We kick off with the root module or the main module, treating it as the starting point. This module is fetched from its source. Afterward, it's established as the foundational module within the module graph, which is like a visual representation of how different modules are connected. To make sense of the main module's content, it's parsed – essentially, we break down its code to understand its structure and components. But a module rarely stands alone; it often relies on other modules. Thus, we traverse through its direct dependencies, each time fetching a dependency, parsing it, and then adding it to our module graph. This recursive procedure keeps going until all the required dependencies have been processed. As the code in these modules contains imports, these imports signify ES modules, a way of structuring code for better organization and reusability. In this journey of exploration, the code dives deeper into the module tree, loading each module it encounters along the way. Having constructed the graph step by step, the next phase involves transpilation – converting the graph into a format that's more suitable for execution. Now, it's time to traverse this transpiled graph, registering each module as we encounter them. The purpose of registration is to furnish all the necessary module details to the V8 engine, which is responsible for executing JavaScript code. Once everything is set up, modules come to life. The process of instantiation involves taking each module and its dependencies and initializing them within the V8 engine. This final step ensures that your code can run seamlessly, leveraging the interconnected modules to perform the tasks you've programmed. Recursive module loading is akin to navigating a forest of interconnected trees, where each tree symbolizes a module. Our journey ensures effective exploration, understanding, and assembly of these modules for seamless execution. While the process may seem complex, with many steps involved, the v8 engine simplifies module loading into two essential steps: 1. Loading the module 2. Instantiating it ### [](https://choubey.gitbook.io/internals-of-deno/foundations/load-module#register-modules) Register modules Iterate through the graph in an asynchronous manner while registering all the modules. This process is notably simpler when contrasted with loading modules recursively. By adopting this approach, the complexity of handling module registrations is significantly reduced. In comparison to the intricate nature of recursive module loading, this method offers a straightforward and efficient means of managing module dependencies. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/load-module#instantiate-the-main-module) Instantiate the main module Upon the completion of the registration process, both the main module and its dependencies are loaded into the V8 engine. Now comes the moment to create an instance of the root module. This action is only necessary for the root module; V8 will take care of managing and retrieving all the required dependencies using callbacks. \-- That was just a brief overview to get us started. Now, let's dive deeper into the next section, where we'll explore the construction of the module graph in greater detail. This graph is a crucial data structure in Deno's internal workings, and understanding it is essential to appreciating how Deno operates. [Previous5.9 Run main module](https://choubey.gitbook.io/internals-of-deno/foundations/execute_module) [Next5.11 Recursive module loading](https://choubey.gitbook.io/internals-of-deno/foundations/4.11-recursive-module-loading-and-module-graphs) Last updated 1 year ago --- # 6.1 Imports and ops | The Internals of Deno In the previous chapter, we explored the execution of a simple "hello world" program in Deno, examining key concepts in depth. The program's simplicity was intentional, allowing us to focus on understanding Deno's functionality rather than the program itself. Our exploration covered a range of crucial topics, including: * The execution command known as "Run" * The central entity referred to as the "Main Worker" * The mechanism responsible for loading modules * Visualization of the module relationships through the module graph * The component handling the retrieval of files * The JavaScript Runtime, which is pivotal for execution * The multifaceted process of module loading * The intricate evaluation procedure of these loaded modules * And many other interconnected elements. This chapter marks a new stage in our exploration. We will build on our initial "hello world" example by adding imports, synchronous and asynchronous operations. This will significantly improve our understanding of how imports work in Deno's framework and how Deno and the V8 engine manage operations. Note that this chapter is a direct continuation of the previous one. With a solid foundation in place, we will focus on new concepts and insights, rather than revisiting the basics. [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/console-print-with-delay#chapter-contents) Chapter contents ------------------------------------------------------------------------------------------------------------------------------- [6.1 Imports and ops](https://choubey.gitbook.io/internals-of-deno/import-and-ops/console-print-with-delay) [6.2 Hello world program v2](https://choubey.gitbook.io/internals-of-deno/import-and-ops/basic-console-print-with-delay) [6.3 Module graph with imports](https://choubey.gitbook.io/internals-of-deno/import-and-ops/building-module-graph) [6.4 Transpile](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.4-transpile) [6.5 Registration and instantiation](https://choubey.gitbook.io/internals-of-deno/import-and-ops/registration-and-instantiation) [6.6 Registration of ops](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.6-registration-of-ops) [6.7 Evaluate module](https://choubey.gitbook.io/internals-of-deno/import-and-ops/evaluate-module) [6.8 Sync OPs](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.8-sync-op) [6.9 Debug logs](https://choubey.gitbook.io/internals-of-deno/import-and-ops/6.9-debug-logs) [Previous6.0 Cover page](https://choubey.gitbook.io/internals-of-deno/import-and-ops/chapter-cover-page) [Next6.2 Hello world program v2](https://choubey.gitbook.io/internals-of-deno/import-and-ops/basic-console-print-with-delay) Last updated 1 year ago --- # 5.16 Instantiate module | The Internals of Deno After completing the registration process, the ES module gets compiled within the V8 engine. Now, it's the right moment to create an instance of the module(s). This step involves initializing and preparing the module for execution. In other words, it's like setting up a workspace for the module to do its job. [](https://choubey.gitbook.io/internals-of-deno/foundations/instantiate-module#overview) Overview ------------------------------------------------------------------------------------------------------ This phase comes right after the registration process and serves as the final stage in loading modules. By the completion of this step, all the modules that are loaded statically have been processed. Should there be any dynamic imports in the code, those are managed during runtime, meaning they're handled while the program is already running. During the registration step, we obtained the unique identifier for the root module. This identifier is then employed to create an instance of the root module within the V8 engine. Given that our example doesn't involve any imports from other modules, there's just one module that needs to be instantiated at this point. Next, let's examine the final step in the `load_main_module` function: Copy let root_id = load.root_module_id.expect("Root module should be loaded"); self.instantiate_module(isolate, root_id).map_err(|e| { let scope = &mut self.handle_scope(isolate); let exception = v8::Local::new(scope, e); exception_to_err_result::<()>(scope, exception, false).unwrap_err() })?; Ok(root_id) [](https://choubey.gitbook.io/internals-of-deno/foundations/instantiate-module#instantiation) Instantiation ---------------------------------------------------------------------------------------------------------------- The module instantiation work is quite simple. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MPM7HmaFCTSGE6FWRtF%252F-MPM7jP8e_spijGq3-Ve%252Fdeno%2520instantiate%2520steps.png%3Falt%3Dmedia%26token%3Daaf5639d-9f03-4ea9-b595-4e456bbcba34&width=768&dpr=4&quality=100&sign=b70322d7&sv=2) Here is the code for the main instantiation function: Copy pub(crate) fn instantiate_module( &mut self, scope: &mut v8::HandleScope, id: ModuleId, ) -> Result<(), v8::Global> { let tc_scope = &mut v8::TryCatch::new(scope); let module = self .get_handle(id) .map(|handle| v8::Local::new(tc_scope, handle)) .expect("ModuleInfo not found"); if module.get_status() == v8::ModuleStatus::Errored { return Err(v8::Global::new(tc_scope, module.get_exception())); } tc_scope.set_slot(self as *const _); let instantiate_result = module.instantiate_module(tc_scope, Self::module_resolve_callback); tc_scope.remove_slot::<*const Self>(); if instantiate_result.is_none() { let exception = tc_scope.exception().unwrap(); return Err(v8::Global::new(tc_scope, exception)); } Ok(()) } This is a straightforward function with these steps: * Obtain the module related to the input id (which is the root module id). * Instantiate this module into v8 format The process of instantiation is usually a repeating one, but for our basic "hello world" instance, it isn't. This is because there aren't any external elements being brought in. When a module is transformed into v8's format, it's done in a manner that goes in circles, yet for our uncomplicated illustration, this looping doesn't happen. The rationale for this is that there aren't any dependencies being pulled in. Whenever the module-to-v8 conversion occurs, it requires a particular callback: * module\_resolve\_callback In scenarios where there are dependencies, v8 leverages this callback to acquire their references. However, in our instance, this callback won't be invoked due to the absence of dependencies. \-- We've reached the end of the loading phase - what a journey! From scratch, we explored various concepts and successfully loaded a module. Module loading is complete once instantiation occurs. Now, let's revisit the worker code: Copy pub async fn execute_main_module( &mut self, module_specifier: &ModuleSpecifier, ) -> Result<(), AnyError> { let id = self.preload_main_module(module_specifier).await?; self.evaluate_module(id).await } The loading process is now complete. By this stage, all the modules along with their necessary dependencies have been obtained, stored in memory for quick access, transformed into a suitable format for execution, compiled if needed, and successfully brought into the V8 engine. Moving forward, the subsequent and final action in the execute\_main\_module sequence entails assessing the module's content using the primary module ID. This evaluation of the module essentially involves executing its instructions. And so, at last, the moment has come to set our code into motion. [Previous5.15 Register / compile module](https://choubey.gitbook.io/internals-of-deno/foundations/register-and-instantation) [Next5.17 Evaluate module](https://choubey.gitbook.io/internals-of-deno/foundations/evaluate-module) Last updated 1 year ago --- # 5.11 Recursive module loading | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/foundations/4.11-recursive-module-loading-and-module-graphs#overview) Overview ----------------------------------------------------------------------------------------------------------------------------------- As demonstrated in the preceding section, the process of loading modules recursively is a complex task. The result of this recursive module loading process is a module graph, which holds parsed modules. All the dependencies of these modules are loaded upon the completion of this phase. Going further, let's explore a segment of code within the JavaScript runtime's `load_main_module()` function that holds significance for the recursive module loading process: Copy let mut load = ModuleMap::load_main(module_map_rc.clone(), &specifier).await?; while let Some(load_result) = load.next().await { let (request, info) = load_result?; let scope = &mut self.handle_scope(isolate); load.register_and_recurse(scope, &request, info).map_err( |e| match e { ModuleError::Exception(exception) => { let exception = v8::Local::new(scope, exception); exception_to_err_result::<()>(scope, exception, false).unwrap_err() } ModuleError::Other(error) => error, }, )?; } Here are the required procedures in this section: 1. _Establishing a Recursive Module Loader_: To begin, we must construct a module loader that works in a loop, repeatedly calling itself. This loader takes various arguments, including the module loader itself. 2. _Getting Modules Ready_: Next, we need to make sure our modules are ready for use. This involves two main actions: * Fetching: We retrieve the necessary modules, ensuring they are available for the program. * Inserting into Graph: We organize these modules within a graphical structure, creating connections between them based on their dependencies. 3. _Iterating Through Modules and Registration_: After the preparation phase, we proceed to loop through all the modules. This process is also referred to as instantiation or registration. During this loop, each module is formally registered, allowing it to be utilized as part of the program's execution. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MOyKwurD1PJnIl2ESn5%252F-MOyMF4yGFM4d3SEuxjx%252Fdeno%2520recursive%2520module%2520loading.png%3Falt%3Dmedia%26token%3D8c90d2e9-9c07-452a-a371-afb9e009b584&width=768&dpr=4&quality=100&sign=b6b0cf4b&sv=2) The primary function is called "register\_and\_recurse," and it's quite intricate in its workings: * It performs a recursive journey through all the modules using a graph-like structure of data. * During this process, it covers all the distinct modules present. * The function retrieves all the necessary imports that a module needs. * This action involves further recursive exploration of all the imports. Copy pub(crate) fn register_and_recurse( &mut self, scope: &mut v8::HandleScope, module_request: &ModuleRequest, module_source: ModuleSource, ) -> Result<(), ModuleError> { let expected_asserted_module_type = module_source.module_type.into(); let module_url_found = module_source.module_url_found; let module_url_specified = module_source.module_url_specified; if module_request.asserted_module_type != expected_asserted_module_type { return Err(ModuleError::Other(generic_error(format!( "Expected a \"{}\" module but loaded a \"{}\" module.", module_request.asserted_module_type, module_source.module_type, )))); } // Register the module in the module map unless it's already there. If the // specified URL and the "true" URL are different, register the alias. let module_url_found = if let Some(module_url_found) = module_url_found { let (module_url_found1, module_url_found2) = module_url_found.into_cheap_copy(); self.module_map_rc.borrow_mut().alias( module_url_specified, expected_asserted_module_type, module_url_found1, ); module_url_found2 } else { module_url_specified }; let maybe_module_id = self .module_map_rc .borrow() .get_id(&module_url_found, expected_asserted_module_type); let module_id = match maybe_module_id { Some(id) => { debug!( "Already-registered module fetched again: {:?}", module_url_found ); id } None => match module_source.module_type { ModuleType::JavaScript => { self.module_map_rc.borrow_mut().new_es_module( scope, self.is_currently_loading_main_module(), module_url_found, module_source.code, self.is_dynamic_import(), )? } ModuleType::Json => self.module_map_rc.borrow_mut().new_json_module( scope, module_url_found, module_source.code, )?, }, }; // Recurse the module's imports. There are two cases for each import: // 1. If the module is not in the module map, start a new load for it in // `self.pending`. The result of that load should eventually be passed to // this function for recursion. // 2. If the module is already in the module map, queue it up to be // recursed synchronously here. // This robustly ensures that the whole graph is in the module map before // `LoadState::Done` is set. let mut already_registered = VecDeque::new(); already_registered.push_back((module_id, module_request.clone())); self.visited.insert(module_request.clone()); while let Some((module_id, module_request)) = already_registered.pop_front() { let referrer = ModuleSpecifier::parse(&module_request.specifier).unwrap(); let imports = self .module_map_rc .borrow() .get_requested_modules(module_id) .unwrap() .clone(); for module_request in imports { if !self.visited.contains(&module_request) && !self .visited_as_alias .borrow() .contains(&module_request.specifier) { if let Some(module_id) = self.module_map_rc.borrow().get_id( module_request.specifier.as_str(), module_request.asserted_module_type, ) { already_registered.push_back((module_id, module_request.clone())); } else { let request = module_request.clone(); let specifier = ModuleSpecifier::parse(&module_request.specifier).unwrap(); let visited_as_alias = self.visited_as_alias.clone(); let referrer = referrer.clone(); let loader = self.loader.clone(); let is_dynamic_import = self.is_dynamic_import(); let fut = async move { // `visited_as_alias` unlike `visited` is checked as late as // possible because it can only be populated after completed // loads, meaning a duplicate load future may have already been // dispatched before we know it's a duplicate. if visited_as_alias.borrow().contains(specifier.as_str()) { return Ok(None); } let load_result = loader .load(&specifier, Some(&referrer), is_dynamic_import) .await; if let Ok(source) = &load_result { if let Some(found_specifier) = &source.module_url_found { visited_as_alias .borrow_mut() .insert(found_specifier.as_str().to_string()); } } load_result.map(|s| Some((request, s))) }; self.pending.push(fut.boxed_local()); } self.visited.insert(module_request); } } } // Update `self.state` however applicable. if self.state == LoadState::LoadingRoot { self.root_module_id = Some(module_id); self.root_asserted_module_type = Some(module_source.module_type.into()); self.state = LoadState::LoadingImports; } if self.pending.is_empty() { self.state = LoadState::Done; } Ok(()) } } From the long function above, we can look at two important points: _Get the imports present in a module_ Copy while let Some((module_id, module_request)) = already_registered.pop_front() { let imports = self .module_map_rc .borrow() .get_requested_modules(module_id) .unwrap() .clone(); ..... for module_request in imports { .... already_registered.push_back((module_id, module_request.clone())); .... } } Add imports to the list and visit them if not already visited. \-- In the next section, we'll explore the process of building module graphs, which will reveal the interconnectedness of modules. We'll also examine the specific module graph related to our example, providing a detailed visual representation of its structure and relationships. [Previous5.10 Load module](https://choubey.gitbook.io/internals-of-deno/foundations/load-module) [Next5.12 Module graphs](https://choubey.gitbook.io/internals-of-deno/foundations/4.12-module-graphs) Last updated 1 year ago --- # 5.3 Main program of Deno | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/foundations/main-program#deno-command) Deno command -------------------------------------------------------------------------------------------------------- We'll use the following command to run our code in Deno: Copy > deno run helloLog.ts * deno is the name of the executable * run is the subcommand * helloLog.ts is the program to run In this subsection, we will explore a simple program that doesn't require special permissions. Although this program doesn't need permissions, we will discuss the concept of permissions in this chapter. Permissions are a fundamental part of Deno, creating a protective barrier around the runtime environment and ensuring security. The 'deno run' command has a specific purpose: it converts TypeScript code into JavaScript, ready for execution. It prioritizes quick startup over type checking, preparing the code for running without extensively verifying the types involved. For strict type checking of your application's code, Deno offers a separate command called 'deno check'. This command thoroughly examines the types in your codebase, ensuring they align correctly and your code is error-free. This additional step can help catch potential issues before they cause problems during execution. [](https://choubey.gitbook.io/internals-of-deno/foundations/main-program#denos-main-program) Deno's main program --------------------------------------------------------------------------------------------------------------------- Deno is crafted using the Rust programming language, and as a result, its primary codebase is also composed in Rust. The central component of Deno, known as the main program, is housed within the CLI. This CLI serves a dual role, functioning as both the conductor that coordinates various tasks and the supplier of essential services that Deno offers. Contained within this CLI is the heart of Deno, its main program. This program acts as the control center, governing the execution of Deno's functionalities and managing how it interacts with your commands and scripts. It's responsible for overseeing tasks such as module loading, security checks, and runtime environment management. Copy pub fn main() { setup_panic_hook(); util::unix::raise_fd_limit(); util::windows::ensure_stdio_open(); #[cfg(windows)] colors::enable_ansi(); // For Windows 10 deno_runtime::permissions::set_prompt_callbacks( Box::new(util::draw_thread::DrawThread::hide), Box::new(util::draw_thread::DrawThread::show), ); let args: Vec = env::args().collect(); let future = async move { let current_exe_path = current_exe()?; let standalone_res = match standalone::extract_standalone(¤t_exe_path, args.clone()) .await { Ok(Some((metadata, eszip))) => standalone::run(eszip, metadata).await, Ok(None) => Ok(()), Err(err) => Err(err), }; // TODO(bartlomieju): doesn't handle exit code set by the runtime properly unwrap_or_exit(standalone_res); let flags = match flags_from_vec(args) { Ok(flags) => flags, Err(err @ clap::Error { .. }) if err.kind() == clap::error::ErrorKind::DisplayHelp || err.kind() == clap::error::ErrorKind::DisplayVersion => { err.print().unwrap(); std::process::exit(0); } Err(err) => unwrap_or_exit(Err(AnyError::from(err))), }; let default_v8_flags = match flags.subcommand { DenoSubcommand::Lsp => vec!["--max-old-space-size=3072".to_string()], _ => vec![], }; init_v8_flags(&default_v8_flags, &flags.v8_flags, get_v8_flags_from_env()); util::logger::init(flags.log_level); run_subcommand(flags).await }; let exit_code = unwrap_or_exit(create_and_run_current_thread_with_maybe_metrics(future)); std::process::exit(exit_code); } The deno command functions as a toolchain, but it cannot operate on its own to execute tasks. Deno relies on subcommands to perform its various functions and capabilities. Regarding the main function code of Deno, it undertakes the following tasks: ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MOPr5rbwUsRfUFxRpy_%252F-MOTJJpVQUEmKp-CdR4q%252Fdeno%2520main%2520program.png%3Falt%3Dmedia%26token%3D9edc2762-d42b-4b8d-90f5-92bc3fe00033&width=768&dpr=4&quality=100&sign=df5090b0&sv=2) * Build flags from command line args * Initialize logger * Run the subcommand asynchronously (notice the await at the end) Drawing a parallel between Rust's futures and JavaScript's promises can provide a clearer understanding of their roles in asynchronous programming. In the Rust programming language, futures operate similarly to promises in JavaScript. Just as promises represent values that may not be available immediately but will be fulfilled at some point, Rust futures encapsulate computations that will be completed in the future. These futures serve as powerful tools for handling asynchronous tasks, making them a important in writing efficient and responsive code. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/main-program#flags) Flags A flag acts like a tool that examines and understands command-line details related to specific sub-commands. Its purpose is to make these details easily accessible. Deno comes with a variety of sub-commands, and each of these sub-commands comes with its own set of arguments. To simplify things, flags are used to sort through these arguments based on the specific sub-command being used. This way, the important arguments are picked out and stored for further use, all depending on which sub-command you're working with. In essence, flags help organize the command-line information so that Deno can effectively handle the various sub-commands and their unique requirements. Each sub-command in Deno has its own set of flags, which are specific to its functions. Let's look at a few commonly used sub-commands to understand them better. Copy pub struct CheckFlags { pub files: Vec, } pub struct CoverageFlags { pub files: FileFlags, pub output: Option, pub include: Vec, pub exclude: Vec, pub lcov: bool, } pub struct RunFlags { pub script: String, pub watch: Option, } In Deno, there are certain attributes that are in the form of booleans, such as `allow_read`, `allow_write`, and `allow_env`. These attributes can be thought of as settings that determine whether specific actions are permitted or not. For instance, `allow_read` controls whether the script is allowed to read files or not, `allow_write` governs the ability to write to files, and `allow_env` manages access to environment variables. By setting these attributes to either "true" or "false," you can specify the permissions granted to the script. Additionally, Deno provides attributes that are in the form of lists, such as `write_allowlist` and `read_allowlist`. These lists contain specific paths or resources that are explicitly permitted. When you include paths in the `write_allowlist`, the script is only allowed to write to the specified paths, and similarly, when you include paths in the `read_allowlist`, the script can only read from those specified paths. This mechanism allows for a more granular control over the file system and resource access, enhancing security and minimizing unintended interactions. Below is a compilation of frequently utilized command line options in Deno. These options allow you to interact with the Deno runtime effectively: * \--allow-all * \--allow-read * \--deny-read * \--allow-write * \--deny-write * \--allow-net * \--deny-net * \--unsafely-ignore-certificate-errors * \--allow-env * \--deny-env * \--allow-run * \--deny-run * \--allow-sys * \--deny-sys * \--allow-ffi * \--deny-ffi * \--allow-hrtime * \--deny-hrtime Deno offers an extensive array of subcommands, yet the primary focus of this book revolves around the execution of code. Therefore, our in-depth exploration will be solely dedicated to the 'run' command. While Deno presents a wide range of subcommands catering to various functionalities, our attention within this book remains centered on the intricacies of the 'run' command's utilization. [](https://choubey.gitbook.io/internals-of-deno/foundations/main-program#run-command) Run command ------------------------------------------------------------------------------------------------------ The run command in Deno serves the purpose of executing your program. Although the implementation of this command may appear straightforward, it actually undertakes numerous tasks behind the scenes. Let's look into the mechanics of the run command and explore its inner workings. Copy pub async fn run_script( flags: Flags, run_flags: RunFlags, ) -> Result { if !flags.has_permission() && flags.has_permission_in_argv() { log::warn!( "{}", crate::colors::yellow( r#"Permission flags have likely been incorrectly set after the script argument. To grant permissions, set them before the script argument. For example: deno run --allow-read=. main.js"# ) ); } if let Some(watch_flags) = run_flags.watch { return run_with_watch(flags, watch_flags).await; } // TODO(bartlomieju): actually I think it will also fail if there's an import // map specified and bare specifier is used on the command line let factory = CliFactory::from_flags(flags).await?; let deno_dir = factory.deno_dir()?; let http_client = factory.http_client(); let cli_options = factory.cli_options(); // Run a background task that checks for available upgrades. If an earlier // run of this background task found a new version of Deno. super::upgrade::check_for_upgrades( http_client.clone(), deno_dir.upgrade_check_file_path(), ); let main_module = cli_options.resolve_main_module()?; maybe_npm_install(&factory).await?; let permissions = PermissionsContainer::new(Permissions::from_options( &cli_options.permissions_options(), )?); let worker_factory = factory.create_cli_main_worker_factory().await?; let mut worker = worker_factory .create_main_worker(main_module, permissions) .await?; let exit_code = worker.run().await?; Ok(exit_code) } And that's all there is to it! The code you see here constitutes the entirety of the implementation for the run command. It may seem remarkably straightforward on the surface. Nevertheless, beneath this apparent simplicity, a significant amount of activity takes place within the functions that are invoked from the run\_script function. Now, let's delve into the essential aspects of the `run_script` function and gain an overview of its crucial functions. As we proceed, we'll explore these concepts in greater depth within the upcoming sections. Our focus for now will be on the pertinent segments of the code that deserve attention. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/main-program#modulespecifier) ModuleSpecifier The initial file used in the application, which is provided as input to the 'deno run' command, is referred to as the main file or main module of the application. This main module serves as the entry point where the application's execution begins. When you run your Deno application, Deno looks for the specified main file and starts executing the code from there. The primary file you use in Deno can either exist on your computer (what we call "local") or be hosted on the internet (which we refer to as "remote"). Deno is smart enough to change the script's location into a special web address called a URL. URLs can come in different flavors, like file:// or http:// or even https://. Interestingly, Deno treats local files the same way as remote ones, using the file:// format. This might seem a bit strange, but it's actually a way to make sure that Deno handles all types of files in a consistent manner. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/main-program#check-for-upgrades) Check for upgrades When you execute a program in Deno, you may have observed an informational message when it starts up. This message notifies you about the availability of a new version of Deno. This happens because Deno is designed to keep you updated with its latest versions, ensuring you have access to the newest features, improvements, and security enhancements. Copy > deno run -A main.ts A new release of Deno is available: 1.35.3 → 1.36.1 Run `deno upgrade` to install it. Listening on http://localhost:8000/ ### [](https://choubey.gitbook.io/internals-of-deno/foundations/main-program#main-worker) Main worker The following step involves launching a main worker factory, which is then used to construct a main worker. The main worker is essentially the core thread that operates the primary module. This central worker is inherently generated and consistently linked with the main module. In comparison, web workers are forged when required. After the establishment of the main worker, the program commences its execution through the employment of the main worker's run API. This API facilitates the initiation of the program's operational journey within the main worker's domain. \-- The earlier explanation gave a basic overview of the processes in the run command. Now, let's break down each step in detail. We will start by understanding the basic concept of ModuleSpecifier. [Previous5.2 Basic hello world](https://choubey.gitbook.io/internals-of-deno/foundations/basic-hello-world) [Next5.4 Module Specifier](https://choubey.gitbook.io/internals-of-deno/foundations/resolve_url_or_path) Last updated 1 year ago --- # 7.4 What's next | The Internals of Deno This concludes our exploration of Deno's internal storage APIs, where we dissected both persistent and transient storage mechanisms. This also marks the culmination of the book's third edition. [Previous7.3 Session storage](https://choubey.gitbook.io/internals-of-deno/chapter-7-local-and-session-storage/7.3-session-storage) [NextAfterword](https://choubey.gitbook.io/internals-of-deno/afterword/afterword) Last updated 1 year ago --- # 6.10 What's next | The Internals of Deno That was all about import and ops work in Deno. This has been the most important chapter of this book. This is where we've learned how everything falls into place, even slightly complex concepts. * * * In the next chapter, we'll check the internals of two built-in storage that comes with Deno: * Local storage, and * Session storage The next chapter dives into the internal workings of two built-in Deno storage options: * Local storage * Session storage Familiar to frontend developers, these browser storage mechanisms are readily available within Deno and offer powerful data persistence capabilities. [Previous6.9 Debug logs](https://choubey.gitbook.io/internals-of-deno/import-and-ops/6.9-debug-logs) [Next7.0 Cover page](https://choubey.gitbook.io/internals-of-deno/chapter-7-local-and-session-storage/7.0-cover-page) Last updated 1 year ago --- # 5.6 Permissions | The Internals of Deno The subsequent phase in executing code involves establishing permissions. Permissions stand out as a distinctive feature within the Deno runtime system. Deno provides an exceptional and safeguarded environment for running programs. This exclusive safeguarding is achieved through the mechanism of permissions. Deno's permissions framework allows users to control precisely what actions and resources their programs can access, adding an extra layer of security to the execution process. Copy let permissions = PermissionsContainer::new(Permissions::from_options( &cli_options.permissions_options(), )?); [](https://choubey.gitbook.io/internals-of-deno/foundations/permissions#overview) Overview ----------------------------------------------------------------------------------------------- Permissions play a crucial role in enabling Deno to provide a secure and distinctive sandboxed environment. Within Deno, a wide array of permissions and permission types exist, each serving a specific purpose. These permissions empower developers to control what their Deno programs can access and interact with. By categorizing permissions into types like read, write, env, and net, Deno tailors its security model to cater to different scenarios and use cases. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/permissions#list-of-permissions) List of permissions Deno offers a variety of permissions to enhance its functionality. As the number of permissions increases, the level of sandboxing becomes more finely tuned. Below is a compilation of frequently employed permissions along with their explanations: 1. **Read Permission:** This permission allows Deno to access and read files from the local system. It is essential for tasks that involve reading data or configurations stored on the user's device. 2. **Write Permission:** With the write permission, Deno gains the ability to modify and create files. This is crucial when your program needs to update files or generate new ones during its execution. 3. **Net Permission:** The net permission permits Deno to initiate network connections. This is necessary for tasks like making API requests or establishing socket connections with remote servers. 4. **Environment Permission:** This permission provides access to the environment variables of the system. It comes in handy when your application needs to gather information about the environment it's running in. 5. **Run Permission:** Deno's run permission allows it to execute other programs or scripts. This is useful for scenarios where your code needs to interact with external executables or scripts. 6. **FFI Permission:** The FFI (Foreign Function Interface) permission is more advanced. It enables Deno to interface with functions written in languages like C or Rust, expanding the range of capabilities your application can harness. 7. **Hrtime Permission:** This permission is related to high-resolution time measurements. It permits Deno to access accurate timing information, which is crucial for tasks that require precise timing, such as benchmarking or performance optimization. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MOwh3zubn8l3o641_pp%252F-MOwiEJLxGgCFLWjiR3-%252Fdeno%2520permissions.png%3Falt%3Dmedia%26token%3D73b41cbd-069d-45ec-b48a-6d2382505fa3&width=768&dpr=4&quality=100&sign=c23d7c6&sv=2) Certain types of permissions in Deno can be boiled down to a simple true or false value, making them boolean in nature. To put it plainly, these permissions are all about whether access is given or denied. On the flip side, some permission types require a more complex approach. Deno maintains lists for these permissions, adding an extra layer of granularity to the access control. This meticulous handling of permissions results in a stricter form of sandboxing, where each action is examined under a magnifying glass. Let's look into the permission types that benefit from this meticulous treatment by having additional lists associated with them: 1. **Net Allow List**: This permission type revolves around network-related actions. By maintaining an additional list, Deno ensures that only approved network interactions take place. This is especially crucial in environments where network security is paramount. 2. **Read Allow List**: Reading files or data is a common task, but it can also be risky if not properly managed. Deno mitigates this risk by using a read allow list. Files specified in this list are the only ones that can be accessed, preventing unauthorized data exposure. 3. **Write Allow List**: Writing to files or altering data demands even greater caution. Deno acknowledges this by using a write allow list. Only the files granted explicit permission can be modified, maintaining data integrity and thwarting any potential malicious activities. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/permissions#states) States IIn Deno, permissions have three possible states. You can learn about these states by exploring Deno's internal mechanisms or by looking at TypeScript/JavaScript code written by users These three states are: 1. **Prompt**: When a permission is in this state, Deno seeks the user's input to decide whether to grant or deny it. 2. **Granted**: In this state, Deno has given the green light to the permission, allowing the code to carry out its intended actions without any hindrance. 3. **Denied**: When a permission is denied, Deno firmly puts its foot down, preventing the code from exercising the particular action associated with the permission. To witness the transition of a permission through these states, let's take a closer look: ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MOwiG49ITqSzLttknwz%252F-MOwj6lVcxk_V_fp7Czb%252Fdeno%2520permission%2520states.png%3Falt%3Dmedia%26token%3D3ede245d-e8b8-4f21-b404-10def7964d1f&width=768&dpr=4&quality=100&sign=d3f411fc&sv=2) * If --allow-YYY is specified * The permission is granted * If --deny-YYY is specified * The permission is denied * Else * Prompt for permission * If prompt resulted in G (or grant) * The permission is granted * Else * The permission is denied [](https://choubey.gitbook.io/internals-of-deno/foundations/permissions#building-permissions) Building permissions ----------------------------------------------------------------------------------------------------------------------- The function called "from\_options" is rather straightforward. It works by going through the flags that have been analyzed and constructs an object related to permissions for each of the permissions that are supported. Certain permissions are binary in nature, meaning they can only be either "true" or "false." However, there are other permissions that might also come with a list of items. Let's take a look at a code snippet that illustrates how to create a permission along with an optional list: Copy pub fn from_options(opts: &PermissionsOptions) -> Result { Ok(Self { read: Permissions::new_read( &opts.allow_read, &opts.deny_read, opts.prompt, )?, write: Permissions::new_write( &opts.allow_write, &opts.deny_write, opts.prompt, )?, net: Permissions::new_net(&opts.allow_net, &opts.deny_net, opts.prompt)?, env: Permissions::new_env(&opts.allow_env, &opts.deny_env, opts.prompt)?, sys: Permissions::new_sys(&opts.allow_sys, &opts.deny_sys, opts.prompt)?, run: Permissions::new_run(&opts.allow_run, &opts.deny_run, opts.prompt)?, ffi: Permissions::new_ffi(&opts.allow_ffi, &opts.deny_ffi, opts.prompt)?, hrtime: Permissions::new_hrtime(opts.allow_hrtime, opts.deny_hrtime), }) } pub fn new_net( allow_list: &Option>, deny_list: &Option>, prompt: bool, ) -> Result, AnyError> { Ok(UnaryPermission:: { granted_global: global_from_option(allow_list), granted_list: parse_net_list(allow_list)?, flag_denied_global: global_from_option(deny_list), flag_denied_list: parse_net_list(deny_list)?, prompt, ..Default::default() }) } In Deno, when it comes to actions involving reading, writing, and network operations, there are distinct permission levels to consider. These levels help control what the program is allowed to do. Let's break down the details: 1. **Global State Permission:** This level of permission involves specifying whether the program has the authority to access global state information. It's like giving the program the key to the central vault where important data is kept. This permission is either set to true (allowed) or false (not allowed). * **Granted List:** If there is an allow list in place, the program's requested actions are reviewed against this list. It's akin to having a guest list at a party. The program's actions are compared to the list, and if the program's request matches an entry on the list, it's granted permission to proceed. This way, only invited actions get through. * **Deny List:** Conversely, if there's a deny list established, the program's requested actions are cross-checked with this list. Imagine a "do not enter" list at a restricted area. If the program's actions align with any entry on this list, access is denied. This ensures that certain actions are explicitly forbidden. 2. **Boolean Permission for Other Actions:** For actions like handling environment variables, running programs, utilizing plugins, and measuring high-resolution time (hrtime), the permission structure is simpler. It's a straightforward yes-or-no scenario. These actions are governed by a boolean permission. It's like a simple switch: either the permission is specified (on) or not (off). There's no middle ground here. Either the program is allowed to carry out these actions, or it isn't. Here is the code which converts any permission to boolean: Copy fn unit_permission_from_flag_bools( allow_flag: bool, deny_flag: bool, name: &'static str, description: &'static str, prompt: bool, ) -> UnitPermission { UnitPermission { name, description, state: if deny_flag { PermissionState::Denied } else if allow_flag { PermissionState::Granted } else { PermissionState::Prompt }, prompt, } } If --deny-YYY is there, permission is denied. If --allow-YYY is not there, permission state is granted, otherwise prompt. Permissions are typically provided at the beginning of the main program, unless Deno requests them during runtime. Once these permissions are established, they are consistently checked whenever needed. [](https://choubey.gitbook.io/internals-of-deno/foundations/permissions#querying-permissions) Querying permissions ----------------------------------------------------------------------------------------------------------------------- All the permissions usually get built at startup. For a production system, it'd be impractical to prompt for permissions at runtime. Although Deno supports prompting for permissions, it's unlikely that it'd ever get used for production code. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/permissions#query-for-permission) Query for permission Obtaining a boolean permission using code in Deno is remarkably straightforward, as the process essentially involves retrieving the current state of the permission. Below, you'll find a few illustrative examples that showcase how this is done: Copy fn query_desc( &self, desc: &Option, allow_partial: AllowPartial, ) -> PermissionState { if self.is_flag_denied(desc) || self.is_prompt_denied(desc) { PermissionState::Denied } else if self.is_granted(desc) { match allow_partial { AllowPartial::TreatAsGranted => PermissionState::Granted, AllowPartial::TreatAsDenied => { if self.is_partial_flag_denied(desc) { PermissionState::Denied } else { PermissionState::Granted } } AllowPartial::TreatAsPartialGranted => { if self.is_partial_flag_denied(desc) { PermissionState::GrantedPartial } else { PermissionState::Granted } } } } else if matches!(allow_partial, AllowPartial::TreatAsDenied) && self.is_partial_flag_denied(desc) { PermissionState::Denied } else { PermissionState::Prompt } } ### [](https://choubey.gitbook.io/internals-of-deno/foundations/permissions#query-for-a-permission-with-an-optional-list) Query for a permission with an optional list Some permissions can either be boolean or a list. For example - * For read permission, input is path * For write permission, input is path * For net permission, input is URL Here is the code for querying any list kind of permission: Copy fn is_granted(&self, desc: &Option) -> bool { Self::list_contains(desc, self.granted_global, &self.granted_list) } fn list_contains( desc: &Option, list_global: bool, list: &HashSet, ) -> bool { match desc.as_ref() { Some(desc) => list_global || list.iter().any(|v| v.stronger_than(desc)), None => list_global, } } [](https://choubey.gitbook.io/internals-of-deno/foundations/permissions#request-and-revoke-permissions) Request and revoke permissions ------------------------------------------------------------------------------------------------------------------------------------------- You can also ask for and withdraw permissions during the program's execution. Although this might not be very practical for real-world usage, Deno does allow for it. Let's take a brief look at these two aspects using a simple example involving the 'env' permission: In some situations, you might need to change the permissions your Deno program has while it's running. For instance, you might want to access certain environment variables using the 'env' permission. This permission lets your program interact with the environment it's running in, like reading sensitive information such as API keys or configuration settings. The following code requests a permission at runtime: Copy pub fn request(&mut self, env: Option<&str>) -> PermissionState { self.request_desc(&env.map(EnvDescriptor::new), || None) } fn request_desc( &mut self, desc: &Option, get_display_name: impl Fn() -> Option, ) -> PermissionState { let state = self.query_desc(desc, AllowPartial::TreatAsPartialGranted); if state == PermissionState::Granted { self.insert_granted(desc.clone()); return state; } if state != PermissionState::Prompt { return state; } let mut message = String::with_capacity(40); message.push_str(&format!("{} access", T::flag_name())); match get_display_name() { Some(display_name) => { message.push_str(&format!(" to \"{}\"", display_name)) } None => match desc { Some(desc) => message.push_str(&format!(" to \"{}\"", desc.name())), None => {} }, } match permission_prompt( &message, T::flag_name(), Some("Deno.permissions.request()"), true, ) { PromptResponse::Allow => { self.insert_granted(desc.clone()); PermissionState::Granted } PromptResponse::Deny => { self.insert_prompt_denied(desc.clone()); PermissionState::Denied } PromptResponse::AllowAll => { self.insert_granted(None); PermissionState::Granted } } } Revoke is straightforward. It simply moves permission from granted to prompt. Copy pub fn revoke(&mut self, env: Option<&str>) -> PermissionState { self.revoke_desc(&env.map(EnvDescriptor::new)) } fn revoke_desc(&mut self, desc: &Option) -> PermissionState { match desc.as_ref() { Some(desc) => self.granted_list.retain(|v| !v.stronger_than(desc)), None => { self.granted_global = false; // Revoke global is a special case where the entire granted list is // cleared. It's inconsistent with the granular case where only // descriptors stronger than the revoked one are purged. self.granted_list.clear(); } } self.query_desc(desc, AllowPartial::TreatAsPartialGranted) } The process of requesting permission in Deno involves a significant step that impacts how permissions are managed. When a permission is requested, it starts in a "prompt" state, awaiting a response. If the user responds to the prompt by denying the permission (typically indicated by the letter "D"), the permission transitions from the prompt state to the "denied" state. This means that the user's choice to deny the permission is acknowledged and recorded. On the other hand, if the user responds by granting the permission (often indicated by the letter "G"), the permission undergoes a different transition. It shifts from the prompt state to the "granted" state. This transition signifies that the user has allowed the requested permission, and Deno is now authorized to proceed with the associated actions. [Previous5.5 CLI Factory](https://choubey.gitbook.io/internals-of-deno/foundations/program-state) [Next5.7 Main Worker](https://choubey.gitbook.io/internals-of-deno/foundations/mainworker) Last updated 1 year ago --- # 6.2 Hello world program v2 | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/basic-console-print-with-delay#ts-and-js-code) TS and JS code --------------------------------------------------------------------------------------------------------------------------------- We'll continue to use Typescript for our examples. The program is the same, with some additions: * There are two imports with one import has another inside it * There is a synchronous op * There is an asynchronous op * Also, there are some console logs which are also synchronous ops The file name is `helloV2.ts.` The location of the source file is: `/Users/mayankc/Work/source/denoExamples/helloV2.ts` Here is the code that we'll use to understand some new concepts: Copy import { nanoid } from "npm:nanoid"; import { getMachineId } from "https://deno.land/x/machine_id/mod.ts"; const id = nanoid(); const machineId = await getMachineId(); const homeDir = Deno.env.get("HOME"); function printNumber(input: number) { console.log(input); } function printString(input: string) { console.log(input); } printNumber(1); printString("One"); console.log("Nanoid=", id, ", MachineId=", machineId, ", homeDir=", homeDir); This remains a straightforward program consisting of a single TypeScript file. At the program's outset, two imports are used to access ES modules: * nanoid * machine\_id The 'nanoid' module is sourced from the NPM registry, while the 'machine\_id' module is acquired from Deno's registry. The program can be partitioned into three distinct segments: 1. _Import Section_: The initial portion encompasses the two imports, serving the purpose of illustrating the internal mechanics of the import process. 2. _Synchronous and Asynchronous Operations_: This phase involves invocations of both synchronous and asynchronous operations. The operations involving 'nanoid' and 'env.get' are categorized as synchronous, whereas the 'getMachineId' operation is asynchronous in nature. 3. _TypeScript Code_: The subsequent part entails the inclusion of code extracted from 'helloLog.ts', showcasing the conversion process from TypeScript to JavaScript. This step is taken to offer a demonstration of how TypeScript code translates into its JavaScript equivalent. Following the conversion to JavaScript, the program's structure transforms as illustrated below: Copy import { nanoid } from "npm:nanoid"; import { getMachineId } from "https://deno.land/x/machine_id/mod.ts"; const id = nanoid(); const machineId = await getMachineId(); const homeDir = Deno.env.get("HOME"); function printNumber(input) { console.log(input); } function printString(input) { console.log(input); } printNumber(1); printString("One"); console.log("Nanoid=", id, ", MachineId=", machineId, ", homeDir=", homeDir); //# sourceMappingURL=data:application/json;base64,eyJ2ZXJzaW9uIjozLCJzb3VyY2VzIjpbImZpbGU6Ly8vVXNlcnMvbWF5YW5rYy9Xb3JrL3NvdXJjZS9kZW5vRXhhbXBsZXMvaGVsbG9WMi50cyJdLCJzb3VyY2VzQ29udGVudCI6WyJpbXBvcnQgeyBuYW5vaWQgfSBmcm9tIFwibnBtOm5hbm9pZFwiO1xuaW1wb3J0IHsgZ2V0TWFjaGluZUlkIH0gZnJvbSBcImh0dHBzOi8vZGVuby5sYW5kL3gvbWFjaGluZV9pZC9tb2QudHNcIjtcblxuY29uc3QgaWQgPSBuYW5vaWQoKTtcbmNvbnN0IG1hY2hpbmVJZCA9IGF3YWl0IGdldE1hY2hpbmVJZCgpO1xuY29uc3QgaG9tZURpciA9IERlbm8uZW52LmdldChcIkhPTUVcIik7XG5cbmZ1bmN0aW9uIHByaW50TnVtYmVyKGlucHV0OiBudW1iZXIpIHtcbiAgY29uc29sZS5sb2coaW5wdXQpO1xufVxuXG5mdW5jdGlvbiBwcmludFN0cmluZyhpbnB1dDogc3RyaW5nKSB7XG4gIGNvbnNvbGUubG9nKGlucHV0KTtcbn1cblxucHJpbnROdW1iZXIoMSk7XG5wcmludFN0cmluZyhcIk9uZVwiKTtcbmNvbnNvbGUubG9nKFwiTmFub2lkPVwiLCBpZCwgXCIsIE1hY2hpbmVJZD1cIiwgbWFjaGluZUlkLCBcIiwgaG9tZURpcj1cIiwgaG9tZURpcik7XG4iXSwibmFtZXMiOltdLCJtYXBwaW5ncyI6IkFBQUEsU0FBUyxNQUFNLFFBQVEsYUFBYTtBQUNwQyxTQUFTLFlBQVksUUFBUSx3Q0FBd0M7QUFFckUsTUFBTSxLQUFLO0FBQ1gsTUFBTSxZQUFZLE1BQU07QUFDeEIsTUFBTSxVQUFVLEtBQUssR0FBRyxDQUFDLEdBQUcsQ0FBQztBQUU3QixTQUFTLFlBQVksS0FBYTtFQUNoQyxRQUFRLEdBQUcsQ0FBQztBQUNkO0FBRUEsU0FBUyxZQUFZLEtBQWE7RUFDaEMsUUFBUSxHQUFHLENBQUM7QUFDZDtBQUVBLFlBQVk7QUFDWixZQUFZO0FBQ1osUUFBUSxHQUFHLENBQUMsV0FBVyxJQUFJLGdCQUFnQixXQUFXLGNBQWMifQ== This section discusses the similarities and slight differences in the current version of the 'hello world' program compared to the previous one. The primary changes revolve around imports and operations (ops). It's worth noting that some ops, like `console.log`, were present in the previous version as well. \-- Let's take a moment to review. The primary worker in Deno operates through two main phases: * Module loading * Module evaluation The process of loading a module can vary due to the presence of imports. Now, we will discuss creating a module graph for a program with imports. This involves showing how modules connect and interact in a program. To create a module graph, Deno analyzes the code to find all imported modules and their dependencies. It then organizes these modules in a structured way, showing their connections. [Previous6.1 Imports and ops](https://choubey.gitbook.io/internals-of-deno/import-and-ops/console-print-with-delay) [Next6.3 Module graph with imports](https://choubey.gitbook.io/internals-of-deno/import-and-ops/building-module-graph) Last updated 1 year ago --- # 6.0 Cover page | The Internals of Deno ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252FaM1gnDO5FOzPMnVKJTvP%252F7.png%3Falt%3Dmedia%26token%3D385ae175-5cf8-4ff2-88f8-1798a8598913&width=768&dpr=4&quality=100&sign=85b292b&sv=2) [Previous5.18 What's next](https://choubey.gitbook.io/internals-of-deno/foundations/4.18-whats-next) [Next6.1 Imports and ops](https://choubey.gitbook.io/internals-of-deno/import-and-ops/console-print-with-delay) Last updated 1 year ago --- # 6.9 Debug logs | The Internals of Deno Just like the last chapter, the following are Deno's debug logs while running helloV2 application: Copy DEBUG RS - deno::args::package_json:147 - package.json file found at '/Users/mayankc/Work/source/denoExamples/package.json' Main module: file:///Users/mayankc/Work/source/denoExamples/helloV2.ts DEBUG RS - deno::cache::cache_db:130 - Opening cache /Users/mayankc/Library/Caches/deno/dep_analysis_cache_v1... DEBUG RS - deno::cache::cache_db:130 - Opening cache /Users/mayankc/Library/Caches/deno/node_analysis_cache_v1... DEBUG RS - deno::js:11 - Deno isolate init with snapshots. DEBUG JS - args [] DEBUG RS - deno::worker:133 - main_module file:///Users/mayankc/Work/source/denoExamples/helloV2.ts DEBUG RS - deno::module_loader:111 - Preparing module load. DEBUG RS - deno::module_loader:125 - Creating module graph. DEBUG RS - deno::file_fetcher:516 - FileFetcher::fetch() - specifier: file:///Users/mayankc/Work/source/denoExamples/helloV2.ts DEBUG RS - deno_runtime::permissions:86 - ⚠️️ Granted read access to "/Users/mayankc/Work/source/denoExamples/helloV2.ts" DEBUG RS - deno::file_fetcher:516 - FileFetcher::fetch() - specifier: https://deno.land/x/machine_id/mod.ts DEBUG RS - deno_runtime::permissions:86 - ⚠️️ Granted net access to "deno.land" DEBUG RS - deno::file_fetcher:332 - FileFetcher::fetch_remote() - specifier: https://deno.land/x/machine_id/mod.ts DEBUG RS - deno_runtime::permissions:86 - ⚠️️ Granted net access to "deno.land" DEBUG RS - deno::file_fetcher:243 - FileFetcher::fetch_cached - specifier: https://deno.land/x/machine_id/mod.ts DEBUG RS - reqwest::connect:429 - starting new connection: https://deno.land/ DEBUG RS - reqwest::connect:429 - starting new connection: https://dl.deno.land/ DEBUG RS - reqwest::async_impl::client:2287 - redirect policy disallowed redirection to 'https://deno.land/x/machine_id@v0.3.0/mod.ts' Warning Implicitly using latest version (v0.3.0) for https://deno.land/x/machine_id/mod.ts DEBUG RS - deno::http_util:59 - Redirecting to "/x/machine_id@v0.3.0/mod.ts"... DEBUG RS - deno::file_fetcher:332 - FileFetcher::fetch_remote() - specifier: https://deno.land/x/machine_id@v0.3.0/mod.ts DEBUG RS - deno_runtime::permissions:86 - ⚠️️ Granted net access to "deno.land" DEBUG RS - deno::file_fetcher:243 - FileFetcher::fetch_cached - specifier: https://deno.land/x/machine_id@v0.3.0/mod.ts DEBUG RS - reqwest::connect:429 - starting new connection: https://registry.npmjs.org/ DEBUG RS - deno_npm::resolution::snapshot:715 - Resolved nanoid@* to nanoid@4.0.2 DEBUG RS - deno_npm::resolution::graph:972 - - Resolved nanoid@4.0.2 to nanoid@4.0.2 DEBUG RS - deno::util::fs:546 - Acquiring file lock at /Users/mayankc/Work/source/denoExamples/node_modules/.deno/.deno.lock DEBUG RS - deno::util::fs:565 - Acquired file lock at /Users/mayankc/Work/source/denoExamples/node_modules/.deno/.deno.lock DEBUG RS - deno::module_loader:188 - Prepared module load. Transpiling: file:///Users/mayankc/Work/source/denoExamples/helloV2.ts DEBUG RS - deno::npm::resolvers:121 - Resolved package folder of nanoid@4.0.2 to /Users/mayankc/Work/source/denoExamples/node_modules/.deno/nanoid@4.0.2/node_modules/nanoid DEBUG RS - deno::npm::resolvers:141 - Resolved package folder of file:///Users/mayankc/Work/source/denoExamples/node_modules/.deno/nanoid@4.0.2/node_modules/nanoid/index.js to /Users/mayankc/Work/source/denoExamples/node_modules/.deno/nanoid@4.0.2/node_modules/nanoid DEBUG RS - deno::npm::resolvers:141 - Resolved package folder of file:///Users/mayankc/Work/source/denoExamples/node_modules/.deno/nanoid@4.0.2/node_modules/nanoid/index.js to /Users/mayankc/Work/source/denoExamples/node_modules/.deno/nanoid@4.0.2/node_modules/nanoid DEBUG RS - deno::npm::resolvers:141 - Resolved package folder of file:///Users/mayankc/Work/source/denoExamples/node_modules/.deno/nanoid@4.0.2/node_modules/nanoid/index.js to /Users/mayankc/Work/source/denoExamples/node_modules/.deno/nanoid@4.0.2/node_modules/nanoid DEBUG RS - deno::npm::resolvers:141 - Resolved package folder of file:///Users/mayankc/Work/source/denoExamples/node_modules/.deno/nanoid@4.0.2/node_modules/nanoid/url-alphabet to /Users/mayankc/Work/source/denoExamples/node_modules/.deno/nanoid@4.0.2/node_modules/nanoid Transpiling: https://deno.land/x/machine_id@v0.3.0/mod.ts DEBUG RS - deno::npm::resolvers:141 - Resolved package folder of file:///Users/mayankc/Work/source/denoExamples/node_modules/.deno/nanoid@4.0.2/node_modules/nanoid/url-alphabet/index.js to /Users/mayankc/Work/source/denoExamples/node_modules/.deno/nanoid@4.0.2/node_modules/nanoid Module resolve callback: npm:nanoid DEBUG RS - deno::npm::resolvers:121 - Resolved package folder of nanoid@4.0.2 to /Users/mayankc/Work/source/denoExamples/node_modules/.deno/nanoid@4.0.2/node_modules/nanoid Module resolve callback: https://deno.land/x/machine_id/mod.ts Module resolve callback: crypto DEBUG RS - deno::npm::resolvers:141 - Resolved package folder of file:///Users/mayankc/Work/source/denoExamples/node_modules/.deno/nanoid@4.0.2/node_modules/nanoid/index.js to /Users/mayankc/Work/source/denoExamples/node_modules/.deno/nanoid@4.0.2/node_modules/nanoid Module resolve callback: ./url-alphabet/index.js DEBUG RS - deno::npm::resolvers:141 - Resolved package folder of file:///Users/mayankc/Work/source/denoExamples/node_modules/.deno/nanoid@4.0.2/node_modules/nanoid/index.js to /Users/mayankc/Work/source/denoExamples/node_modules/.deno/nanoid@4.0.2/node_modules/nanoid DEBUG RS - deno::npm::resolvers:141 - Resolved package folder of file:///Users/mayankc/Work/source/denoExamples/node_modules/.deno/nanoid@4.0.2/node_modules/nanoid/url-alphabet to /Users/mayankc/Work/source/denoExamples/node_modules/.deno/nanoid@4.0.2/node_modules/nanoid DEBUG RS - deno::npm::resolvers:141 - Resolved package folder of file:///Users/mayankc/Work/source/denoExamples/node_modules/.deno/nanoid@4.0.2/node_modules/nanoid/index.js to /Users/mayankc/Work/source/denoExamples/node_modules/.deno/nanoid@4.0.2/node_modules/nanoid DEBUG RS - deno_runtime::permissions:86 - ⚠️️ Granted run access to "ioreg" Polling event loop... Polling event loop... Polling event loop... Polling event loop... Polling event loop... DEBUG RS - deno_runtime::permissions:86 - ⚠️️ Granted env access to "HOME" 1 One Nanoid= XCnwo4iJtWHlYqUntgiva , MachineId= 2D809C23-917A-5052-BB4F-38794CCCE9FA , homeDir= /Users/mayankc DEBUG RS - deno_runtime::worker:518 - received module evaluate Ok( Ok( (), ), ) Polling event loop... Polling event loop... Timeout occured [Previous6.8 Sync OPs](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.8-sync-op) [Next6.10 What's next](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.11-whats-next) Last updated 1 year ago --- # 5.14 Transpile | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/foundations/4.13-check-and-or-transpilation#overview) Overview ------------------------------------------------------------------------------------------------------------------- Once the graph of dependencies has been constructed, and all the required modules have been fetched and parsed, it's possible that these modules contain a combination of JavaScript (JS) and TypeScript (TS) code. However, since the V8 engine, which Deno uses, can only process JavaScript code, the next step is to handle the modules appropriately. This involves processing the dependency graph and transforming any TypeScript code into JavaScript. This phase is known as "checking" or "transpilation." It's worth noting that the Deno runtime's primary concern is quick startup, which is why the `deno run` command is designed to focus on transpilation. In other words, when you use `deno run`, the modules will be transpiled into JavaScript to ensure a faster execution. On the other hand, if you specifically want to perform type checking for explicit type information verification, you should utilize the `deno check` command. This command ensures that the TypeScript code is checked for correctness in terms of types and other related aspects. In the upcoming sections, we will explore both of these processes—transpilation and type checking. The choice between "check" and "transpile" depends on the sub-command you use: * When using `deno run`, the SWC compiler will be employed to perform transpilation. * When utilizing `deno check`, the TSC (TypeScript Compiler) will be used to perform a thorough type check on the codebase. By understanding these two distinct approaches, you'll be well-equipped to handle the necessary steps to ensure your Deno applications are both efficiently executed and free from type-related issues. [](https://choubey.gitbook.io/internals-of-deno/foundations/4.13-check-and-or-transpilation#functionality) Functionality ----------------------------------------------------------------------------------------------------------------------------- In previous iterations of Deno, type checking used to be integrated into the 'deno run' command. However, in more recent versions, this feature has been completely phased out from the 'deno run' command. When you execute a program now, there is no mandatory requirement for type checking during the startup phase. This is particularly true if your code editor has inherent type-checking capabilities; in such instances, it's likely that your code has been composed accurately in terms of types. In scenarios where the editor provides robust type-checking, the initial type-checking step can be omitted. The primary motivation behind this omission is to enhance the startup speed of Deno. This optimization is aimed at facilitating quicker execution of your programs. It's worth noting that the SWC compiler, which is leveraged by Deno, boasts an approximate eightfold improvement in speed when compared to the traditional TypeScript Compiler (TSC). If you find yourself needing to perform type-checking, either because your editor doesn't support it or as an extra layer of protection, you have a couple of options to consider. One option is to make use of Microsoft's TypeScript (TSC) compiler, which is coded in JavaScript. It's worth noting that this compiler tends to run rather slowly in comparison to SWC. So, when it comes down to it, you have a choice to make based on your preferences. You can opt for the TSC compiler if you need that type-checking, even though it might be slower, or you can go with SWC for potentially faster performance. The decision ultimately rests in your hands as the user. [](https://choubey.gitbook.io/internals-of-deno/foundations/4.13-check-and-or-transpilation#transpile) Transpile --------------------------------------------------------------------------------------------------------------------- In the process of transpilation in Deno, two significant files are generated as outputs. Let's take the example of a file named "helloLog.ts": 1. **helloLog.ts.js**: This file holds the code that has been transformed and converted. It takes the TypeScript code from "helloLog.ts" and compiles it into JavaScript. This JavaScript version can be executed directly by the Deno runtime. It's the result of the transpilation process that allows TypeScript code, which is a higher-level language for developers, to be translated into the lower-level JavaScript code that computers understand. 2. **helloLog.ts.meta**: In this file, you'll find a hash value. This hash serves an essential purpose – it acts as a fingerprint for the "helloLog.ts" file. Think of it as a unique identifier that represents the content and structure of the original TypeScript file. This hash is used by Deno to determine whether any changes have occurred in the "helloLog.ts" file. When you make modifications to the TypeScript code, Deno can compare the new hash with the one stored in the meta file. If the hashes differ, Deno knows that the file has been altered and requires transpilation again. The journey begins with the initial TypeScript code you write. It might include functions, classes, variables, and other programming constructs. This code, residing in the "helloLog.ts" file, serves as the foundation. Deno's transpilation process takes this TypeScript code and generates the corresponding JavaScript code in the ".js" file. This conversion ensures that your code can be executed seamlessly within the Deno environment. The "helloLog.ts.meta" file, though seemingly small, plays a crucial role in maintaining the integrity and efficiency of the development process. Its hash value acts as a guardian, helping Deno decide when it's time to perform transpilation. When you alter the TypeScript code, remember that the hash in the meta file will change too, signaling to Deno that a transformation is needed before the updated code can be executed. The original file is as follows: Copy // File helloLog.ts function printNumber(input: number) { console.log(input); } function printString(input: string) { console.log(input); } printNumber("One"); printString("One"); The transpiled JS file is as follows: Copy // File helloLog.ts function printNumber(input) { console.log(input); } function printString(input) { console.log(input); } printNumber("One"); printString("One"); //# sourceMappingURL=data:application/json;base64,eyJ2ZXJzaW9uIjozLCJzb3VyY2VzIjpbImZpbGU6Ly8vVXNlcnMvbWF5YW5rYy9Xb3JrL3NvdXJjZS9kZW5vRXhhbXBsZXMvaGVsbG9Mb2cudHMiXSwic291cmNlc0NvbnRlbnQiOlsiLy8gRmlsZSBoZWxsb0xvZy50c1xuXG5mdW5jdGlvbiBwcmludE51bWJlcihpbnB1dDogbnVtYmVyKSB7XG4gIGNvbnNvbGUubG9nKGlucHV0KTtcbn1cblxuZnVuY3Rpb24gcHJpbnRTdHJpbmcoaW5wdXQ6IHN0cmluZykge1xuICBjb25zb2xlLmxvZyhpbnB1dCk7XG59XG5cbnByaW50TnVtYmVyKFwiT25lXCIpO1xucHJpbnRTdHJpbmcoXCJPbmVcIik7XG4iXSwibmFtZXMiOltdLCJtYXBwaW5ncyI6IkFBQUEsbUJBQW1CO0FBRW5CLFNBQVMsWUFBWSxLQUFhO0lBQ2hDLFFBQVEsSUFBSTtBQUNkO0FBRUEsU0FBUyxZQUFZLEtBQWE7SUFDaEMsUUFBUSxJQUFJO0FBQ2Q7QUFFQSxZQUFZO0FBQ1osWUFBWSJ9 The generated .meta file contains metadata associated with the source file. It's a JSON file with the following data: Copy {"source_hash":"11056719879154141555","emit_hash":"14271523416398949797"} The term "source\_hash" refers to the CRC checksum of the source file. For every individual file, a unique source\_hash is calculated, and this calculated hash is then compared with the hash that is stored. Whenever these two hashes match each other, it signifies that the source content remains unchanged. As a result, there is no need to go through the transpilation process, which is a step that converts code from one programming language to another. Copy fn get_source_hash(&self, source_text: &str) -> u64 { FastInsecureHasher::new() .write_str(source_text) .write_u64(self.emit_options_hash) .finish() } In the Deno environment, the task of transpilation is carried out by the lightning fast SWC compiler, which is built upon the Rust programming language. This innovative compiler offers a significant speed advantage, being approximately 8 times quicker compared to the conventional TSC method. This means that code transformation, a crucial process in preparing code for execution, is accomplished more rapidly and efficiently with SWC. Copy pub fn transpile(&self, options: &EmitOptions) -> Result { let program = (*self.program()).clone(); let source_map = Rc::new(SourceMap::default()); let source_map_config = SourceMapConfig { inline_sources: options.inline_sources, }; let file_name = match ModuleSpecifier::parse(self.specifier()) { Ok(specifier) => FileName::Url(specifier), Err(_) => FileName::Custom(self.specifier().to_string()), }; source_map.new_source_file(file_name, self.text_info().text().to_string()); // needs to align with what's done internally in source map assert_eq!(1, self.text_info().range().start.as_byte_pos().0); // we need the comments to be mutable, so make it single threaded let comments = self.comments().as_single_threaded(); let globals = Globals::new(); crate::swc::common::GLOBALS.set(&globals, || { let top_level_mark = Mark::fresh(Mark::root()); let program = fold_program( program, options, source_map.clone(), &comments, top_level_mark, self.diagnostics(), )?; let mut src_map_buf = vec![]; let mut buf = vec![]; { let mut writer = Box::new(JsWriter::new( source_map.clone(), "\n", &mut buf, Some(&mut src_map_buf), )); writer.set_indent_str(" "); // two spaces let config = crate::swc::codegen::Config { minify: false, ascii_only: false, omit_last_semi: false, target: ES_VERSION, }; let mut emitter = crate::swc::codegen::Emitter { cfg: config, comments: Some(&comments), cm: source_map.clone(), wr: writer, }; program.emit_with(&mut emitter)?; } let mut src = String::from_utf8(buf)?; let mut map: Option = None; { let mut buf = Vec::new(); source_map .build_source_map_with_config(&src_map_buf, None, source_map_config) .to_writer(&mut buf)?; if options.inline_source_map { src.push_str("//# sourceMappingURL=data:application/json;base64,"); base64::encode_config_buf( buf, base64::Config::new(base64::CharacterSet::Standard, true), &mut src, ); } else { map = Some(String::from_utf8(buf)?); } } Ok(TranspiledSource { text: src, source_map: map, }) }) } Once the transpilation process is complete, we end up with two important components at our disposal: * The JavaScript source code * The source hash The JavaScript source code is stored within a file named `.js`, while both the source hash and emit hash are stored in a separate `.js.meta` file. These files are neatly organized in a specific directory within DENO\_DIR. To illustrate, let's consider the initial code file named `helloLog.ts`, which was originally situated at the path `/Users/mayankc/Work/source/denoExamples/helloLog.ts`. Following the transpilation, the resulting files are arranged in the following location: Copy > ls /Users/mayankc/Library/Caches/deno/gen/file/Users/mayankc/Work/source/denoExamples/helloLog.ts.* /Users/mayankc/Library/Caches/deno/gen/file/Users/mayankc/Work/source/denoExamples/helloLog.ts.js /Users/mayankc/Library/Caches/deno/gen/file/Users/mayankc/Work/source/denoExamples/helloLog.ts.meta The code to save the generated JS & meta file is as follows: Copy fn set_emit_code_result( &self, specifier: &ModuleSpecifier, source_hash: u64, code: &str, ) -> Result<(), AnyError> { let meta_filename = self .get_meta_filename(specifier) .ok_or_else(|| anyhow!("Could not get meta filename."))?; let emit_filename = self .get_emit_filename(specifier) .ok_or_else(|| anyhow!("Could not get emit filename."))?; // save the metadata let metadata = EmitMetadata { source_hash: source_hash.to_string(), emit_hash: compute_emit_hash(code.as_bytes(), self.cli_version), }; self .disk_cache .set(&meta_filename, &serde_json::to_vec(&metadata)?)?; // save the emit source self.disk_cache.set(&emit_filename, code.as_bytes())?; Ok(()) } \-- That covered the process of obtaining, transpiling, and caching the code. We are now prepared to move on to the subsequent phase, which involves initiating the loading of the code into the V8 engine. [Previous5.13 File fetching](https://choubey.gitbook.io/internals-of-deno/foundations/4.13-file-fetcher) [Next5.15 Register / compile module](https://choubey.gitbook.io/internals-of-deno/foundations/register-and-instantation) Last updated 1 year ago --- # 7.1 Introduction | The Internals of Deno Data persistence plays a crucial role in building robust Deno applications. Local storage and session storage, two built-in client-side storage options, empower developers to store and retrieve data, enhancing user experience and application functionality. Not limiting to browsers, these are also provided by Deno. Local storage excels in storing data that persists across browser sessions (or Deno restarts), making it ideal for user preferences, settings, and persistent application state. Consider storing font size choices, preferred product categories, or frequently used shipping addresses—all accessible even after closing and reopening the webpage (or restarting Deno application). Session storage, in contrast, offers a temporary home for data relevant only within a single browsing session (or a single run of Deno runtime). It's perfect for shopping cart contents, form data between page redirects, or other transient information that can safely vanish when the tab or window closes (or Deno stops). Deno provides web standard APIs to interact with these storage mechanisms: * **Setting Data:** * `localStorage.setItem(key, value)` stores a key-value pair within local storage. * `sessionStorage.setItem(key, value)` stores a key-value pair within session storage. * **Retrieving Data:** * `localStorage.getItem(key)` retrieves the value associated with a specific key from local storage. * `sessionStorage.getItem(key)` retrieves the value associated with a specific key from session storage. * **Removing Data:** * `localStorage.removeItem(key)` deletes a key-value pair from local storage. * `sessionStorage.removeItem(key)` deletes a key-value pair from session storage. * **Clearing All Data:** * `localStorage.clear()` empties the entire local storage. * `sessionStorage.clear()` empties the entire session storage. Some key considerations when working with these storage mechanisms are: * Both storage mechanisms are restricted to storing strings. To store objects or arrays, serialization techniques like `JSON.stringify()` and `JSON.parse()` are required. * Accessing local and session storage involves asynchronous operations, necessitating the use of Promises or async/await syntax for optimal handling. In the two upcoming sections of this chapter, we'll learn: * 7.2 How local storage works in Deno [7.2 Local storage](https://choubey.gitbook.io/internals-of-deno/chapter-7-local-and-session-storage/7.2-local-storage) * 7.3 How session storage works in Deno [7.3 Session storage](https://choubey.gitbook.io/internals-of-deno/chapter-7-local-and-session-storage/7.3-session-storage) Let's get started with the inner workings of local storage. [Previous7.0 Cover page](https://choubey.gitbook.io/internals-of-deno/chapter-7-local-and-session-storage/7.0-cover-page) [Next7.2 Local storage](https://choubey.gitbook.io/internals-of-deno/chapter-7-local-and-session-storage/7.2-local-storage) Last updated 1 year ago --- # 6.7 Evaluate module | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/evaluate-module#overview) Overview ------------------------------------------------------------------------------------------------------ Module evaluation is akin to executing a program. Deno triggers v8's module evaluation function, prompting V8 to swiftly assess the module's content. Remember, this evaluation process occurs solely on the main module, similar to how the primary program runs in languages like C, C++, or Java. [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/evaluate-module#evaluation-of-hello-v2) Evaluation of hello v2 ---------------------------------------------------------------------------------------------------------------------------------- Below is the important evaluation code from the worker: Copy pub async fn execute_main_module_possibly_with_npm( &mut self, ) -> Result<(), AnyError> { let id = self.worker.preload_main_module(&self.main_module).await?; self.evaluate_module_possibly_with_npm(id).await } The result of the preload\_main\_module function is modules that are created with a return value that holds the ID of the main module. After this, the next task is to assess the main module. The preload module provides the ID of the main module, which is used in the module evaluation process. Sure, here's the rewritten version of the subsection with corrected grammar: As a reminder, mod\_evaluate carries out two actions: 1. It invokes mod\_evaluate. 2. This triggers v8's evaluate function. 3. It establishes a global promise that will either be fulfilled or rejected at some point in the future. 4. It waits asynchronously in run\_event\_loop/poll\_event\_loop until the global promise transitions from pending to fulfilled status. Copy pub async fn evaluate_module( &mut self, id: ModuleId, ) -> Result<(), AnyError> { self.wait_for_inspector_session(); let mut receiver = self.js_runtime.mod_evaluate(id); tokio::select! { // Not using biased mode leads to non-determinism for relatively simple // programs. biased; maybe_result = &mut receiver => { debug!("received module evaluate {:#?}", maybe_result); maybe_result.expect("Module evaluation result not provided.") } event_loop_result = self.run_event_loop(false) => { event_loop_result?; let maybe_result = receiver.await; maybe_result.expect("Module evaluation result not provided.") } } } In the previous chapter, the basic hello world program didn't contain any asynchronous operations. As a result, the poll event loop inside mod\_evaluate completed immediately, since the global promise transitioned to a fulfilled state. After the module evaluation finished, the program progressed through window load, the event loop, window unload, and then concluded. However, in this scenario, we have both asynchronous and synchronous operations: * Console logs use synchronous operations. * getMachineId employs an asynchronous operation. * nanoid relies on synchronous operations. Although there's a combination of operations, they won't consume much time, as they are minimal and straightforward. The only exception arises when a setTimeout function is present, which would introduce a delay in module evaluation, causing a pause. To demonstrate this, let's adjust our code by incorporating the setTimeout function. The updated app code is as follows: Copy import { nanoid } from "npm:nanoid"; import { getMachineId } from "https://deno.land/x/machine_id/mod.ts"; const id = nanoid(); const machineId = await getMachineId(); const homeDir = Deno.env.get("HOME"); function printNumber(input: number) { console.log(input); } function printString(input: string) { console.log(input); } printNumber(1); printString("One"); console.log("Nanoid=", id, ", MachineId=", machineId, ", homeDir=", homeDir); setTimeout(() => { console.log("Timeout occured"); }, 10000); Once all the ops are completed, the program will print: Copy 1 One Nanoid= iDNVv3wRy9w3zNDplgb_h , MachineId= 2D809C23-917A-5052-BB4F-38794CCCE9FA , homeDir= /Users/mayankc Timeout occured As we discussed before, we will now cover the final two operations to demonstrate their implementation in Deno: 1. nanoid 2. console.log By understanding these, we will grasp the process of crossing the bridge, executing ops, and handling the outcomes. Moving forward, we will delve into the mechanics of a synchronous op in the upcoming section. [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/evaluate-module#event-loop) Event loop ---------------------------------------------------------------------------------------------------------- Once you've sent the main module for evaluation, the event loop starts. During this time, the loop continues until the following conditions are met: * There are no pending operations. * There are no pending dynamic imports. * All pending dynamic imports have been evaluated. * All pending top-level modules have been evaluated. When the evaluation of the main module is finished, all operations will also be completed. However, due to a timer set at the end, the program will keep running until the timer runs out. Once this happens, the program will end because there are no more tasks to perform. The remaining steps, such as window loading, the event loop, and window unloading, will follow the same pattern as the "hello world" program. \-- The final part of this chapter will cover the internal workings of synchronous operations (sync ops). With our existing knowledge of sync ops, we are prepared to deepen our understanding in the upcoming section. [Previous6.6 Registration of ops](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.6-registration-of-ops) [Next6.8 Sync OPs](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.8-sync-op) Last updated 1 year ago --- # 5.13 File fetching | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/foundations/4.13-file-fetcher#overview) Overview ----------------------------------------------------------------------------------------------------- In the preceding section, we explored the process of constructing module graphs. While we touched upon the topic of fetching modules, we didn't go deep into the specifics. In this upcoming section, we will elaborate on how Deno retrieves files. Both the main module and its imports are essentially files. These files must be obtained from their respective sources, which can take the form of local locations or remote destinations such as HTTP, HTTPS, data URLs, or even from the NPM package repository. Deno's mechanism for obtaining these files plays a crucial role in how the program functions. Let's delve into the details of how Deno goes about fetching these files, whether they're in close proximity or situated across the vast landscape of the internet. By understanding this process, we can gain insights into the inner workings of Deno's module resolution and loading procedures. [](https://choubey.gitbook.io/internals-of-deno/foundations/4.13-file-fetcher#functionality) Functionality --------------------------------------------------------------------------------------------------------------- In Deno, when it comes to fetching files, you have four main options to choose from. These options are like different paths you can take to get the file you need: 1. **Local Source:** * This is when the file you want is right on your computer's disk. It's stored locally. * For example, you might have a file saved on your computer, and you want to use it in your Deno program. 2. **Remote Source:** * This happens when the file is not on your computer, but it's available on the internet. * You can access remote files through HTTP or HTTPS, just like when you browse websites, or even through NPM, a package manager for JavaScript. 3. **Data URL:** * This is a bit different. Instead of fetching a file from a regular location, you're getting it from a special kind of web standard called a data URL. * It's like embedding the file's content directly into the URL itself. This can be useful for small files or pieces of data. 4. **Cache:** * This is like a storage area where Deno keeps files that it has already fetched before. If you ask for a file that's in the cache, Deno can quickly give it to you without needing to download it again. * Think of it like a saved copy of a file that Deno keeps handy for you. Now, when we look at the code, fetching from the cache is actually a part of the remote fetching process. However, to avoid confusion, we show it separately. This helps to understand that even though the file might already be in the cache, Deno still uses the process it uses for remote files. So, it's like a small detour within the main remote fetching journey. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MPIVWeo_dZDKwjkVkk7%252F-MPIW2VVRMRgneWL-Ni4%252Fdeno%2520file%2520fetching.png%3Falt%3Dmedia%26token%3D6bfd7a52-3309-4093-85dc-24edda77ca74&width=768&dpr=4&quality=100&sign=aa7bdd60&sv=2) The file fetcher function is responsible for obtaining a file when provided with a module specifier. This handy function conceals the intricate workings by consistently delivering the requested file, regardless of where it originates from. By doing so, it shields users from the complexity of file retrieval operations and ensures a seamless experience in accessing the needed files. Here is the source of the main fetch function: Copy pub async fn fetch( &self, specifier: &ModuleSpecifier, permissions: PermissionsContainer, ) -> Result { debug!("FileFetcher::fetch() - specifier: {}", specifier); self.fetch_with_accept(specifier, permissions, None).await } pub async fn fetch_with_accept( &self, specifier: &ModuleSpecifier, permissions: PermissionsContainer, maybe_accept: Option<&str>, ) -> Result { let scheme = get_validated_scheme(specifier)?; permissions.check_specifier(specifier)?; if let Some(file) = self.cache.get(specifier) { Ok(file) } else if scheme == "file" { // we do not in memory cache files, as this would prevent files on the // disk changing effecting things like workers and dynamic imports. fetch_local(specifier) } else if scheme == "data" { self.fetch_data_url(specifier) } else if scheme == "blob" { self.fetch_blob_url(specifier).await } else if !self.allow_remote { Err(custom_error( "NoRemote", format!("A remote specifier was requested: \"{specifier}\", but --no-remote is specified."), )) } else { let result = self .fetch_remote( specifier, permissions, 10, maybe_accept.map(String::from), ) .await; if let Ok(file) = &result { self.cache.insert(specifier.clone(), file.clone()); } result } } The code here is quite straightforward. It retrieves a file, either from your computer or from a distant location, depending on the type of URL the file has. These URLs come in different forms: file://, data:, http://, and https://. When dealing with files fetched from a remote location, the code takes an additional step. It stores these remote files in an internal cache. This cache is like a temporary storage space that helps speed up the process. Fetching files from a remote location can take a lot of time compared to grabbing them from your own computer. So, by keeping a copy of the remote file in the cache, future fetches can be quicker. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/4.13-file-fetcher#local-fetch) Local fetch If the file is situated on the disk (using the file:// protocol), it will be retrieved from your own computer. This process is referred to as "fetch\_local" within Deno. This means that Deno will access the file directly from your local storage instead of over the internet. Copy fn fetch_local(specifier: &ModuleSpecifier) -> Result { let local = specifier.to_file_path().map_err(|_| { uri_error(format!("Invalid file path.\n Specifier: {specifier}")) })?; let bytes = fs::read(local)?; let charset = text_encoding::detect_charset(&bytes).to_string(); let source = get_source_from_bytes(bytes, Some(charset))?; let media_type = MediaType::from_specifier(specifier); Ok(File { maybe_types: None, media_type, source: source.into(), specifier: specifier.clone(), maybe_headers: None, }) } The function fetch\_local operates by reading a file directly from the disk and subsequently providing the content of the file as output. It's important to note that local files obtained through this function are not stored in a cache for future use. Instead, the function retrieves the file's content anew each time it's called, ensuring that the most up-to-date version is always obtained. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/4.13-file-fetcher#remote-fetch) Remote fetch If the file is positioned on a different server, it will be obtained using an HTTP client. This client is responsible for fetching the file either through regular HTTP or the more secure HTTPS. This fetching process is done remotely, meaning it's pulled from a different location, which can be a bit slower and resource-intensive. To tackle the potential inefficiency of remote fetching, Deno employs a caching mechanism. Once the file is fetched for the first time, it is saved in a cache. This way, if the same file is needed again, Deno can retrieve it from the cache rather than fetching it anew from the remote source. This caching strategy helps improve performance and reduces the need for repeated remote fetches. Copy fn fetch_remote( &self, specifier: &ModuleSpecifier, permissions: PermissionsContainer, redirect_limit: i64, maybe_accept: Option, ) -> Pin> + Send>> { debug!("FileFetcher::fetch_remote() - specifier: {}", specifier); if redirect_limit < 0 { return futures::future::err(custom_error("Http", "Too many redirects.")) .boxed(); } if let Err(err) = permissions.check_specifier(specifier) { return futures::future::err(err).boxed(); } if self.should_use_cache(specifier) { match self.fetch_cached(specifier, redirect_limit) { Ok(Some(file)) => { return futures::future::ok(file).boxed(); } Ok(None) => {} Err(err) => { return futures::future::err(err).boxed(); } } } if self.cache_setting == CacheSetting::Only { return futures::future::err(custom_error( "NotCached", format!( "Specifier not found in cache: \"{specifier}\", --cached-only is specified." ), )) .boxed(); } let mut maybe_progress_guard = None; if let Some(pb) = self.progress_bar.as_ref() { maybe_progress_guard = Some(pb.update(specifier.as_str())); } else { log::log!( self.download_log_level, "{} {}", colors::green("Download"), specifier ); } let maybe_etag = self .http_cache .cache_item_key(specifier) .ok() .and_then(|key| self.http_cache.read_metadata(&key).ok().flatten()) .and_then(|metadata| metadata.headers.get("etag").cloned()); let maybe_auth_token = self.auth_tokens.get(specifier); let specifier = specifier.clone(); let client = self.http_client.clone(); let file_fetcher = self.clone(); // A single pass of fetch either yields code or yields a redirect, server // error causes a single retry to avoid crashing hard on intermittent failures. async fn handle_request_or_server_error( retried: &mut bool, specifier: &Url, err_str: String, ) -> Result<(), AnyError> { // Retry once, and bail otherwise. if !*retried { *retried = true; log::debug!("Import '{}' failed: {}. Retrying...", specifier, err_str); tokio::time::sleep(std::time::Duration::from_millis(50)).await; Ok(()) } else { Err(generic_error(format!( "Import '{}' failed: {}", specifier, err_str ))) } } async move { let mut retried = false; let result = loop { let result = match fetch_once( &client, FetchOnceArgs { url: specifier.clone(), maybe_accept: maybe_accept.clone(), maybe_etag: maybe_etag.clone(), maybe_auth_token: maybe_auth_token.clone(), maybe_progress_guard: maybe_progress_guard.as_ref(), }, ) .await? { FetchOnceResult::NotModified => { let file = file_fetcher.fetch_cached(&specifier, 10)?.unwrap(); Ok(file) } FetchOnceResult::Redirect(redirect_url, headers) => { file_fetcher.http_cache.set(&specifier, headers, &[])?; file_fetcher .fetch_remote( &redirect_url, permissions, redirect_limit - 1, maybe_accept, ) .await } FetchOnceResult::Code(bytes, headers) => { file_fetcher .http_cache .set(&specifier, headers.clone(), &bytes)?; let file = file_fetcher.build_remote_file(&specifier, bytes, &headers)?; Ok(file) } FetchOnceResult::RequestError(err) => { handle_request_or_server_error(&mut retried, &specifier, err) .await?; continue; } FetchOnceResult::ServerError(status) => { handle_request_or_server_error( &mut retried, &specifier, status.to_string(), ) .await?; continue; } }; break result; }; drop(maybe_progress_guard); result } .boxed() } Remote fetching refers to a process in Deno where a function calls itself in a loop-like manner due to the possibility of encountering redirects while making HTTP requests. When making an HTTP call, there's a chance that the server might respond with a redirect instruction, asking the client to retrieve the resource from a different URL. To handle this possibility, the function responsible for remote fetching repeats its own execution whenever a redirection occurs. The following steps outline the remote fetching process in more detail: 1. **Check for Redirect Limit:** The function first checks if the redirection limit has been exceeded. If the number of redirections has reached a certain threshold, it will return an error, indicating that the process has encountered too many redirects. 2. **Cache Check:** If the specified resource is already present in the cache, the function returns that cached version of the resource. This helps in saving time and resources by avoiding unnecessary redundant downloads. 3. **HTTP Request:** The function initiates an HTTP request to fetch the desired file from a remote server. 4. **Redirection Handling:** If the server responds with a redirection instruction, indicating that the requested resource has moved to a different location, the function doesn't stop there. Instead, it calls itself again, passing the redirected URL as the new parameter. This recursive behavior ensures that the function can follow multiple redirections until the final resource is reached. 5. **Non-Redirection Response:** If no redirection occurs, the function returns the fetched file as the output. This recursive process of remote fetching is managed by the `fetch_remote` function, which is designed to handle up to ten consecutive redirections. This means that if a series of redirections leads to a final resource, Deno's fetching mechanism can navigate through it effectively. As the remote fetching process takes place, Deno provides feedback to the developer by displaying a familiar message on the console, commonly referred to as the "Download" message. This message signifies that a remote resource is being downloaded and integrated into the local environment. It serves as an indication that the remote fetching process is active and that the necessary content is being retrieved for further usage. `Download` [`https://examples.deno.land/hello-world.ts`](https://examples.deno.land/hello-world.ts) [Previous5.12 Module graphs](https://choubey.gitbook.io/internals-of-deno/foundations/4.12-module-graphs) [Next5.14 Transpile](https://choubey.gitbook.io/internals-of-deno/foundations/4.13-check-and-or-transpilation) Last updated 1 year ago --- # 6.3 Module graph with imports | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/building-module-graph#overview) Overview ------------------------------------------------------------------------------------------------------------ This marks the primary departure from the hello world program discussed in the preceding chapter. Unlike the earlier hello world program that lacked any imports, resulting in a graph containing only one node for the main module, the hello v2 example incorporates imports, inevitably causing the graph to expand. Let's explore the process of graph construction and its subsequent expansion. The process of building the graph unfolds through the following sequential steps: 1. **Inclusion of the Main Module**: As in the previous scenario, the main module assumes its position within the graph. 2. **Fetching the Main Module**: This involves retrieving the main module. 3. **Fetching Associated Modules**: All modules tied to the main module are fetched. 4. **Integration into the Graph**: Each fetched module is integrated into the evolving graph. 5. **Enlistment in Pending Modules**: Alongside integration, these fetched modules are enlisted in the list of pending modules. 6. **Iterating through Pending Modules**: A loop iterates over the pending modules, enabling further processing. 7. **Visiting Pending Modules**: Each pending module is visited in turn. 8. **Parsing the Module**: The module in question undergoes a parsing process. 9. **Traversal of Dependencies**: The module's dependencies are traversed. 10. **Fetching Dependencies**: These dependencies are fetched from their sources. In this revamped hello v2 example, just as in the previous version, the main module is promptly integrated. However, in contrast, the inclusion of dependencies is deferred until a later stage. This strategic alteration leads to the gradual growth of the graph as each dependency is sequentially added. [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/building-module-graph#build-the-module-graph) Build the module graph ---------------------------------------------------------------------------------------------------------------------------------------- In the process of creating a hello v2 application in Deno, the construction of the module graph occurs through several distinct steps. The initial phase involves the recursive retrieval, inclusion, and exploration of all the necessary dependencies. This sequence of actions continues until there are no remaining dependencies left to consider. _**Add module:**_ ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MP0ifp7LPZFOdMaMVhd%252F-MP0jnEEXyc05jbYvQCZ%252Fdeno%2520graph%2520add%2520fc.png%3Falt%3Dmedia%26token%3D2320ffa9-5250-497f-9ac4-87406c269491&width=768&dpr=4&quality=100&sign=c88a88d2&sv=2) _**Visit module:**_ ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MP0lruYSevNz24ktIp-%252F-MP0nDju7llLdkD8qaWP%252Fdeno%2520graph%2520visit%2520fc.png%3Falt%3Dmedia%26token%3Dda4ff718-38f2-4b99-9947-b498285d45fe&width=768&dpr=4&quality=100&sign=2ef6e43&sv=2) Let's walk through the process and understand how the graph is constructed for the hello V2 application: #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/building-module-graph#step-1) Step 1 The initial action involves invoking the 'fill' function using the main module. Module specifier is `file:///Users/mayankc/Work/source/denoExamples/helloLogV2.ts`. #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/building-module-graph#step-2) Step 2 The FileFetcher function is invoked using the primary module specifier. #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/building-module-graph#step-3) Step 3 A Rust future is generated to retrieve the specifier in Deno. This Rust future will be marked as complete once the fetching process finishes. Once generated, the Rust future is included in the list of pending tasks. `pending list = [ file:///Users/mayankc/Work/source/denoExamples/helloLogV2.ts ]` #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/building-module-graph#step-4) Step 4 The 'fill' function is placed within the loop where it patiently awaits the resolution of the upcoming future in the pending list. This singular future happens to be the one associated with fetching the main module. It's important to note that, at this juncture, the graph is composed of just a single node, representing the main module. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252FfezL7wMhYgHUsVq2fAqJ%252Fhellov2_graph.png%3Falt%3Dmedia%26token%3D1467fa78-9370-406e-a097-c28c42d017ef&width=768&dpr=4&quality=100&sign=3ce51705&sv=2) #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/building-module-graph#step-5) Step 5 The Rust future of the main module is resolved in Deno. This means that Deno finishes handling the operations related to the main module. The main module specifier, which is like the address of the main module, is fetched from its source and then stored in a cache for future use. Following this, the "visit" function is triggered for the main module that's now stored in the cache. This function is designed to handle actions related to the cached module. It's important to understand that when the "visit" function is called, it receives the cached module as an input. `visit = file:///Users/mayankc/Work/source/denoExamples/helloLogV2.ts` #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/building-module-graph#step-6) Step 6 When Deno processes a cached module, it carefully examines its contents. In the context of the hello world v2 program, the main module comes with several dependencies. These dependencies are like pieces of a puzzle that need to be gathered before the full picture can be assembled. Deno takes these dependencies and prepares them for retrieval. It sets up a plan for fetching them from their respective sources. `pending list = [ https://deno.land/x/machine_id/mod.ts, npm:nanoid ]` At this particular juncture, the module graph consists of three nodes: ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252FhaUX5wZCZRepXV0nXzYG%252Fhellov2_graph.png%3Falt%3Dmedia%26token%3D38987bc6-37e5-4838-9b33-a06ba4a97b64&width=768&dpr=4&quality=100&sign=588c0ccb&sv=2) #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/building-module-graph#step-8) Step 8 The upcoming Rust future will be resolved as soon as the module is fetched. Inside the fill function, the loop handles the next item that's ready to be processed from the pending list. Moving forward, the npm:nanoid module is the next one in line to be visited and explored. `visit = npm:nanoid` #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/building-module-graph#step-9) Step 9 When Deno processes a cached module, it carefully examines its contents. Within this parsed module, there exist certain dependencies. Deno takes these dependencies and allocates them to a fetch future, which is then included in the list of tasks awaiting execution. In the case of an NPM module, it's worth noting that the dependencies that need to be retrieved will be stored conveniently within the designated "node\_modules" folder. `pending list = [ file:///Users/mayankc/Work/source/denoExamples/node_modules/.deno/nanoid@4.0.2/node_modules/nanoid/url-alphabet/index.js ]` At this point, the module graph has four nodes: ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252FxQtwhngERiV6DV7o2OtC%252Fhellov2_graph.png%3Falt%3Dmedia%26token%3Dbd34172b-e642-45cd-a11c-5f44b3f3d0df&width=768&dpr=4&quality=100&sign=32a148c6&sv=2) #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/building-module-graph#step-10) Step 10 Let's go a bit faster now. The loop within the "fill" function is responsible for handling the upcoming item in the list of tasks to be done. In this case, the next item waiting to be processed is located at a specific web address: "file:///Users/mayankc/Work/source/denoExamples/node\_modules/.deno/nanoid@4.0.2/node\_modules/nanoid/url-alphabet/index.js". `visit = file:///Users/mayankc/Work/source/denoExamples/node_modules/.deno/nanoid@4.0.2/node_modules/nanoid/url-alphabet/index.js` #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/building-module-graph#step-11) Step 11 The cached module undergoes parsing, during which no additional dependencies are detected. As a result, no further nodes are introduced into the module graph. This ensures that the module graph retains its original count of 4 nodes. #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/building-module-graph#step-14) Step 14 `pending list = []` The pending list is now empty, signifying that all tasks have been completed. The loop within the "fill" function has ceased its operation, leading to a break. Consequently, the module graph construction has concluded. As a result, we are presented with the ultimate form of the graph, which can be visualized in the following manner: ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252FafuUCP6hwqjaaa6SvmM5%252Fhellov2_graph.png%3Falt%3Dmedia%26token%3Df669dfd6-0df7-4706-996b-23e31999c32a&width=768&dpr=4&quality=100&sign=39d227e6&sv=2) \-- Except for dynamic imports, Deno processes all static imports first. It fetches these imports, interprets their contents, and then loads them into the V8 engine before the program or application commences its execution. The handling of dynamic imports is deferred until a later stage in the process. [Previous6.2 Hello world program v2](https://choubey.gitbook.io/internals-of-deno/import-and-ops/basic-console-print-with-delay) [Next6.4 Transpile](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.4-transpile) Last updated 1 year ago --- # 7.0 Cover page | The Internals of Deno ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252FT0WJNq39NUzjPDRjxUIo%252Fdeno%2520book%2520chapter%25207.png%3Falt%3Dmedia%26token%3D030bc9f6-36b8-46f0-85f1-6150bc4ae6c1&width=768&dpr=4&quality=100&sign=d223aa22&sv=2) [Previous6.10 What's next](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.11-whats-next) [Next7.1 Introduction](https://choubey.gitbook.io/internals-of-deno/chapter-7-local-and-session-storage/7.1-introduction) Last updated 1 year ago --- # 6.8 Sync OPs | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.8-sync-op#overview) Overview -------------------------------------------------------------------------------------------------- Synchronous operations halt the ongoing execution of the current thread (whether it's the main thread or a worker) until a result becomes available. Many operations in Deno are of the synchronous kind. File system operations are the exception, as they are available in both synchronous and asynchronous forms. This distinction exists because file system operations can consume a significant amount of time. For other scenarios, either synchronous operations are not logical or the operations are so straightforward that implementing them as asynchronous would be excessive. In our example, we will focus on a synchronous operation called `Deno.env.get`. This operation is uncomplicated yet highly instructive for grasping the underlying principles. The purpose of this operation is to take an input and produce a corresponding output. To thoroughly comprehend how this operation functions from start to finish, we will dissect it step by step. Our exploration will encompass both JavaScript and Rust implementations. [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.8-sync-op#js-part) JS part ------------------------------------------------------------------------------------------------ #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.8-sync-op#user-code) User code In the JS space, the application makes a call to get the value of an environment variable named HOME: Copy const v = Deno.env.get('HOME'); #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.8-sync-op#deno.env.get) Deno.env.get The Deno.env.get function is mapped to getEnv: Copy const env = { get: getEnv, toObject() { return ops.op_env(); }, set: setEnv, has(key) { return getEnv(key) !== undefined; }, delete: deleteEnv, }; #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.8-sync-op#getenv) getEnv This immediately calls the JS function getEnv as env.get is mapped to getEnv: Copy function getEnv(key) { return ops.op_get_env(key) ?? undefined; } #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.8-sync-op#ops) OPS The Deno core gives us the global ops object. As we discussed earlier, the ops object is populated using the registerOp API, which is called when Deno starts up. Deno core handles the registration of every supported op. Here's a brief reminder of how the registerOp function looks: Copy Deno.__op__registerOp = function (isAsync, op, opName) { const core = Deno.core; if (isAsync) { if (core.ops[opName] !== undefined) { return; } core.asyncOps[opName] = op; const fn = function (...args) { if (this !== core.ops) { // deno-lint-ignore prefer-primordials throw new Error( "An async stub cannot be separated from Deno.core.ops. Use ???", ); } return core.asyncStub(opName, args); }; fn.name = opName; core.ops[opName] = fn; } else { core.ops[opName] = op; } }; The \_\_op\_\_registerOp function is called during the initialization of context. There are three inputs for each op: * isAsync: True if the op is async * Op: This is the Rust-side ID of the op * OpName: The name of the OP Copy for op_ctx in op_ctxs { if op_ctx.decl.enabled { _ = writeln!( codegen, "Deno.__op__registerOp({}, opFns[{}], \"{}\");", op_ctx.decl.is_async, op_ctx.id, op_ctx.decl.name ); } else { _ = writeln!( codegen, "Deno.__op__unregisterOp({}, \"{}\");", op_ctx.decl.is_async, op_ctx.decl.name ); } } #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.8-sync-op#rust-code) Rust code As op is registered as an external reference, V8 makes an external function call (like C extern functions). Copy fn op_get_env( state: &mut OpState, key: String, ) -> Result, AnyError> { let skip_permission_check = NODE_ENV_VAR_ALLOWLIST.contains(&key); if !skip_permission_check { state.borrow_mut::().check_env(&key)?; } if key.is_empty() { return Err(type_error("Key is an empty string.")); } if key.contains(&['=', '\0'] as &[char]) { return Err(type_error(format!( "Key contains invalid characters: {key:?}" ))); } let r = match env::var(key) { Err(env::VarError::NotPresent) => None, v => Some(v?), }; Ok(r) } \-- That was all about sync ops. It is worth noting that the sync ops are quite easy compared to async ops. Async ops are quite complex to implement. The next revision of this book (4th edition) will have a new chapter, Chapter 8, dedicated to async ops. [Previous6.7 Evaluate module](https://choubey.gitbook.io/internals-of-deno/import-and-ops/evaluate-module) [Next6.9 Debug logs](https://choubey.gitbook.io/internals-of-deno/import-and-ops/6.9-debug-logs) Last updated 1 year ago --- # Afterword | The Internals of Deno We've accomplished it! We've covered the key points of how a program operates in Deno. We dived deep into Deno's architecture, threading model, the connection between Rust and V8, foundations, imports, and ops. We grasped fundamental ideas such as the main worker, permissions, program state, runtime, core, module graphs, file fetching, transpilation, instantiation, evaluation, and more. We've also looked at the internals of Deno's storage APIs. That's the extent of our discussion for now. I hope this book has increased your knowledge. \-- Before you go, if you like to share your review with fellow learners, you can email me at author.mayank.c@gmail.com. I'm open to all feedback, suggestions, etc. Thanks for reading! \-Mayank Choubey [Previous7.4 What's next](https://choubey.gitbook.io/internals-of-deno/chapter-7-local-and-session-storage/7.4-whats-next) Last updated 1 year ago --- # 7.3 Session storage | The Internals of Deno Session storage is just like local storage, except for the lack of data persistence across Deno runtime restarts. The session storage mechanism uses SQLite's in-memory database feature. The SQLite's in-memory database feature is (credit: SQLite documentation): _An SQLite database is normally stored in a single ordinary disk file. However, in certain circumstances, the database might be stored in memory._ _The most common way to force an SQLite database to exist purely in memory is to open the database using the special filename ":memory:". In other words, instead of passing the name of a real disk file into one of the_ [_sqlite3\_open()_](https://www.sqlite.org/c3ref/open.html) _,_ [_sqlite3\_open16()_](https://www.sqlite.org/c3ref/open.html) _, or_ [_sqlite3\_open\_v2()_](https://www.sqlite.org/c3ref/open.html) _functions, pass in the string ":memory:". For example:_ > Copy > > rc = sqlite3_open(":memory:", &db); _When this is done, no disk file is opened. Instead, a new database is created purely in memory. The database ceases to exist as soon as the database connection is closed. Every :memory: database is distinct from every other. So, opening two database connections each with the filename ":memory:" will create two independent in-memory databases._ The transient nature of SQLite's in-memory database perfectly matches the requirements of web standard's session storage. An overview of the session storage architecture is as follows: ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252FjYjFo5Ve4S2d94CCsWPJ%252Fdeno%2520session%2520storage%2520arch.png%3Falt%3Dmedia%26token%3Db12b8bcf-4555-4b43-90f5-c486ceffc9de&width=768&dpr=4&quality=100&sign=ecac6c75&sv=2) The user applications call the session storage APIs provided by Deno. The session storage APIs in JS space are supported by the OPs in the rust space. The session storage APIs in rust, in turn, calls the SQLite APIs, which works with the in-memory database. The primary difference between local storage and session storage is how the database is opened (in-memory or file). Once opened, all the APIs and database queries are the same. As all the APIs are the same, we'll take a detailed look at the opening of the database rather than the APIs and queries that we've already seen in local storage. We'll still look at internals of the one of the API: _length_. [](https://choubey.gitbook.io/internals-of-deno/chapter-7-local-and-session-storage/7.3-session-storage#opening-of-database) Opening of database ----------------------------------------------------------------------------------------------------------------------------------------------------- As we understand that, the session storage mechanisms open the SQLite database using a special path called '**:memory:**'. This indicates SQLite to open a temporary database in memory that should cease to exist as soon as the database connection is closed. You would have noticed that there is a flag called 'persistent' in all the OPs related to storage APIs. This flag determines whether the storage is transient or persistent. Copy let localStorageStorage; function localStorage() { if (!localStorageStorage) { localStorageStorage = createStorage(true); } return localStorageStorage; } let sessionStorageStorage; function sessionStorage() { if (!sessionStorageStorage) { sessionStorageStorage = createStorage(false); } return sessionStorageStorage; } Deno creates both storage the same way, except for their nature: persistent or transient. Let's take a look at the _get\_storage_ function again: Copy fn get_webstorage( state: &mut OpState, persistent: bool, ) -> Result<&Connection, AnyError> { let conn = if persistent { if state.try_borrow::().is_none() { let path = state.try_borrow::().ok_or_else(|| { DomExceptionNotSupportedError::new( "LocalStorage is not supported in this context.", ) })?; std::fs::create_dir_all(&path.0)?; let conn = Connection::open(path.0.join("local_storage"))?; // Enable write-ahead-logging and tweak some other stuff. let initial_pragmas = " -- enable write-ahead-logging mode PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL; PRAGMA temp_store=memory; PRAGMA page_size=4096; PRAGMA mmap_size=6000000; PRAGMA optimize; "; conn.execute_batch(initial_pragmas)?; conn.set_prepared_statement_cache_capacity(128); { let mut stmt = conn.prepare_cached( "CREATE TABLE IF NOT EXISTS data (key VARCHAR UNIQUE, value VARCHAR)", )?; stmt.execute(params![])?; } state.put(LocalStorage(conn)); } &state.borrow::().0 } else { if state.try_borrow::().is_none() { let conn = Connection::open_in_memory()?; { let mut stmt = conn.prepare_cached( "CREATE TABLE data (key VARCHAR UNIQUE, value VARCHAR)", )?; stmt.execute(params![])?; } state.put(SessionStorage(conn)); } &state.borrow::().0 }; Ok(conn) } The relevant part of this function is the bottom 'else' part where, Deno uses rusqlite's _open\_in\_memory_ API to create an in-memory database. This makes it interesting to confirm how rusqlite implements the _open\_in\_memory_ API. The following code is from rusqlite: Copy #[inline] pub fn open_in_memory() -> Result { let flags = OpenFlags::default(); Connection::open_in_memory_with_flags(flags) } #[inline] pub fn open_in_memory_with_flags(flags: OpenFlags) -> Result { Connection::open_with_flags(":memory:", flags) } As we can see that, the rusqlite API opens the SQLite database connection using a special path, ':memory:". The following diagram makes the distinction clearer: ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252FCAozqcNE4cZuSmqD2d39%252Fdeno%2520all%2520storage%2520arch.drawio.png%3Falt%3Dmedia%26token%3D651927c0-bb7e-4560-b76b-14a65d9f9d26&width=768&dpr=4&quality=100&sign=13977e93&sv=2) Once the database connection is opened, all the subsequent APIs and queries are the same. We'll still take a look at one of the API we haven't seen earlier: _length_. [](https://choubey.gitbook.io/internals-of-deno/chapter-7-local-and-session-storage/7.3-session-storage#getting-number-of-items-in-database) Getting number of items in database ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- The length read-only property of the Storage interface returns the number of data items stored in a given Storage object. **JS space** Copy get length() { webidl.assertBranded(this, StoragePrototype); return op_webstorage_length(this[_persistent]); } **Rust space** Copy #[op2(fast)] pub fn op_webstorage_length( state: &mut OpState, persistent: bool, ) -> Result { let conn = get_webstorage(state, persistent)?; let mut stmt = conn.prepare_cached("SELECT COUNT(*) FROM data")?; let length: u32 = stmt.query_row(params![], |row| row.get(0))?; Ok(length) } The query executed to fetch the number of records is COUNT(\*). This query works on a SQLite database whether it is in-memory or on a disk. * * * That's all about the internals of session storage. [Previous7.2 Local storage](https://choubey.gitbook.io/internals-of-deno/chapter-7-local-and-session-storage/7.2-local-storage) [Next7.4 What's next](https://choubey.gitbook.io/internals-of-deno/chapter-7-local-and-session-storage/7.4-whats-next) Last updated 1 year ago --- # 5.8 JS Runtime | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/foundations/jsruntime#overview) Overview --------------------------------------------------------------------------------------------- The JS runtime serves as the central handler for executing JavaScript code within Deno. Instead of being a standalone JavaScript engine, it acts as a bridge between Deno and the v8 engine. The JS runtime leverages a component called "rusty\_v8" to establish communication with the v8 engine. This communication facilitates various operations, ranging from loading and initializing modules to executing code and managing the well-known event loop. A substantial portion of code in the JS runtime is dedicated to interfacing with v8. In the context of any user-level program, the JS runtime offers a "future" that reaches completion under three primary conditions: 1. Encountering an error during execution. 2. Successful evaluation of all modules. 3. Completion of all pending operations. Should none of the above scenarios occur, the future remains in a continuous polling state within the event loop. To illustrate, consider our "hello world" example where the future concludes quickly due to the absence of waiting operations. The JS runtime performs many tasks, making it intricate. Here, we'll provide a high-level overview of its functionality. As we explore script execution, we'll gain a deeper understanding of its inner workings. The JS runtime is a vital hub where various crucial actions occur, making it a critical component in Deno's ecosystem. [](https://choubey.gitbook.io/internals-of-deno/foundations/jsruntime#initialization-of-js-runtime) Initialization of JS runtime ------------------------------------------------------------------------------------------------------------------------------------- The process of initializing the JavaScript runtime occurs through several sequential steps. Let's take a closer look at these steps to gain a better understanding. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MOwkPpkAl_6Wltb1ZsN%252F-MOwl5Qye9SQMRRmG08g%252Fdeno%2520js%2520runtime.png%3Falt%3Dmedia%26token%3D275c7eb3-bf52-45ca-a8e3-77d86d781cdf&width=768&dpr=4&quality=100&sign=31832529&sv=2) ### [](https://choubey.gitbook.io/internals-of-deno/foundations/jsruntime#v8_init) v8\_init The initial step involves setting up v8, the core engine of Deno. * Begin by generating a fresh default platform within the v8 framework. * Proceed to initialize this platform. * Finally, kickstart the v8 engine itself. Copy pub fn v8_init() { let platform = v8::new_default_platform(0, false).make_shared(); v8::V8::initialize_platform(platform); v8::V8::initialize(); } ### [](https://choubey.gitbook.io/internals-of-deno/foundations/jsruntime#global-context) Global context In the world of v8, we encounter the concept of "contexts." These contexts pertain to scripts or modules and play an important role. Yet, alongside them, there exists what we call the "global context." This special global context comes into existence as soon as v8 is initialized for operation. It serves a vital purpose in various aspects, including module loading, instantiation, and more. The process of establishing this global context unfolds through a series of steps, with the initial stage involving the creation of what are known as "isolates." These isolates are integral components originating from v8 itself. If you're curious to delve deeper into the specifics of isolates, you can find valuable insights on the website v8.dev. Essentially, isolates are set up and initialized at this point in our discussion. This marks the foundation of the broader context in which code and modules operate within Deno. Copy let refs = bindings::external_references(&op_ctxs, &additional_references); // V8 takes ownership of external_references. let refs: &'static v8::ExternalReferences = Box::leak(Box::new(refs)); let mut isolate = if will_snapshot { snapshot_util::create_snapshot_creator( refs, options.startup_snapshot.take(), ) } else { let mut params = options .create_params .take() .unwrap_or_default() .embedder_wrapper_type_info_offsets( V8_WRAPPER_TYPE_INDEX, V8_WRAPPER_OBJECT_INDEX, ) .external_references(&**refs); if let Some(snapshot) = options.startup_snapshot.take() { params = match snapshot { Snapshot::Static(data) => params.snapshot_blob(data), Snapshot::JustCreated(data) => params.snapshot_blob(data), Snapshot::Boxed(data) => params.snapshot_blob(data), }; } v8::Isolate::new(params) }; for op_ctx in op_ctxs.iter_mut() { op_ctx.isolate = isolate.as_mut() as *mut Isolate; } context_state.borrow_mut().op_ctxs = op_ctxs; isolate.set_capture_stack_trace_for_uncaught_exceptions(true, 10); isolate.set_promise_reject_callback(bindings::promise_reject_callback); isolate.set_host_initialize_import_meta_object_callback( bindings::host_initialize_import_meta_object_callback, ); isolate.set_host_import_module_dynamically_callback( bindings::host_import_module_dynamically_callback, ); isolate.set_wasm_async_resolve_promise_callback( bindings::wasm_async_resolve_promise_callback, ); let (main_context, snapshotted_data) = { let scope = &mut v8::HandleScope::new(&mut isolate); let context = create_context( scope, &global_template_middlewares, &global_object_middlewares, ); // Get module map data from the snapshot let snapshotted_data = if init_mode == InitMode::FromSnapshot { Some(snapshot_util::get_snapshotted_data(scope, context)) } else { None }; (v8::Global::new(scope, context), snapshotted_data) }; ### [](https://choubey.gitbook.io/internals-of-deno/foundations/jsruntime#state) State The JsRuntime requires a complex state that it manages using v8's slots. This state is composed of several different elements, including: Copy isolate.set_slot(Rc::new(RefCell::new(JsRuntimeState { global_context: Some(global_context), pending_promise_exceptions: HashMap::new(), pending_dyn_mod_evaluate: HashMap::new(), pending_mod_evaluate: None, shared_ab: None, js_recv_cb: None, js_macrotask_cb: None, js_error_create_fn, shared: SharedQueue::new(RECOMMENDED_SIZE), pending_ops: FuturesUnordered::new(), pending_unref_ops: FuturesUnordered::new(), op_state: Rc::new(RefCell::new(op_state)), have_unpolled_ops: Cell::new(false), modules: Modules::new(), loader, dyn_import_map: HashMap::new(), preparing_dyn_imports: FuturesUnordered::new(), pending_dyn_imports: FuturesUnordered::new(), waker: AtomicWaker::new(), }))); Deno operates by maintaining multiple states to ensure its proper functioning. As we continue to explore running our code, we will discover some of these states. This concludes our discussion on initialization. Now, let's examine how the JsRuntime executes programs. For now, we'll maintain a high-level perspective to understand the overall process. [](https://choubey.gitbook.io/internals-of-deno/foundations/jsruntime#execute) Execute ------------------------------------------------------------------------------------------- In Deno, the JsRuntime offers a handy tool called "execute\_script()" for carrying out script execution within the V8 engine. It's crucial to keep in mind that Deno serves as a runtime environment, not merely a JavaScript execution engine. The actual running of scripts takes place within the V8 engine. The purpose of the "execute\_script()" function is to handle the execution of conventional JavaScript code, which refers to code without ES modules. If your code follows the traditional structure, this function is your go-to. However, if you're dealing with code organized into ES modules, fret not! Deno provides distinct functions tailored for such module-based JavaScript. An interesting point to note is that the "execute\_script()" function operates within the global context. This means that any code executed using this function will interact with the broader environment and variables. This global context approach might impact how your code behaves and communicates with other parts of your program. In essence, Deno's JsRuntime simplifies script execution through the "execute\_script()" function, emphasizing compatibility with traditional JavaScript while accommodating module-based code through separate functions. Understanding the distinction between these execution methods can significantly enhance your understanding of the internals of Deno. Copy pub fn execute_script( &self, isolate: &mut v8::Isolate, name: &'static str, source_code: ModuleCode, ) -> Result, Error> { let scope = &mut self.0.handle_scope(isolate); let source = Self::string_from_code(scope, &source_code).unwrap(); debug_assert!(name.is_ascii()); let name = v8::String::new_external_onebyte_static(scope, name.as_bytes()).unwrap(); let origin = bindings::script_origin(scope, name); let tc_scope = &mut v8::TryCatch::new(scope); let script = match v8::Script::compile(tc_scope, source, Some(&origin)) { Some(script) => script, None => { let exception = tc_scope.exception().unwrap(); return exception_to_err_result(tc_scope, exception, false); } }; match script.run(tc_scope) { Some(value) => { let value_handle = v8::Global::new(tc_scope, value); Ok(value_handle) } None => { assert!(tc_scope.has_caught()); let exception = tc_scope.exception().unwrap(); exception_to_err_result(tc_scope, exception, false) } } } The execution of a script in Deno involves several significant steps that work together to make things happen: * **Obtaining the Global Context**: At the outset, Deno fetches the global context, which provides the foundational environment for your script's execution. * **Initializing the Script or File**: Next, Deno takes the necessary actions to set up and prepare the script or file for execution, ensuring that everything is in place. * **Setting Up v8's TryCatch Error Handler**: Deno allocates v8's TryCatch error handler, a tool that helps manage and capture errors during the execution process, enhancing the reliability of the script. * **Compiling the Script**: The script is then compiled, which involves transforming the human-readable code into a form that the computer can understand and execute. * **Running the Script**: With all the groundwork laid, Deno finally runs the compiled script, bringing it to life and allowing it to perform its intended tasks. This sequence of steps showcases the intricate process Deno follows to take your code from its initial form to actual execution. Each step plays a crucial role in ensuring that your script runs smoothly and successfully carries out its desired operations. [](https://choubey.gitbook.io/internals-of-deno/foundations/jsruntime#es-modules) ES Modules ------------------------------------------------------------------------------------------------- The JS runtime plays a vital role in managing ES modules, which have unique requirements compared to traditional JS code. The JS runtime performs this task through various essential functions. The JS runtime has several functions dedicated to supporting ES modules. Let's take a brief look at some of these key functions: Function Use new\_es\_module * Called during module loading or dynamic import loading * Creates a module * Compiles module in v8 * Returns compiled module id instantiate\_module * Needs module id (this comes from new\_es\_module) * Instantiates an ES module with module id mod\_evaluate * Evaluate an instantiated ES module * This is equivalent to running a JS code dyn\_mod\_evaluate * This is for dynamic imports (similar functionality as dyn\_mod\_evaluate) load\_main\_module * Asynchronously load the main module and all of its dependencies * This is a recursive procedure load\_side\_module * Used to load some module that is not part of the main module and it's dependencies One crucial role of the JS runtime is to facilitate and manage ES modules, which have distinct requirements compared to traditional JS code. The JS runtime accomplishes this through a range of significant functions. The JS runtime has various functions that support ES modules. Let's explore a few of these key functions. [](https://choubey.gitbook.io/internals-of-deno/foundations/jsruntime#event-loop) Event loop ------------------------------------------------------------------------------------------------- The JavaScript runtime also operates the renowned event loop, which has gained prominence within the universe of JavaScript, particularly through Node.js. This event loop can be considered as a kind of enchanting cycle that remains active until a program has completed its tasks. In essence, the event loop represents a path to the future, persistently operating as long as there are tasks yet to be accomplished. Its fundamental code structure can be depicted as follows: Copy pub async fn run_event_loop( &mut self, wait_for_inspector: bool, ) -> Result<(), Error> { poll_fn(|cx| self.poll_event_loop(cx, wait_for_inspector)).await } pub fn poll_event_loop( &mut self, cx: &mut Context, wait_for_inspector: bool, ) -> Poll> { let has_inspector: bool; { let state = self.inner.state.borrow(); has_inspector = state.inspector.is_some(); state.op_state.borrow().waker.register(cx.waker()); } if has_inspector { // We poll the inspector first. let _ = self.inspector().borrow().poll_sessions(Some(cx)).unwrap(); } self.pump_v8_message_loop()?; // Dynamic module loading - ie. modules loaded using "import()" { // Run in a loop so that dynamic imports that only depend on another // dynamic import can be resolved in this event loop iteration. // // For example, a dynamically imported module like the following can be // immediately resolved after `dependency.ts` is fully evaluated, but it // wouldn't if not for this loop. // // await delay(1000); // await import("./dependency.ts"); // console.log("test") // // These dynamic import dependencies can be cross-realm: // // await delay(1000); // await new ShadowRealm().importValue("./dependency.js", "default"); // loop { let mut has_evaluated = false; let state = self.inner.state.borrow(); if state.known_realms.len() == 1 { drop(state); // Try and resolve as many dynamic imports in each realm as possible // before moving to the next. let realm = self.inner.main_realm.as_ref().unwrap(); loop { let poll_imports = realm.prepare_dyn_imports(&mut self.inner.v8_isolate, cx)?; assert!(poll_imports.is_ready()); let poll_imports = realm.poll_dyn_imports(&mut self.inner.v8_isolate, cx)?; assert!(poll_imports.is_ready()); if realm.evaluate_dyn_imports(&mut self.inner.v8_isolate) { has_evaluated = true; } else { break; } } } else { // TODO(bartlomieju|mmastrac): Remove cloning in the runtime loop let realms = state.known_realms.clone(); drop(state); for inner_realm in realms { let realm = JsRealm::new(inner_realm); // Try and resolve as many dynamic imports in each realm as possible // before moving to the next. loop { let poll_imports = realm.prepare_dyn_imports(&mut self.inner.v8_isolate, cx)?; assert!(poll_imports.is_ready()); let poll_imports = realm.poll_dyn_imports(&mut self.inner.v8_isolate, cx)?; assert!(poll_imports.is_ready()); if realm.evaluate_dyn_imports(&mut self.inner.v8_isolate) { has_evaluated = true; } else { break; } } } } if !has_evaluated { break; } } } // Resolve async ops, run all next tick callbacks and macrotasks callbacks // and only then check for any promise exceptions (`unhandledrejection` // handlers are run in macrotasks callbacks so we need to let them run // first). let dispatched_ops = self.do_js_event_loop_tick(cx)?; self.check_promise_rejections()?; // Event loop middlewares let mut maybe_scheduling = false; { let op_state = self.inner.state.borrow().op_state.clone(); for f in &self.event_loop_middlewares { if f(op_state.clone(), cx) { maybe_scheduling = true; } } } // Top level module { let state = self.inner.state.borrow(); if state.known_realms.len() == 1 { drop(state); let realm = self.inner.main_realm.as_ref().unwrap(); realm.evaluate_pending_module(&mut self.inner.v8_isolate); } else { // TODO(bartlomieju|mmastrac): Remove cloning in the runtime loop let realms = state.known_realms.clone(); drop(state); for inner_realm in realms { let realm = JsRealm::new(inner_realm); realm.evaluate_pending_module(&mut self.inner.v8_isolate); } } } let pending_state = self.event_loop_pending_state(); if !pending_state.is_pending() && !maybe_scheduling { if has_inspector { let inspector = self.inspector(); let has_active_sessions = inspector.borrow().has_active_sessions(); let has_blocking_sessions = inspector.borrow().has_blocking_sessions(); if wait_for_inspector && has_active_sessions { // If there are no blocking sessions (eg. REPL) we can now notify // debugger that the program has finished running and we're ready // to exit the process once debugger disconnects. if !has_blocking_sessions { let context = self.main_context(); let scope = &mut self.handle_scope(); inspector.borrow_mut().context_destroyed(scope, context); println!("Program finished. Waiting for inspector to disconnect to exit the process..."); } return Poll::Pending; } } return Poll::Ready(Ok(())); } let state = self.inner.state.borrow(); // Check if more async ops have been dispatched // during this turn of event loop. // If there are any pending background tasks, we also wake the runtime to // make sure we don't miss them. // TODO(andreubotella) The event loop will spin as long as there are pending // background tasks. We should look into having V8 notify us when a // background task is done. if pending_state.has_pending_background_tasks || pending_state.has_tick_scheduled || maybe_scheduling { state.op_state.borrow().waker.wake(); } // If ops were dispatched we may have progress on pending modules that we should re-check if (pending_state.has_pending_module_evaluation || pending_state.has_pending_dyn_module_evaluation) && dispatched_ops { state.op_state.borrow().waker.wake(); } drop(state); if pending_state.has_pending_module_evaluation { if pending_state.has_pending_refed_ops || pending_state.has_pending_dyn_imports || pending_state.has_pending_dyn_module_evaluation || pending_state.has_pending_background_tasks || pending_state.has_tick_scheduled || maybe_scheduling { // pass, will be polled again } else { let known_realms = &self.inner.state.borrow().known_realms; return Poll::Ready(Err( find_and_report_stalled_level_await_in_any_realm( &mut self.inner.v8_isolate, known_realms, ), )); } } if pending_state.has_pending_dyn_module_evaluation { if pending_state.has_pending_refed_ops || pending_state.has_pending_dyn_imports || pending_state.has_pending_background_tasks || pending_state.has_tick_scheduled { // pass, will be polled again } else if self.inner.state.borrow().dyn_module_evaluate_idle_counter >= 1 { let known_realms = &self.inner.state.borrow().known_realms; return Poll::Ready(Err( find_and_report_stalled_level_await_in_any_realm( &mut self.inner.v8_isolate, known_realms, ), )); } else { let mut state = self.inner.state.borrow_mut(); // Delay the above error by one spin of the event loop. A dynamic import // evaluation may complete during this, in which case the counter will // reset. state.dyn_module_evaluate_idle_counter += 1; state.op_state.borrow().waker.wake(); } } Poll::Pending } fn event_loop_pending_state(&mut self) -> EventLoopPendingState { let mut scope = v8::HandleScope::new(self.inner.v8_isolate.as_mut()); EventLoopPendingState::new(&mut scope, &mut self.inner.state.borrow_mut()) } } Understanding the event loop can be challenging without exploring the code. We'll gain a better understanding of the event loop as we go through the main module evaluation process in the "Hello World" program. For now, let's get a high-level overview of the event loop's role. The event loop mechanism can be best understood by examining the `poll_event_loop` function, which runs asynchronously until certain conditions are met: 1. An error is encountered, or 2. All tasks are completed, implying that there's nothing left to be done. This involves checking for various scenarios: * No pending operations are waiting to be executed. * No dynamic imports are in the queue, waiting for their turn. * All pending dynamic imports have been successfully evaluated. * All modules in the evaluation queue have been processed. * No active inspector sessions are ongoing. By adhering to these conditions, the event loop ensures the orderly execution of tasks within the Deno runtime environment. As we proceed, we will dive into the intricacies of these concepts, shedding light on the orchestration of asynchronous operations and module evaluations, ultimately resulting in a well-functioning Deno application. If there's nothing to be done, the event loop comes to a halt. In our illustration, the event loop would conclude fairly quickly. However, in real-world scenarios, this loop would hardly ever reach its end. Take, for instance, a web server—it perpetually remains waiting to receive new connections. Applications in production would generally only terminate when an error surfaces. Let's look into some higher-level tasks that the poll event loop undertakes: 1. **Polling Pending Operations:** It scans for operations that are pending, waiting for them to become available. 2. **Verifying Responses from Asynchronous Operations:** The loop checks the responses from asynchronous operations, making sure they are ready. 3. **Draining Macrotasks:** It processes larger-scale tasks, often known as "macrotasks," from the queue. 4. **Handling Promise Exceptions:** The loop takes a look at promise-based operations to catch and handle any exceptions that might have occurred. 5. **Prepping Dynamic Module Imports:** It readies the environment for importing dynamic modules, which are modules that can be loaded on demand. 6. **Polling Dynamic Imports:** The loop keeps an eye out for dynamically imported modules, ensuring they're fetched when necessary. 7. **Evaluating Dynamic Module Imports:** It assesses and evaluates the dynamically imported modules. 8. **Checking Promise Exceptions Again:** The loop revisits promise-based operations to double-check for exceptions. 9. **Assessing Pending Top-Level Modules:** It evaluates and processes any pending top-level modules that are waiting to be executed. 10. **Verifying for Exceptions:** The loop examines the code for any potential exceptions before proceeding. As we progress through our program in this and the following chapters, we will explore various aspects of the event loop. Understanding the event loop's intricacies can be difficult without a practical context or a concrete example. [](https://choubey.gitbook.io/internals-of-deno/foundations/jsruntime#next-steps) Next steps ------------------------------------------------------------------------------------------------- That concludes our discussion on the JS runtime. So far, we've explored the initialization process, which primarily focused on preparation. Now, we'll move on to the exciting phase of executing user-level code in our simple "hello world" program. Our focus shifts to the next segment, which covers the execution of the main module. In the following passages, we'll delve into the details of loading and running our code, uncovering the steps that bring our program to life. [Previous5.7 Main Worker](https://choubey.gitbook.io/internals-of-deno/foundations/mainworker) [Next5.9 Run main module](https://choubey.gitbook.io/internals-of-deno/foundations/execute_module) Last updated 1 year ago --- # 5.12 Module graphs | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/foundations/4.12-module-graphs#overview) Overview ------------------------------------------------------------------------------------------------------ Module graphs play a crucial role in Deno's underlying framework. To put it simply, a module graph is like a building block puzzle, connecting different pieces together to cover all the parts needed starting from the main module. The creation of this module graph happens step by step, like following a recipe to bake a cake. Let's take a closer look at each of these steps, so we can really grasp how it all comes together. At its core, a module graph is like a map that shows how different modules depend on each other. Imagine you're planning a trip, and you need to figure out which cities you'll pass through to reach your destination. Similarly, Deno's module graph helps the system understand the journey of dependencies from the starting point, which is the root module. [](https://choubey.gitbook.io/internals-of-deno/foundations/4.12-module-graphs#module-graphs) Module graphs ---------------------------------------------------------------------------------------------------------------- In Deno, every module utilized within the application follows a process governed by module graphs. These graphs serve to retrieve, interpret, and uphold modules. The concept is quite straightforward: modules are incorporated into the graph based on their unique specifiers, and subsequently, the graph's nodes, representing modules, are traversed. For an in-depth comprehension of these graphs, one can refer to the standard graph data structure. In our present illustration, we're dealing with a principal module that doesn't involve any additional imports. Thus, our graph is anticipated to comprise just a solitary node. However, our exploration into modules doesn't conclude here; the subsequent chapter goes into programs involving multiple modules. This graph-building journey involves two fundamental phases: 1. Introduction of the main module: * Retrieval of the module * Integration into the graph * Addition to the list of modules pending processing 2. Iteration through pending modules: * Examination of the pending module * Parsing of the module * Iteration through its dependencies * Retrieval of these dependencies ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MOyu6dNQpsMzrkdT1r3%252F-MOyvFUd5bv1vm-itUR2%252Fdeno%2520build%2520graph.png%3Falt%3Dmedia%26token%3Dae20a2e8-252e-4f10-823b-e1c30d9b0fc1&width=768&dpr=4&quality=100&sign=9a3fa4f5&sv=2) In the beginning, the primary module is obtained and included in the graph. Following that, the graph is explored, beginning with the main module. Exploring a module includes examining its contents and retrieving any dependencies it has. This process occurs in a repeating pattern. First, we'll explore the basic operations, including add, fetch, and visit. Then, we'll use a concrete example to illustrate how these functions work, providing a clearer understanding of their inner mechanics. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/4.12-module-graphs#fill) Fill GraphBuilder.fill() serves the purpose of incorporating both the main module (known as the root module) and any accompanying dependencies into the structure. In real-world scenarios, the number of dependencies could be substantial, leading to the creation of complex graphs. Especially the cases when applications use dependencies from NPM, which in-turn, depends on tens of hundreds of modules/packages. This complexity emerges due to the interconnections among various components. On the other hand, when we examine our uncomplicated "hello world" illustration, the graph assumes a straightforward form, featuring only a singular node. It's worth noting that as the application's scale and complexity increase, so does the intricacy of the graph, showcasing the interdependency of modules and components. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MP0ifp7LPZFOdMaMVhd%252F-MP0jnEEXyc05jbYvQCZ%252Fdeno%2520graph%2520add%2520fc.png%3Falt%3Dmedia%26token%3D2320ffa9-5250-497f-9ac4-87406c269491&width=768&dpr=4&quality=100&sign=c88a88d2&sv=2) Here is the code of the fill function: Copy async fn fill( &mut self, roots: Vec, imports: Vec, ) { let roots = roots .into_iter() .filter(|r| !self.graph.roots.contains(r)) .collect::>(); let imports = imports .into_iter() .filter(|r| !self.graph.imports.contains_key(&r.referrer)) .collect::>(); self.graph.roots.extend(roots.clone()); for root in roots { self.load(&root, None, self.in_dynamic_branch, None); } // Process any imports that are being added to the graph. for referrer_imports in imports { let referrer = referrer_imports.referrer; let imports = referrer_imports.imports; let graph_import = GraphImport::new(&referrer, imports, self.resolver); for dep in graph_import.dependencies.values() { if let Resolution::Ok(resolved) = &dep.maybe_type { self.load( &resolved.specifier, Some(&resolved.range), self.in_dynamic_branch, None, ); } } self.graph.imports.insert(referrer, graph_import); } loop { let specifier = match self.pending.next().await { Some(PendingInfo { specifier, maybe_range, result, }) => match result { Ok(Some(response)) => { let assert_types = self.pending_specifiers.remove(&specifier).unwrap(); for maybe_assert_type in assert_types { self.visit( &specifier, &response, maybe_assert_type, maybe_range.clone(), ) } Some(specifier) } Ok(None) => { self.graph.module_slots.insert( specifier.clone(), ModuleSlot::Err(ModuleGraphError::ModuleError( ModuleError::Missing(specifier.clone(), maybe_range), )), ); Some(specifier) } Err(err) => { self.graph.module_slots.insert( specifier.clone(), ModuleSlot::Err(ModuleGraphError::ModuleError( ModuleError::LoadingErr( specifier.clone(), maybe_range, Arc::new(err), ), )), ); Some(specifier) } }, None => None, }; if let (Some(specifier), Some(reporter)) = (specifier, self.reporter) { let modules_total = self.graph.module_slots.len(); let modules_done = modules_total - self.pending.len(); reporter.on_load(&specifier, modules_done, modules_total); } if self.pending.is_empty() { // Start visiting queued up dynamic branches. We do this in a separate // pass after all static dependencies have been visited because: // - If a module is both statically and dynamically imported, we want // the static import to take precedence and only load it with // `is_dynamic: false`. // - It's more convenient for tracking whether or not we are currently // visiting a dynamic branch. if !self.in_dynamic_branch { self.in_dynamic_branch = true; for (specifier, (range, maybe_assert_type)) in std::mem::take(&mut self.dynamic_branches) { if !self.graph.module_slots.contains_key(&specifier) { self.load(&specifier, Some(&range), true, maybe_assert_type); } } } else { break; } } } // Enrich with cache info from the loader for slot in self.graph.module_slots.values_mut() { if let ModuleSlot::Module(ref mut module) = slot { match module { Module::Json(module) => { module.maybe_cache_info = self.loader.get_cache_info(&module.specifier); } Module::Esm(module) => { module.maybe_cache_info = self.loader.get_cache_info(&module.specifier); } Module::External(_) | Module::Npm(_) | Module::Node(_) => {} } } } // Now resolve any npm package requirements NpmSpecifierResolver::fill_builder(self).await; } The role of the `fill()` function extends beyond simply adding the root or main module. This versatile function serves to incorporate various ES modules or NPM modules into the system. While the primary module establishes the graph's foundation, the `fill()` function accommodates any general ES module by integrating it as a dependency within the graph's structure. A sequence of steps defines the `fill()` function's operation: 1. Commence by fetching/loading the root module. 2. Proceed through a loop until the pending list is devoid of items. 3. Process each pending item during traversal. The loop contained within the `fill()` function persists until the pending list has been emptied. An absence of items within the pending list signifies the comprehensive recursive processing and integration of all dependencies into the graph. Consequently, the `fill()` function's conclusion marks the completion of the graph, encompassing all the incorporated modules. This meticulous process ensures that the graph embodies a coherent and interconnected representation of the modules, ready for utilization. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/4.12-module-graphs#visit) Visit The "visit" function is a fundamental building block in constructing the module graph within Deno. Compared to the "fill" function, "visit" is more extensive in its role. Its purpose is to handle modules as they are encountered. This function is invoked for each individual module, a process initiated by the "add" function. This cycle of calling "visit" continues until the pending list of modules becomes empty. The primary task of the "visit" function is to manage the processing of fetched modules. As modules are fetched and introduced into the workflow, the "visit" function takes charge of orchestrating their integration into the existing graph. This involves understanding their dependencies, connections to other modules, and their position within the broader context of the application. By effectively managing these aspects, the "visit" function contributes significantly to the overall structure and coherence of the module graph. To provide a visual representation of how the "visit" function operates, a flowchart has been designed. This flowchart visually depicts the step-by-step process through which the "visit" function manages modules. This graphical representation offers insight into the sequential progression of actions undertaken by the "visit" function as it processes modules and integrates them into the module graph. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MP0lruYSevNz24ktIp-%252F-MP0nDju7llLdkD8qaWP%252Fdeno%2520graph%2520visit%2520fc.png%3Falt%3Dmedia%26token%3Dda4ff718-38f2-4b99-9947-b498285d45fe&width=768&dpr=4&quality=100&sign=2ef6e43&sv=2) Let's take a closer look at the process step by step: * It begins with visiting a module that has been fetched. * The module is then parsed to understand its contents. * Dependencies are examined one by one in an iterative manner. This is where things start to become recursive. During the visiting process, all dependencies are fetched. When a dependency is fetched, it's marked for future consideration and placed in a pending list. The loop within the 'fill' function will keep running as long as there are items in the pending list. This loop ensures that all pending items are processed. This iterative process continues until every node in the interconnected graph has been visited. In other words, the loop in the 'fill' function only concludes when all nodes in the graph have been covered. Let's take a look at the source code for the 'visit' function: Copy fn visit( &mut self, requested_specifier: &ModuleSpecifier, response: &LoadResponse, maybe_assert_type: Option, maybe_referrer: Option, ) { let (specifier, module_slot) = match response { LoadResponse::External { specifier } => { self.check_specifier(requested_specifier, specifier); let module_slot = ModuleSlot::Module(Module::External(ExternalModule { specifier: specifier.clone(), })); (specifier, module_slot) } LoadResponse::Module { specifier, content, maybe_headers, } => { self.check_specifier(requested_specifier, specifier); ( specifier, self.visit_module( specifier, maybe_headers.as_ref(), content.clone(), maybe_assert_type, maybe_referrer, ), ) } }; self .graph .module_slots .insert(specifier.clone(), module_slot); } /// Visit a module, parsing it and resolving any dependencies. fn visit_module( &mut self, specifier: &ModuleSpecifier, maybe_headers: Option<&HashMap>, content: Arc, maybe_assert_type: Option, maybe_referrer: Option, ) -> ModuleSlot { use std::borrow::BorrowMut; let is_root = self.roots_contain(specifier); let mut module_slot = match parse_module( specifier, maybe_headers, content, maybe_assert_type, maybe_referrer, self.resolver, self.module_analyzer, is_root, self.in_dynamic_branch, ) { Ok(module) => ModuleSlot::Module(module), Err(err) => ModuleSlot::Err(err), }; if let ModuleSlot::Module(Module::Esm(module)) = module_slot.borrow_mut() { if matches!(self.graph.graph_kind, GraphKind::All | GraphKind::CodeOnly) || module.maybe_types_dependency.is_none() { for dep in module.dependencies.values_mut() { if matches!( self.graph.graph_kind, GraphKind::All | GraphKind::CodeOnly ) || dep.maybe_type.is_none() { if let Resolution::Ok(resolved) = &dep.maybe_code { let specifier = &resolved.specifier; let range = &resolved.range; let maybe_assert_type_with_range = dep .maybe_assert_type .as_ref() .map(|assert_type| AssertTypeWithRange { range: range.clone(), kind: assert_type.clone(), }); if dep.is_dynamic && !self.in_dynamic_branch { self.dynamic_branches.insert( specifier.clone(), (range.clone(), maybe_assert_type_with_range), ); } else { self.load( specifier, Some(range), self.in_dynamic_branch, maybe_assert_type_with_range, ); } } } else { dep.maybe_code = Resolution::None; } if matches!( self.graph.graph_kind, GraphKind::All | GraphKind::TypesOnly ) { if let Resolution::Ok(resolved) = &dep.maybe_type { let specifier = &resolved.specifier; let range = &resolved.range; let maybe_assert_type_with_range = dep .maybe_assert_type .as_ref() .map(|assert_type| AssertTypeWithRange { range: range.clone(), kind: assert_type.clone(), }); if dep.is_dynamic && !self.in_dynamic_branch { self.dynamic_branches.insert( specifier.clone(), (range.clone(), maybe_assert_type_with_range), ); } else { self.load( specifier, Some(range), self.in_dynamic_branch, maybe_assert_type_with_range, ); } } } else { dep.maybe_type = Resolution::None; } } } else { module.dependencies.clear(); } if matches!(self.graph.graph_kind, GraphKind::All | GraphKind::TypesOnly) { if let Some(Resolution::Ok(resolved)) = module .maybe_types_dependency .as_ref() .map(|d| &d.dependency) { self.load(&resolved.specifier, Some(&resolved.range), false, None); } } else { module.maybe_types_dependency = None; } } module_slot } } The functions, such as 'fill' and 'visit', continue to operate until all dependencies have been processed. Now that we've understood the basic functions, let's explore the 'hello world' example and see how the graph is constructed. It's important to differentiate between parsing a module and transforming TypeScript code into JavaScript code. The latter process is referred to as transpiling and occurs after the graph has been fully constructed. On the other hand, parsing takes place during each visit to a node in the graph. This ensures that the structure and contents of the modules are properly understood and processed. [](https://choubey.gitbook.io/internals-of-deno/foundations/4.12-module-graphs#graph-building-for-hello-world) Graph building for hello-world -------------------------------------------------------------------------------------------------------------------------------------------------- With a grasp of the "fill" and "visit" concepts, let's see how they're applied in our "hello world" example. We'll break it down step by step, examining how these processes build the program's graph. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/4.12-module-graphs#step-1) Step 1 The fill function is invoked using the main module as its argument. Module specifier is `file:///Users/mayankc/Work/source/deno-vs-nodejs/helloLog.ts`. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/4.12-module-graphs#step-2) Step 2 The main module is loaded. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/4.12-module-graphs#step-3) Step 3 A new process is initiated to retrieve the specifier in Deno. This process involves setting up a future that will be responsible for keeping track of the fetching procedure. Once the fetching task is completed, this future will reach a resolution point. This accomplished task is then incorporated into the list of pending operations, awaiting its turn to be processed further. `pending list = [ main module fetch future ]` ### [](https://choubey.gitbook.io/internals-of-deno/foundations/4.12-module-graphs#step-4) Step 4 The "fill" function is placed within the loop, where it patiently awaits the resolution of the upcoming futures in the pending list. At this point, there's just one future in line, which happens to be the future responsible for fetching the main module. This fetch future holds the key to the next steps in our process. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/4.12-module-graphs#step-5) Step 5 Deno has successfully retrieved and stored the main module's specifier in its cache for future use. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/4.12-module-graphs#step-6) Step 6 The visit function gets triggered for the cached main module. It's important to note that the visit function receives the cached module as its input, without requiring a specifier to be provided. This means that when the visit function is invoked, it works with the already cached main module and doesn't need additional information like a specifier to do its job. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/4.12-module-graphs#step-7) Step 7 The cached module undergoes parsing. This means that the module's code is carefully examined and understood. In this case, the module doesn't rely on any other pieces of code, so there are no dependencies or imports to worry about. This also means that there's no need to fetch additional information from other sources. Once the parsing is complete and it's confirmed that no external code is needed, the main module is then visited and processed. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/4.12-module-graphs#step-8) Step 8 The loop within the "fill" function stops running because the list of pending items is now empty. This means that there are no more tasks left for the loop to process. As a result, the loop concludes its execution. \-- The graph containing just a single module or node is now prepared and operational. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MPwgr15HSt8RqT4zJxS%252F-MPwhMS0HxiZMTjW9Qm0%252Fdeno%2520graph%2520build%2520one%2520module%2520hl.png%3Falt%3Dmedia%26token%3D948ae341-e1e8-4998-b5ca-623c6916a2a3&width=768&dpr=4&quality=100&sign=21b37d09&sv=2) Let's take a moment to review. When the graph is constructed, it entails the following steps: * The primary module is obtained, stored in memory, and analyzed. * All the associated requirements are obtained, stored in memory, and analyzed. However, we still need to explore the process of converting TypeScript to JavaScript and how it takes place. [Previous5.11 Recursive module loading](https://choubey.gitbook.io/internals-of-deno/foundations/4.11-recursive-module-loading-and-module-graphs) [Next5.13 File fetching](https://choubey.gitbook.io/internals-of-deno/foundations/4.13-file-fetcher) Last updated 1 year ago --- # 5.15 Register / compile module | The Internals of Deno Until now, we've gathered and prepared all the pieces of code, known as modules, including the main one. But at this juncture, they're in a state of readiness, waiting for their turn to step onto the stage of action within the v8 engine. This phase we're entering is pivotal – it's when we introduce these modules to the v8 engine, a process referred to as registration and instantiation. What's worth emphasizing is that, at this precise moment, our focus is solely on the loading part. We're not yet running the modules; that comes later. This separation of tasks ensures a clear boundary between two distinct phases: * The loading phase, which is like gathering all the actors backstage * The execution phase, which is when the play actually begins on the stage. [](https://choubey.gitbook.io/internals-of-deno/foundations/register-and-instantation#overview) Overview ------------------------------------------------------------------------------------------------------------- By this juncture, all the modules, regardless of their origin—be it local or remote—have been successfully retrieved. This process culminates in the creation of a module graph, while simultaneously transpiling all modules into JavaScript code. It is vital to emphasize that, at this stage, the entirety of the code is in JavaScript; TypeScript (TS) code no longer remains. With the completion of these preliminary steps, all the modules are poised for integration into the V8 engine. This integration process comprises three distinctive phases: 1. **Root Module Registration:** This initial step is intrinsic to the Deno ecosystem. Within V8 world, this procedure corresponds to the compilation of a module. Its unique significance to Deno's operation is underscored here. 2. **Imports Registration:** A comprehensive scan of all imports is undertaken, resulting in their systematic registration and subsequent compilation. This step ensures the seamless interconnection of various modules, allowing for a cohesive functioning of the codebase. 3. **Root Module Instantiation within V8:** In this concluding phase, the root module is actualized within the V8 engine. This marks the culmination of the integration process, where the orchestrated modules become functional entities within the runtime environment. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MPLHGV9kD1R8Fbu0PnI%252F-MPLHNeGXYZ831jFZ262%252Fdeno%2520reg%2520inst.png%3Falt%3Dmedia%26token%3Df853612e-1060-4f39-8b88-c0f5ee014dc0&width=768&dpr=4&quality=100&sign=efe166b4&sv=2) The process of registration within Deno involves a repeating action, as it necessitates examining all the imports involved. We've previously observed the mechanics of this process through the primary API known as "register\_and\_recurse." In our illustrative example of a basic script (helloLog.ts), no external modules are imported. As a result, we won't go deeply into import procedures in this chapter; this aspect will be explored in the subsequent chapter. As a brief refresher, let's review the summarized code for "register\_and\_recurse." However, this time, our focus will be solely on the code that pertains to loading modules into the V8 engine. The underlying concept is remarkably straightforward: * If the module's ID already exists, it signifies that the module has already been loaded into the V8 engine. * Conversely, if the module ID is not found, * The module is loaded into the V8 engine (whether it's an ES or JSON module) * The module ID provided by V8 is then recorded for future reference. This fundamental logic governs the registration process within Deno, ensuring efficient handling of module loading and availability in the V8 runtime environment. Keep in mind that the registration and loading of modules occur as the module graphs are being constructed. This means that when Deno is building the graphical representation of how different modules connect and depend on each other, it's also simultaneously going through the steps of registering those modules and loading their contents. Copy pub(crate) fn register_and_recurse( &mut self, scope: &mut v8::HandleScope, module_request: &ModuleRequest, module_source: ModuleSource, ) -> Result<(), ModuleError> { .... let maybe_module_id = self .module_map_rc .borrow() .get_id(&module_url_found, expected_asserted_module_type); let module_id = match maybe_module_id { Some(id) => { debug!( "Already-registered module fetched again: {:?}", module_url_found ); id } None => match module_source.module_type { ModuleType::JavaScript => { self.module_map_rc.borrow_mut().new_es_module( scope, self.is_currently_loading_main_module(), module_url_found, module_source.code, self.is_dynamic_import(), )? } ModuleType::Json => self.module_map_rc.borrow_mut().new_json_module( scope, module_url_found, module_source.code, )?, }, }; .... } ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MPLWdUYtWcA3hZPcdSS%252F-MPLXMCjhzM7oP9NuNIS%252Fdeno%2520register%2520steps.png%3Falt%3Dmedia%26token%3De932143c-075d-477c-8b73-4d4f4ee048b5&width=768&dpr=4&quality=100&sign=fbaf8ad&sv=2) ### [](https://choubey.gitbook.io/internals-of-deno/foundations/register-and-instantation#step-1-check-if-already-registered) Step 1 - check if already registered A module is only registered once, no matter how many times it's imported. Each module is assigned a distinct identification number. The registration process involves a couple of checks. The initial check determines if the module being registered is a root module. In our provided example, it indeed functions as the root module and stands as the solitary module subject to registration. As it contains no imports from other modules, this primary registration stands alone. The subsequent check involves verifying whether the module has already been registered. This precaution ensures that each module is registered precisely once. Each module possesses a unique identification number to facilitate this process. In the context of our example, the module retains its role as the root module and remains unregistered up to this point. This double-check mechanism guarantees that the module, although a root one, hasn't been previously registered. This prevents redundant registrations and maintains the one-time registration principle. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/register-and-instantation#step-2-register-module) Step 2 - Register module The concept of "registration" is fundamental within the context of Deno. However, when we consider the v8 environment, a comparable action involves the compilation of a module. When a module hasn't been registered, the system invokes either of the subsequent APIs: * new\_es\_module, which serves to load ES modules * new\_json\_module, designed specifically for loading JSON modules. The following is the code of new\_es\_module: Copy pub(crate) fn new_es_module( &mut self, scope: &mut v8::HandleScope, main: bool, name: ModuleName, source: ModuleCode, is_dynamic_import: bool, ) -> Result { let name_str = name.v8(scope); let source_str = source.v8(scope); let origin = module_origin(scope, name_str); let source = v8::script_compiler::Source::new(source_str, Some(&origin)); let tc_scope = &mut v8::TryCatch::new(scope); let maybe_module = v8::script_compiler::compile_module(tc_scope, source); if tc_scope.has_caught() { assert!(maybe_module.is_none()); let exception = tc_scope.exception().unwrap(); let exception = v8::Global::new(tc_scope, exception); return Err(ModuleError::Exception(exception)); } let module = maybe_module.unwrap(); let mut requests: Vec = vec![]; let module_requests = module.get_module_requests(); for i in 0..module_requests.length() { let module_request = v8::Local::::try_from( module_requests.get(tc_scope, i).unwrap(), ) .unwrap(); let import_specifier = module_request .get_specifier() .to_rust_string_lossy(tc_scope); let import_assertions = module_request.get_import_assertions(); let assertions = parse_import_assertions( tc_scope, import_assertions, ImportAssertionsKind::StaticImport, ); // FIXME(bartomieju): there are no stack frames if exception // is thrown here validate_import_assertions(tc_scope, &assertions); if tc_scope.has_caught() { let exception = tc_scope.exception().unwrap(); let exception = v8::Global::new(tc_scope, exception); return Err(ModuleError::Exception(exception)); } let module_specifier = match self.loader.resolve( &import_specifier, name.as_ref(), if is_dynamic_import { ResolutionKind::DynamicImport } else { ResolutionKind::Import }, ) { Ok(s) => s, Err(e) => return Err(ModuleError::Other(e)), }; let asserted_module_type = get_asserted_module_type_from_assertions(&assertions); let request = ModuleRequest { specifier: module_specifier.to_string(), asserted_module_type, }; requests.push(request); } if main { let maybe_main_module = self.info.iter().find(|module| module.main); if let Some(main_module) = maybe_main_module { return Err(ModuleError::Other(generic_error( format!("Trying to create \"main\" module ({:?}), when one already exists ({:?})", name.as_ref(), main_module.name, )))); } } let handle = v8::Global::::new(tc_scope, module); let id = self.create_module_info( name, ModuleType::JavaScript, handle, main, requests, ); Ok(id) } The process involves several simple and clear steps as outlined below: 1. **Perform v8 Specific Initialization:** This step entails setting up various essential components such as context, scope, and an error handler that are specific to the v8 engine. 2. **Compile Module Source Code:** The module's source code is compiled using the v8 engine, transforming it into a format that can be readily executed. 3. **Manage Imports:** The program then proceeds to iterate through the imported modules and adds them to a list, ensuring that all required components are accounted for. 4. **Record Module and Import Specifiers:** The last registration call made in the previous step serves to save the module's details and its associated import specifiers. This information is crucial for maintaining the structure and dependencies of the program. 5. **Provide Module Identifier:** The final step involves returning the unique module identifier, allowing the program to refer back to this module when needed. Similarly, the following is the code for new\_json\_module: Copy pub(crate) fn new_json_module( &mut self, scope: &mut v8::HandleScope, name: ModuleName, source: ModuleCode, ) -> Result { let name_str = name.v8(scope); let source_str = v8::String::new_from_utf8( scope, strip_bom(source.as_bytes()), v8::NewStringType::Normal, ) .unwrap(); let tc_scope = &mut v8::TryCatch::new(scope); let parsed_json = match v8::json::parse(tc_scope, source_str) { Some(parsed_json) => parsed_json, None => { assert!(tc_scope.has_caught()); let exception = tc_scope.exception().unwrap(); let exception = v8::Global::new(tc_scope, exception); return Err(ModuleError::Exception(exception)); } }; let export_names = [v8::String::new(tc_scope, "default").unwrap()]; let module = v8::Module::create_synthetic_module( tc_scope, name_str, &export_names, json_module_evaluation_steps, ); let handle = v8::Global::::new(tc_scope, module); let value_handle = v8::Global::::new(tc_scope, parsed_json); self.json_value_store.insert(handle.clone(), value_handle); let id = self.create_module_info(name, ModuleType::Json, handle, false, vec![]); Ok(id) } Let's explore our introductory Deno example. This simple case involves a single primary module with no external imports. The JavaScript code below is compiled and then loaded into the V8 engine: Copy "use strict"; function printNumber(input) { console.log(input); } function printString(input) { console.log(input); } printNumber(1); printString('One'); This is pure JS code. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/register-and-instantation#step-3-add-imports-to-the-pending-list) Step 3 - Add imports to the pending list The process here is quite similar to what we observed while constructing the module graph. After a module gets compiled within the v8 engine, it generates a list of items it imports. The subsequent phase involves the examination of these imports: * The imports are sequentially traversed. * If a particular import has already been registered, * it is disregarded and the process moves forward. * The module is then included in the pending list through a function call known as add\_import. Applying this to our current instance, where no imports are present, this particular phase does not actively take place. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/register-and-instantation#step-4-save-root-module-id) Step 4 - Save root module id This marks the final stage of the registration process. Every module has its own special module id, but only the root module id is preserved and stored. This particular module id plays a crucial role in the creation process. In the context of our situation, the root module id happens to be 1. This number serves as a significant identifier that guides the instantiation process. Id Specifier 1 file:///Users/mayankc/Work/source/denoExamples/helloLog.ts ### [](https://choubey.gitbook.io/internals-of-deno/foundations/register-and-instantation#step-5-save-module-metadata) Step 5 - Save module metadata The final action involves preserving the module metadata that V8 provides. Among the crucial pieces of information, the module handle stands out. Our task is to ensure that the module handle is associated with the corresponding module ID and kept for future reference. Copy fn create_module_info( &mut self, name: FastString, module_type: ModuleType, handle: v8::Global, main: bool, requests: Vec, ) -> ModuleId { let id = self.handles.len(); let (name1, name2) = name.into_cheap_copy(); self .by_name_mut(module_type.into()) .insert(name1, SymbolicModule::Mod(id)); self.handles.push(handle); self.info.push(ModuleInfo { id, main, name: name2, requests, module_type, }); id } At this point, there is just one more step remaining in module loading. [Previous5.14 Transpile](https://choubey.gitbook.io/internals-of-deno/foundations/4.13-check-and-or-transpilation) [Next5.16 Instantiate module](https://choubey.gitbook.io/internals-of-deno/foundations/instantiate-module) Last updated 1 year ago --- # 6.4 Transpile | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.4-transpile#overview) Overview ---------------------------------------------------------------------------------------------------- Once the module graph is prepared and structured, the next step involves transpiling the code. To prioritize a quicker startup experience, Deno has constrained this phase to focus solely on transpilation. It's important to note that the 'deno run' command exclusively handles this task without engaging in any type checks. In the preceding chapter, the transpilation process concentrated on a single module due to the absence of dependencies. In contrast, the hello v2 program encompasses various imports, necessitating the transpilation of all associated modules that contain TypeScript code. [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.4-transpile#transpile) Transpile ------------------------------------------------------------------------------------------------------ Transpilation is a crucial process that must be carried out for every module that is part of the module graph. This ensures that the code written in TypeScript (TS) is converted into a format that can be understood and executed by the V8 engine. It's important to note that modules without any TypeScript code will be excluded from this process, as there would be no need to convert them. Another aspect to consider is that modules which already have a valid output from a previous transpilation, known as an "emit," will also be skipped during this process. This skipping occurs automatically, unless a specific instruction is given to reload and re-transpile these modules. This optimization strategy is implemented to enhance the startup speed of the application. By omitting unnecessary transpilation for modules with existing valid outputs, the overall time taken to initiate the application is reduced, leading to a faster and more efficient startup experience. For a quick recap, here is the code for the main transpile function: Copy pub fn transpile(&self, options: &EmitOptions) -> Result { let program = (*self.program()).clone(); let source_map = Rc::new(SourceMap::default()); let source_map_config = SourceMapConfig { inline_sources: options.inline_sources, }; let file_name = match ModuleSpecifier::parse(self.specifier()) { Ok(specifier) => FileName::Url(specifier), Err(_) => FileName::Custom(self.specifier().to_string()), }; source_map.new_source_file(file_name, self.text_info().text().to_string()); // needs to align with what's done internally in source map assert_eq!(1, self.text_info().range().start.as_byte_pos().0); // we need the comments to be mutable, so make it single threaded let comments = self.comments().as_single_threaded(); let globals = Globals::new(); crate::swc::common::GLOBALS.set(&globals, || { let top_level_mark = Mark::fresh(Mark::root()); let program = fold_program( program, options, source_map.clone(), &comments, top_level_mark, self.diagnostics(), )?; let mut src_map_buf = vec![]; let mut buf = vec![]; { let mut writer = Box::new(JsWriter::new( source_map.clone(), "\n", &mut buf, Some(&mut src_map_buf), )); writer.set_indent_str(" "); // two spaces let config = crate::swc::codegen::Config { minify: false, ascii_only: false, omit_last_semi: false, target: ES_VERSION, }; let mut emitter = crate::swc::codegen::Emitter { cfg: config, comments: Some(&comments), cm: source_map.clone(), wr: writer, }; program.emit_with(&mut emitter)?; } let mut src = String::from_utf8(buf)?; let mut map: Option = None; { let mut buf = Vec::new(); source_map .build_source_map_with_config(&src_map_buf, None, source_map_config) .to_writer(&mut buf)?; if options.inline_source_map { src.push_str("//# sourceMappingURL=data:application/json;base64,"); base64::encode_config_buf( buf, base64::Config::new(base64::CharacterSet::Standard, true), &mut src, ); } else { map = Some(String::from_utf8(buf)?); } } Ok(TranspiledSource { text: src, source_map: map, }) }) } And to make sure we cover everything comprehensively, let's take a look at the code responsible for traversing the module graph during transpilation: Copy pub fn cache_module_emits( &self, graph: &ModuleGraph, ) -> Result<(), AnyError> { for module in graph.modules() { if let Module::Esm(module) = module { let is_emittable = matches!( module.media_type, MediaType::TypeScript | MediaType::Mts | MediaType::Cts | MediaType::Jsx | MediaType::Tsx ); if is_emittable { self.emit_parsed_source( &module.specifier, module.media_type, &module.source, )?; } } } Ok(()) } pub fn emit_parsed_source( &self, specifier: &ModuleSpecifier, media_type: MediaType, source: &Arc, ) -> Result { let source_hash = self.get_source_hash(source); if let Some(emit_code) = self.emit_cache.get_emit_code(specifier, source_hash) { Ok(emit_code.into()) } else { // this will use a cached version if it exists let parsed_source = self.parsed_source_cache.get_or_parse_module( specifier, source.clone(), media_type, )?; let transpiled_source = parsed_source.transpile(&self.emit_options)?; debug_assert!(transpiled_source.source_map.is_none()); self.emit_cache.set_emit_code( specifier, source_hash, &transpiled_source.text, ); Ok(transpiled_source.text.into()) } } These are the steps explained in detail: 1. Set up the TypeScript compilation configuration. 2. Go through all the modules in the graph. 3. If JavaScript (JS) should be ignored, then skip JS files. 4. If reload is not defined and emit is valid, then skip the file. 5. Parse the module if it hasn't been parsed already. 6. Convert/transpile the module. 7. Provide statistics and the list of loadable modules as output. Let's see the same in a flowchart: ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MPX-Q_hyVvcYAms9-vy%252F-MPX13ZEuW2EqzS4aTml%252Fdeno%2520transpile%2520steps%2520copy.png%3Falt%3Dmedia%26token%3Dfbee61c8-96f5-4010-a8ac-718665bd913d&width=768&dpr=4&quality=100&sign=99cce52b&sv=2) [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.4-transpile#transpile-hello-world-v2) Transpile hello world v2 ------------------------------------------------------------------------------------------------------------------------------------ Now that we understand the transpile logic, let's see how it operates for the hello v2 program. There are two files to transpile: 1\. The main application file 2\. The Machine\_id module Nanoid is already in JavaScript, so it doesn't require transpilation. * `Transpile file:///Users/mayankc/Work/source/denoExamples/helloV2.ts` * `Transpile https://deno.land/x/machine_id@v0.3.0/mod.ts` It's easy to see that the sequence of transpilation follows the order of adding nodes to the graph. When all modules in the graph are transpiled, the result is a set of modules called "loadable modules." These modules can operate independently. The main module is excluded from this list since it's naturally a loadable module. Here's the list of loadable modules once transpilation is complete: * `https://deno.land/x/machine_id@v0.3.0/mod.ts` ### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.4-transpile#output) Output For the output of transpilation, there are two JS files: helloV2.ts Copy import { nanoid } from "npm:nanoid"; import { getMachineId } from "https://deno.land/x/machine_id/mod.ts"; const id = nanoid(); const machineId = await getMachineId(); const homeDir = Deno.env.get("HOME"); function printNumber(input) { console.log(input); } function printString(input) { console.log(input); } printNumber(1); printString("One"); console.log("Nanoid=", id, ", MachineId=", machineId, ", homeDir=", homeDir); machine\_id's mod.ts Copy const { run, build, readAll, readFile, env } = Deno; // Get machine ID // Permission in Windows: --allow-run --allow-env // Permission in MacOS: --allow-run // Permission in Linux: --allow-read export async function getMachineId(): Promise { switch (build.os) { case "linux": return getMachineIDLinux(); case "windows": return getMachineIDWin(); case "darwin": return getMachineIDMac(); default: throw new Error(`Not support your operate system '${build.os}'`); } } function parse(bytes: Uint8Array): string { const output = new TextDecoder().decode(bytes); switch (build.os) { case "linux": return output.trim(); case "windows": return output .toString() .split("REG_SZ")[1] .replace(/\r+|\n+|\s+/gi, "") .trim(); case "darwin": const lines = output.split("\n"); for (const line of lines) { // here is the match line // "IOPlatformUUID" = "A8226C69-2364-5B3E-83CC-1A72D7531679" if (line.indexOf("IOPlatformUUID") > 0) { const [_, val] = line.split(/\s*=\s*/); return val.replace(/^"|"$/g, ""); } } return ""; default: throw new Error(`Not support your operate system '${build.os}'`); } } async function getMachineIDWin(): Promise { const winDir = env.get("windir"); const ps = run({ stdout: "piped", cmd: [\ `${winDir}\\System32\\REG.exe`,\ "QUERY",\ "HKEY_LOCAL_MACHINE\\SOFTWARE\\Microsoft\\Cryptography",\ "/v",\ "MachineGuid",\ ], }); const output = await readAll(ps.stdout!); ps.stdout?.close(); ps.close(); return parse(output); } async function getMachineIDMac(): Promise { const ps = run({ stdout: "piped", cmd: ["ioreg", "-rd1", "-c", "IOPlatformExpertDevice"], }); const output = await readAll(ps.stdout!); ps.stdout?.close(); ps.close(); return parse(output); } async function getMachineIDLinux(): Promise { // dbusPath is the default path for dbus machine id. const dbusPath = "/var/lib/dbus/machine-id"; // dbusPathEtc is the default path for dbus machine id located in /etc. // Some systems (like Fedora 20) only know this path. // Sometimes it's the other way round. const dbusPathEtc = "/etc/machine-id"; return parse( await readFile(dbusPath).catch(() => { // try fallback path return readFile(dbusPathEtc); }), ); } \-- All the modules have been processed and converted, with recursion applied. There's no more TS code from this point. It's time to integrate them into v8. We'll cover the registration and instantiation process in the following section. [Previous6.3 Module graph with imports](https://choubey.gitbook.io/internals-of-deno/import-and-ops/building-module-graph) [Next6.5 Registration and instantiation](https://choubey.gitbook.io/internals-of-deno/import-and-ops/registration-and-instantiation) Last updated 1 year ago --- # 5.7 Main Worker | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/foundations/mainworker#overview) Overview ---------------------------------------------------------------------------------------------- The heart of Deno beats within the MainWorker, serving as its central hub. It plays a role similar to that of an orchestrator, coordinating various activities. Within the MainWorker, the program readies itself for action and eventually springs into operation. The MainWorker's responsibilities span from initializing the JavaScript runtime to loading modules and carrying out executions. An essential collaborator for the MainWorker is the JavaScript (JS) runtime, a substantial component worthy of its own detailed exploration. For now, as we get into the MainWorker's domain, we'll treat the JS runtime as a mysterious black box. We'll unveil the complexities of the JS runtime in our upcoming section. Turning our attention to the run command code, the MainWorker emerges into existence through the execution of the following lines of code: Copy let mut worker = worker_factory .create_main_worker(main_module, permissions) .await?; The function "create\_main\_worker" is responsible for creating a CLIMainWorker. This special worker is constructed using various attributes and settings. Its primary role is to kickstart the execution of code. An interesting addition is its possession of a JS runtime instance, within which the JavaScript code operates. The initiation and setup of the JS runtime occur right within the confines of the main worker itself. Consequently, the JS runtime doesn't need to be transferred to the main worker allocator separately. In a sense, the JS runtime becomes an integral component of the main worker's structure. This approach ensures that the main worker possesses all the necessary components, including the JS runtime, to function effectively. [](https://choubey.gitbook.io/internals-of-deno/foundations/mainworker#functionality) Functionality -------------------------------------------------------------------------------------------------------- The primary worker isn't inherently complex. Instead, think of the main worker as a conductor that directs the execution of code. Let's delve into the core tasks that the main worker handles: 1. **Bootstrap** * This phase involves loading and setting up the essential components of Deno's functionality. 2. **Initialize Ops** * Here, the main worker takes the initiative to start and connect external operations (ops) with the JavaScript runtime (JsRuntime). This establishes the bridge between Deno's core and external functionalities. 3. **Create JsRuntime** * The main worker's role in this step is to forge a V8 isolate, which is like a separate environment to run JavaScript code. It also creates a runtime that manages the execution of JavaScript within this isolated environment. 4. **Load Module** * In this phase, the main worker is responsible for fetching and instantiating a module. A module can be thought of as a piece of code with specific functionality. 5. **Execute Module** * The main worker's task here is to carry out the execution or evaluation of a module. This is where the actual code within the module gets to run. 6. **Create Inspector Session** * For debugging purposes, the main worker takes on the responsibility of generating an inspector session. This allows developers to closely examine and troubleshoot the code's behavior. So, while the main worker might seem like a simple orchestrator, it plays a crucial role in managing Deno's foundational processes. From initiating core functionalities to overseeing module execution and aiding in debugging, the main worker is a vital component in the inner workings of Deno. [](https://choubey.gitbook.io/internals-of-deno/foundations/mainworker#steps-to-create-a-worker) **Steps to create a worker** ---------------------------------------------------------------------------------------------------------------------------------- The process of creating a worker involves several sequential steps. It all begins with the establishment of a CLI module loader and culminates in the initiation of the runtime bootstrap. Now, let's get into each of these steps to gain a comprehensive understanding. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MOwjHuHSYqCMe-Qoxv-%252F-MOwk2_Fo1PNdmUVw2Lv%252Fdeno%2520create%2520main%2520workerr.png%3Falt%3Dmedia%26token%3Db6503774-50a6-4068-b07f-174ef706fd3e&width=768&dpr=4&quality=100&sign=e73c2b91&sv=2) Copy pub async fn create_main_worker( &self, main_module: ModuleSpecifier, permissions: PermissionsContainer, ) -> Result { self .create_custom_worker( main_module, permissions, vec![], Default::default(), ) .await } pub async fn create_custom_worker( &self, main_module: ModuleSpecifier, permissions: PermissionsContainer, mut custom_extensions: Vec, stdio: deno_runtime::deno_io::Stdio, ) -> Result { let shared = &self.shared; let (main_module, is_main_cjs) = if let Ok(package_ref) = NpmPackageReqReference::from_specifier(&main_module) { shared .npm_resolver .add_package_reqs(&[package_ref.req().clone()]) .await?; let node_resolution = self.resolve_binary_entrypoint(&package_ref, &permissions)?; let is_main_cjs = matches!(node_resolution, NodeResolution::CommonJs(_)); if let Some(lockfile) = &shared.maybe_lockfile { // For npm binary commands, ensure that the lockfile gets updated // so that we can re-use the npm resolution the next time it runs // for better performance lockfile .lock() .write() .context("Failed writing lockfile.")?; } (node_resolution.into_url(), is_main_cjs) } else if shared.options.is_npm_main { let node_resolution = shared.node_resolver.url_to_node_resolution(main_module)?; let is_main_cjs = matches!(node_resolution, NodeResolution::CommonJs(_)); (node_resolution.into_url(), is_main_cjs) } else { (main_module, false) }; let module_loader = shared .module_loader_factory .create_for_main(PermissionsContainer::allow_all(), permissions.clone()); let maybe_source_map_getter = shared.module_loader_factory.create_source_map_getter(); let maybe_inspector_server = shared.maybe_inspector_server.clone(); let create_web_worker_cb = create_web_worker_callback(shared.clone(), stdio.clone()); let maybe_storage_key = shared .storage_key_resolver .resolve_storage_key(&main_module); let origin_storage_dir = maybe_storage_key.as_ref().map(|key| { shared .options .origin_data_folder_path .as_ref() .unwrap() // must be set if storage key resolver returns a value .join(checksum::gen(&[key.as_bytes()])) }); let cache_storage_dir = maybe_storage_key.map(|key| { // TODO(@satyarohith): storage quota management // Note: we currently use temp_dir() to avoid managing storage size. std::env::temp_dir() .join("deno_cache") .join(checksum::gen(&[key.as_bytes()])) }); let mut extensions = ops::cli_exts(shared.npm_resolver.clone()); extensions.append(&mut custom_extensions); let options = WorkerOptions { bootstrap: BootstrapOptions { args: shared.options.argv.clone(), cpu_count: std::thread::available_parallelism() .map(|p| p.get()) .unwrap_or(1), log_level: shared.options.log_level, enable_testing_features: shared.options.enable_testing_features, locale: deno_core::v8::icu::get_language_tag(), location: shared.options.location.clone(), no_color: !colors::use_color(), is_tty: colors::is_tty(), runtime_version: version::deno().to_string(), ts_version: version::TYPESCRIPT.to_string(), unstable: shared.options.unstable, user_agent: version::get_user_agent().to_string(), inspect: shared.options.is_inspecting, has_node_modules_dir: shared.options.has_node_modules_dir, maybe_binary_npm_command_name: shared .options .maybe_binary_npm_command_name .clone(), }, extensions, startup_snapshot: crate::js::deno_isolate_init(), create_params: None, unsafely_ignore_certificate_errors: shared .options .unsafely_ignore_certificate_errors .clone(), root_cert_store_provider: Some(shared.root_cert_store_provider.clone()), seed: shared.options.seed, source_map_getter: maybe_source_map_getter, format_js_error_fn: Some(Arc::new(format_js_error)), create_web_worker_cb, maybe_inspector_server, should_break_on_first_statement: shared.options.inspect_brk, should_wait_for_inspector_session: shared.options.inspect_wait, module_loader, fs: shared.fs.clone(), npm_resolver: Some(shared.npm_resolver.clone()), get_error_class_fn: Some(&errors::get_error_class_name), cache_storage_dir, origin_storage_dir, blob_store: shared.blob_store.clone(), broadcast_channel: shared.broadcast_channel.clone(), shared_array_buffer_store: Some(shared.shared_array_buffer_store.clone()), compiled_wasm_module_store: Some( shared.compiled_wasm_module_store.clone(), ), stdio, }; let worker = MainWorker::bootstrap_from_options( main_module.clone(), permissions, options, ); Ok(CliMainWorker { main_module, is_main_cjs, worker, shared: shared.clone(), }) } Apart from the previously mentioned function, Deno also utilizes the "bootstrap\_from\_options()" function to generate the real worker. Although the code within "bootstrap\_from\_options()" is quite extensive, it's worth taking a moment to briefly explore its intriguing details. Copy pub fn bootstrap_from_options( main_module: ModuleSpecifier, permissions: PermissionsContainer, options: WorkerOptions, ) -> Self { let bootstrap_options = options.bootstrap.clone(); let mut worker = Self::from_options(main_module, permissions, options); worker.bootstrap(&bootstrap_options); worker } pub fn from_options( main_module: ModuleSpecifier, permissions: PermissionsContainer, mut options: WorkerOptions, ) -> Self { deno_core::extension!(deno_permissions_worker, options = { permissions: PermissionsContainer, unstable: bool, enable_testing_features: bool, }, state = |state, options| { state.put::(options.permissions); state.put(ops::UnstableChecker { unstable: options.unstable }); state.put(ops::TestingFeaturesEnabled(options.enable_testing_features)); }, ); // Permissions: many ops depend on this let unstable = options.bootstrap.unstable; let enable_testing_features = options.bootstrap.enable_testing_features; let exit_code = ExitCode(Arc::new(AtomicI32::new(0))); let create_cache = options.cache_storage_dir.map(|storage_dir| { let create_cache_fn = move || SqliteBackedCache::new(storage_dir.clone()); CreateCache(Arc::new(create_cache_fn)) }); // NOTE(bartlomieju): ordering is important here, keep it in sync with // `runtime/build.rs`, `runtime/web_worker.rs` and `cli/build.rs`! let mut extensions = vec![\ // Web APIs\ deno_webidl::deno_webidl::init_ops_and_esm(),\ deno_console::deno_console::init_ops_and_esm(),\ deno_url::deno_url::init_ops_and_esm(),\ deno_web::deno_web::init_ops_and_esm::(\ options.blob_store.clone(),\ options.bootstrap.location.clone(),\ ),\ deno_fetch::deno_fetch::init_ops_and_esm::(\ deno_fetch::Options {\ user_agent: options.bootstrap.user_agent.clone(),\ root_cert_store_provider: options.root_cert_store_provider.clone(),\ unsafely_ignore_certificate_errors: options\ .unsafely_ignore_certificate_errors\ .clone(),\ file_fetch_handler: Rc::new(deno_fetch::FsFetchHandler),\ ..Default::default()\ },\ ),\ deno_cache::deno_cache::init_ops_and_esm::(\ create_cache,\ ),\ deno_websocket::deno_websocket::init_ops_and_esm::(\ options.bootstrap.user_agent.clone(),\ options.root_cert_store_provider.clone(),\ options.unsafely_ignore_certificate_errors.clone(),\ ),\ deno_webstorage::deno_webstorage::init_ops_and_esm(\ options.origin_storage_dir.clone(),\ ),\ deno_crypto::deno_crypto::init_ops_and_esm(options.seed),\ deno_broadcast_channel::deno_broadcast_channel::init_ops_and_esm(\ options.broadcast_channel.clone(),\ unstable,\ ),\ deno_ffi::deno_ffi::init_ops_and_esm::(unstable),\ deno_net::deno_net::init_ops_and_esm::(\ options.root_cert_store_provider.clone(),\ unstable,\ options.unsafely_ignore_certificate_errors.clone(),\ ),\ deno_tls::deno_tls::init_ops_and_esm(),\ deno_kv::deno_kv::init_ops_and_esm(\ SqliteDbHandler::::new(\ options.origin_storage_dir.clone(),\ ),\ unstable,\ ),\ deno_napi::deno_napi::init_ops_and_esm::(),\ deno_http::deno_http::init_ops_and_esm::(),\ deno_io::deno_io::init_ops_and_esm(Some(options.stdio)),\ deno_fs::deno_fs::init_ops_and_esm::(\ unstable,\ options.fs.clone(),\ ),\ deno_node::deno_node::init_ops_and_esm::(\ options.npm_resolver,\ options.fs,\ ),\ // Ops from this crate\ ops::runtime::deno_runtime::init_ops_and_esm(main_module.clone()),\ ops::worker_host::deno_worker_host::init_ops_and_esm(\ options.create_web_worker_cb.clone(),\ options.format_js_error_fn.clone(),\ ),\ ops::fs_events::deno_fs_events::init_ops_and_esm(),\ ops::os::deno_os::init_ops_and_esm(exit_code.clone()),\ ops::permissions::deno_permissions::init_ops_and_esm(),\ ops::process::deno_process::init_ops_and_esm(),\ ops::signal::deno_signal::init_ops_and_esm(),\ ops::tty::deno_tty::init_ops_and_esm(),\ ops::http::deno_http_runtime::init_ops_and_esm(),\ deno_permissions_worker::init_ops_and_esm(\ permissions,\ unstable,\ enable_testing_features,\ ),\ runtime::init_ops_and_esm(),\ ]; for extension in &mut extensions { #[cfg(not(feature = "__runtime_js_sources"))] { extension.js_files = std::borrow::Cow::Borrowed(&[]); extension.esm_files = std::borrow::Cow::Borrowed(&[]); extension.esm_entry_point = None; } #[cfg(feature = "__runtime_js_sources")] { for source in extension.esm_files.to_mut() { maybe_transpile_source(source).unwrap(); } for source in extension.js_files.to_mut() { maybe_transpile_source(source).unwrap(); } } } extensions.extend(std::mem::take(&mut options.extensions)); #[cfg(all(feature = "include_js_files_for_snapshotting", feature = "dont_create_runtime_snapshot", not(feature = "__runtime_js_sources")))] options.startup_snapshot.as_ref().expect("Sources are not embedded, snapshotting was disabled and a user snapshot was not provided."); // Clear extension modules from the module map, except preserve `node:*` // modules. let preserve_snapshotted_modules = Some(SUPPORTED_BUILTIN_NODE_MODULES_WITH_PREFIX); let mut js_runtime = JsRuntime::new(RuntimeOptions { module_loader: Some(options.module_loader.clone()), startup_snapshot: options .startup_snapshot .or_else(crate::js::deno_isolate_init), create_params: options.create_params, source_map_getter: options.source_map_getter, get_error_class_fn: options.get_error_class_fn, shared_array_buffer_store: options.shared_array_buffer_store.clone(), compiled_wasm_module_store: options.compiled_wasm_module_store.clone(), extensions, preserve_snapshotted_modules, inspector: options.maybe_inspector_server.is_some(), is_main: true, ..Default::default() }); if let Some(server) = options.maybe_inspector_server.clone() { server.register_inspector( main_module.to_string(), &mut js_runtime, options.should_break_on_first_statement || options.should_wait_for_inspector_session, ); // Put inspector handle into the op state so we can put a breakpoint when // executing a CJS entrypoint. let op_state = js_runtime.op_state(); let inspector = js_runtime.inspector(); op_state.borrow_mut().put(inspector); } let bootstrap_fn_global = { let context = js_runtime.main_context(); let scope = &mut js_runtime.handle_scope(); let context_local = v8::Local::new(scope, context); let global_obj = context_local.global(scope); let bootstrap_str = v8::String::new_external_onebyte_static(scope, b"bootstrap").unwrap(); let bootstrap_ns: v8::Local = global_obj .get(scope, bootstrap_str.into()) .unwrap() .try_into() .unwrap(); let main_runtime_str = v8::String::new_external_onebyte_static(scope, b"mainRuntime").unwrap(); let bootstrap_fn = bootstrap_ns.get(scope, main_runtime_str.into()).unwrap(); let bootstrap_fn = v8::Local::::try_from(bootstrap_fn).unwrap(); v8::Global::new(scope, bootstrap_fn) }; Self { js_runtime, should_break_on_first_statement: options.should_break_on_first_statement, should_wait_for_inspector_session: options .should_wait_for_inspector_session, exit_code, bootstrap_fn_global: Some(bootstrap_fn_global), } } ### [](https://choubey.gitbook.io/internals-of-deno/foundations/mainworker#climoduleloader) **CLIModuleLoader** Copy let module_loader = shared .module_loader_factory .create_for_main(PermissionsContainer::allow_all(), permissions.clone()); The CLIModuleLoader serves as a covering layer for the process of loading modules in Deno. Whenever a ModuleSpecifier is provided, this specialized loader becomes active, managing the tasks of loading and compiling modules. The functions within this module loader play a vital role, offering three primary actions: 1. **Resolve:** * This function is responsible for determining the module specifier associated with an ES module. It helps in finding the correct path to the module. 2. **Prepare and Load:** * The next significant function is the "prepare load." This action involves getting a module ready for loading. It ensures that the necessary preparations are made before the module is loaded. 3. **Load:** * The final function, "load," takes care of the loading process itself. Once a module has been compiled, this function helps to bring it into the runtime environment, making it available for use. As we poke deeper into our discussion on module loading, we will explore these functions in greater detail to gain a comprehensive understanding of how the CLIModuleLoader operates. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/mainworker#js-runtime) JS Runtime MainWorker::from\_options creates the main worker, and the process of creation of the main worker ends with creating an instance of the JS runtime. Js runtime needs the following things to initialize: * Module loader (or CLI module loader) * Snapshot * V8 Isolate * Shared array buffer store * etc. #### [](https://choubey.gitbook.io/internals-of-deno/foundations/mainworker#initialize-isolate) **Initialize isolate** The concept of an "Isolate" stands as a foundational pillar within Google's V8 engine. This Isolate serves as a way to compartmentalize and isolate different JavaScript code executions from one another, preventing unintended interactions. In Deno, an Isolate is established and initiated through the function deno\_isolate\_init(). This particular function is responsible for creating a static instance of the V8 Isolate, which offers advantages in terms of rapid loading speeds. To dig a bit deeper into this, static isolates play a crucial role in expediting the initialization process of the V8 engine. These isolates are pre-configured and optimized, leading to quicker loading times for your applications. The static nature of these isolates also brings about a noteworthy benefit – they are included as part of the Deno package, so you don't need to manage their setup separately. Consider these static isolates as snapshots frozen in time, capturing a moment of optimal performance. They are bundled with Deno, making it simpler for developers to harness their advantages without intricate configuration steps. In fact, the CLI\_SNAPSHOT, an essential component of Deno, is initiatialized from a .bin file. This file format encapsulates a pre-built snapshot of Deno's runtime environment, efficiently packaging essential code and functionalities. This snapshot empowers Deno to quickly start up and execute code by leveraging the pre-prepared groundwork contained within the CLI\_SNAPSHOT. For those eager to dive even deeper into the complexities of V8 isolates, a wealth of information awaits at v8.dev. This ultimate resource serves as a treasure of insights into the inner workings of V8 isolates, shedding light on their significance and how they contribute to the efficiency and effectiveness of applications running on Deno. Copy static RUNTIME_SNAPSHOT: &[u8] = include_bytes!(concat!(env!("OUT_DIR"), "/RUNTIME_SNAPSHOT.bin")); pub fn deno_isolate_init() -> Option { debug!("Deno isolate init with snapshots."); #[cfg(not(feature = "dont_create_runtime_snapshot"))] { Some(Snapshot::Static(RUNTIME_SNAPSHOT)) } #[cfg(feature = "dont_create_runtime_snapshot")] { None } } #### [](https://choubey.gitbook.io/internals-of-deno/foundations/mainworker#initialize-runtime) Initialize runtime The JS runtime is a crucial element that enables the execution of JavaScript programs using the V8 engine. The JS runtime contains significant code that facilitates seamless interaction between JavaScript code and the V8 engine. Notably, each worker has its own separate JS runtime, which is essential because JS runtimes and data are not shared among workers. We will explore the JS runtime's role and significance in more detail in subsequent sections, providing a comprehensive understanding of its importance in the Deno environment. Copy let mut js_runtime = JsRuntime::new(RuntimeOptions { module_loader: Some(options.module_loader.clone()), startup_snapshot: options .startup_snapshot .or_else(crate::js::deno_isolate_init), create_params: options.create_params, source_map_getter: options.source_map_getter, get_error_class_fn: options.get_error_class_fn, shared_array_buffer_store: options.shared_array_buffer_store.clone(), compiled_wasm_module_store: options.compiled_wasm_module_store.clone(), extensions, preserve_snapshotted_modules, inspector: options.maybe_inspector_server.is_some(), is_main: true, ..Default::default() }); ### [](https://choubey.gitbook.io/internals-of-deno/foundations/mainworker#ops-initialization) **Ops initialization** Ops are basic operations written in Rust that serve as the foundation for Deno's advanced features. To better understand this concept, let's consider a practical example. Suppose you want to read a file from your computer's disk. This process involves several steps within Deno's architecture: 1. **High-Level Function:** At the apex of this hierarchy, we encounter a user-friendly function known as `Deno.copyFileSync()`. This function provides a simple way to copy one file to another. 2. **Low-Level Operation:** Facilitating the functionality of the high-level function, there exists a core operation named `op_fs_copy_file_sync()`. This operation operates at a lower level and is implemented in Rust. It's responsible for executing the file copy operation efficiently. Zooming out, this represents the deepest layer within Deno's structure. This tiered arrangement of high-level functions and corresponding low-level ops showcases the organization of Deno's operations. If you're interested in comprehending the nuances of this distinction, examining the following code can be useful: Copy function copyFileSync( fromPath, toPath, ) { ops.op_fs_copy_file_sync( pathFromURL(fromPath), pathFromURL(toPath), ); } The process of working with ops involves transitioning from the V8 engine's domain to Deno's code realm. This is essential because while `copyFileSync()` operates within V8, the corresponding op functions run within Deno. In order to facilitate this seamless interaction, the initialization of ops becomes a necessity. During the initialization of ops, a critical task is the registration of these ops with the JavaScript runtime as external references. Once registered, these ops become accessible for the V8 engine to invoke. These op functions serve a crucial role as they encompass functionalities that are not directly provided by V8. Unlike the fundamental JavaScript functions that V8 meticulously adheres to, these ops expand the capabilities beyond what is defined in the ECMAScript specification. As part of the ops initialization process, various categories of ops are established within the worker's scope. These categories help organize the different types of operations that the ops can perform. This strategic categorization aids in maintaining clarity and efficiency within the Deno runtime environment. Here are the key categories in which ops are initialized within the worker's scope: * runtime * fetch * timers * worker\_host * crypto * errors * fs * fs events * io * net * os * permissions * plugin * process * signal * tls * tty * websocket In a previous code example, we observed how, during the initialization process, Deno visits all the extensions and registers OPs (operations). Let's revisit the pertinent code once more for clarity: Copy for ctx in ops { let ctx_ptr = ctx as *const OpCtx as _; references.push(v8::ExternalReference { pointer: ctx_ptr }); references.push(v8::ExternalReference { function: ctx.decl.v8_fn_ptr, }); if let Some(fast_fn) = &ctx.decl.fast_fn { references.push(v8::ExternalReference { pointer: fast_fn.function as _, }); references.push(v8::ExternalReference { pointer: ctx.fast_fn_c_info.unwrap().as_ptr() as _, }); } } That's all in ops initialization. We'll see how ops get called in detail later. ### [](https://choubey.gitbook.io/internals-of-deno/foundations/mainworker#add-streams) Add streams The second last step in worker initialization is to add standard streams to the resource table (code is from the io extension): Copy state = |state, options| { if let Some(stdio) = options.stdio { let t = &mut state.resource_table; let rid = t.add(fs::FileResource::new( Rc::new(match stdio.stdin { StdioPipe::Inherit => StdFileResourceInner::new( StdFileResourceKind::Stdin, STDIN_HANDLE.try_clone().unwrap(), ), StdioPipe::File(pipe) => StdFileResourceInner::file(pipe), }), "stdin".to_string(), )); assert_eq!(rid, 0, "stdin must have ResourceId 0"); let rid = t.add(FileResource::new( Rc::new(match stdio.stdout { StdioPipe::Inherit => StdFileResourceInner::new( StdFileResourceKind::Stdout, STDOUT_HANDLE.try_clone().unwrap(), ), StdioPipe::File(pipe) => StdFileResourceInner::file(pipe), }), "stdout".to_string(), )); assert_eq!(rid, 1, "stdout must have ResourceId 1"); let rid = t.add(FileResource::new( Rc::new(match stdio.stderr { StdioPipe::Inherit => StdFileResourceInner::new( StdFileResourceKind::Stderr, STDERR_HANDLE.try_clone().unwrap(), ), StdioPipe::File(pipe) => StdFileResourceInner::file(pipe), }), "stderr".to_string(), )); assert_eq!(rid, 2, "stderr must have ResourceId 2"); } ### [](https://choubey.gitbook.io/internals-of-deno/foundations/mainworker#bootstrap) **Bootstrap** At this point, the worker is ready with the JS runtime. However, it's still not ready to process the user program. Bootstrapping makes Deno ready to start executing the user program. Here is the call to the bootstrap function which is present at the very end of the create\_main\_worker function. Copy worker.bootstrap(&bootstrap_options); Copy pub fn bootstrap(&mut self, options: &BootstrapOptions) { let scope = &mut self.js_runtime.handle_scope(); let args = options.as_v8(scope); let bootstrap_fn = self.bootstrap_fn_global.take().unwrap(); let bootstrap_fn = v8::Local::new(scope, bootstrap_fn); let undefined = v8::undefined(scope); bootstrap_fn.call(scope, undefined.into(), &[args]).unwrap(); } The last stage of worker creation, known as bootstrapping, involves carrying out a small script's execution. In this phase, the worker's bootstrap function generates certain options and subsequently runs a concise script named `bootstrap.mainRuntime({})` within the freshly initialized JavaScript runtime. The worker's execute function serves the purpose of running JavaScript code. We will scrabble into the specifics of script execution shortly. For the time being, let's progress forward and shift our attention to the JavaScript bootstrap function. The code snippet `bootstrap.mainRuntime()` represents a piece of JavaScript code that aims to invoke the `mainRuntime` function within the bootstrap namespace. To achieve this, the script seeks to establish a call to the aforementioned function. As we proceed, we'll get deeper into these concepts to gain a more comprehensive understanding. Copy globalThis.bootstrap = { mainRuntime: bootstrapMainRuntime, workerRuntime: bootstrapWorkerRuntime, }; Here is the implementation of bootstrapMainRuntime(): Copy function bootstrapMainRuntime(runtimeOptions) { if (hasBootstrapped) { throw new Error("Worker runtime already bootstrapped"); } const nodeBootstrap = globalThis.nodeBootstrap; const { 0: args, 1: cpuCount, 2: logLevel, 3: denoVersion, 4: locale, 5: location_, 6: noColor, 7: isTty, 8: tsVersion, 9: unstableFlag, 10: pid, 11: target, 12: v8Version, 13: userAgent, 14: inspectFlag, // 15: enableTestingFeaturesFlag 16: hasNodeModulesDir, 17: maybeBinaryNpmCommandName, } = runtimeOptions; performance.setTimeOrigin(DateNow()); globalThis_ = globalThis; // Remove bootstrapping data from the global scope delete globalThis.__bootstrap; delete globalThis.bootstrap; delete globalThis.nodeBootstrap; hasBootstrapped = true; // If the `--location` flag isn't set, make `globalThis.location` `undefined` and // writable, so that they can mock it themselves if they like. If the flag was // set, define `globalThis.location`, using the provided value. if (location_ == null) { mainRuntimeGlobalProperties.location = { writable: true, }; } else { location.setLocationHref(location_); } if (unstableFlag) { ObjectDefineProperties(globalThis, unstableWindowOrWorkerGlobalScope); } ObjectDefineProperties(globalThis, mainRuntimeGlobalProperties); ObjectDefineProperties(globalThis, { close: util.writable(windowClose), closed: util.getterOnly(() => windowIsClosing), }); ObjectSetPrototypeOf(globalThis, Window.prototype); if (inspectFlag) { const consoleFromV8 = core.console; const consoleFromDeno = globalThis.console; wrapConsole(consoleFromDeno, consoleFromV8); } event.setEventTargetData(globalThis); event.saveGlobalThisReference(globalThis); event.defineEventHandler(globalThis, "error"); event.defineEventHandler(globalThis, "load"); event.defineEventHandler(globalThis, "beforeunload"); event.defineEventHandler(globalThis, "unload"); event.defineEventHandler(globalThis, "unhandledrejection"); core.setPromiseRejectCallback(promiseRejectCallback); runtimeStart( denoVersion, v8Version, tsVersion, target, logLevel, noColor, isTty, ); setNumCpus(cpuCount); setUserAgent(userAgent); setLanguage(locale); let ppid = undefined; ObjectDefineProperties(finalDenoNs, { pid: util.readOnly(pid), ppid: util.getterOnly(() => { // lazy because it's expensive if (ppid === undefined) { ppid = ops.op_ppid(); } return ppid; }), noColor: util.readOnly(noColor), args: util.readOnly(ObjectFreeze(args)), mainModule: util.getterOnly(opMainModule), }); if (unstableFlag) { ObjectAssign(finalDenoNs, denoNsUnstable); } // Setup `Deno` global - we're actually overriding already existing global // `Deno` with `Deno` namespace from "./deno.ts". ObjectDefineProperty(globalThis, "Deno", util.readOnly(finalDenoNs)); util.log("args", args); if (nodeBootstrap) { nodeBootstrap(hasNodeModulesDir, maybeBinaryNpmCommandName); } } In essence, the function bootStrapMainRuntime carries out a series of essential tasks: 1. It establishes handlers for both load and unload events, allowing the system to manage the initiation and conclusion of operations efficiently. 2. The function prepares the deno namespace, effectively setting up the fundamental environment and groundwork for the upcoming operations. 3. Additionally, it solidifies core objects such as Deno, Deno.core, and Deno.core.sharedQueue, ensuring that these central components remain unalterable and consistent throughout the runtime. 4. Once this bootstrapping process is successfully completed, the primary worker becomes fully prepared to undertake the execution of the main module or the user's designated program. \-- Now that we've discussed workers, let's move on to initiating code execution. Before we run the code, it's essential to understand the JavaScript runtime's functionalities. This will give us a clearer understanding of the background processes that occur as our code runs. [Previous5.6 Permissions](https://choubey.gitbook.io/internals-of-deno/foundations/permissions) [Next5.8 JS Runtime](https://choubey.gitbook.io/internals-of-deno/foundations/jsruntime) Last updated 1 year ago --- # 5.17 Evaluate module | The Internals of Deno The moment we've been waiting for has finally arrived! It's the culmination of all our efforts. Now, it's time to bring our code to life. In the v8 engine, this process is called module evaluation. Let's explore how our simple "hello world" program is evaluated, executed, and brought to life. [](https://choubey.gitbook.io/internals-of-deno/foundations/evaluate-module#overview) Overview --------------------------------------------------------------------------------------------------- Once all the necessary modules have been fetched, cached, loaded, transpiled, compiled, and prepared for use, the next step is to initiate the execution of the application. Similar to how a C, C++, or Java program begins with the main program, a JavaScript code commences its execution from the main module or what is often referred to as the root module. Immediately after the loading process is completed, the worker's run function triggers the evaluate\_module function. This function is responsible for initiating the evaluation or execution of the root module. To achieve this, the evaluate\_module function requires the module ID of the root module as its input. The evaluate\_module function undertakes two crucial tasks: 1. It invokes the mod\_evaluate function of the JavaScript runtime, thereby initiating the module evaluation within the V8 engine. 2. It facilitates the operation of the well-known event loop, ensuring that the system awaits the completion of the module evaluation process and its resultant output. This step is fundamental for coordinating asynchronous operations and managing the flow of the program. Copy pub async fn evaluate_module( &mut self, id: ModuleId, ) -> Result<(), AnyError> { self.wait_for_inspector_session(); let mut receiver = self.js_runtime.mod_evaluate(id); tokio::select! { // Not using biased mode leads to non-determinism for relatively simple // programs. biased; maybe_result = &mut receiver => { debug!("received module evaluate {:#?}", maybe_result); maybe_result.expect("Module evaluation result not provided.") } event_loop_result = self.run_event_loop(false) => { event_loop_result?; let maybe_result = receiver.await; maybe_result.expect("Module evaluation result not provided.") } } } The mod\_evaluate function initiates the execution process within the V8 engine. After this, the program enters an event loop phase that continually checks for various outcomes such as evaluation results, resolutions of promises, operational (OP) results, handling of dynamic imports, and identification of errors. Let's dive deeper into module evaluation and explore its intricacies. Next, we'll examine the event loop's role in managing asynchronous tasks and their outcomes, as it plays a vital part in this process. [](https://choubey.gitbook.io/internals-of-deno/foundations/evaluate-module#module-evaluation) Module evaluation --------------------------------------------------------------------------------------------------------------------- The initial module evaluation function is called mod\_evaluate and it goes through these steps: * It sends the main module for evaluation. * It looks out for any instant: * Errors * Rejections of Promises It's important to note that not all errors are brought up as soon as the evaluation begins. Nonetheless, certain errors are indeed raised right away in the process. The function called mod\_evaluate is somewhat larger. The next illustration demonstrates the sequence of actions that this function carries out: ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MPNugp495fwYqsH1hmZ%252F-MPNvuQlryogGXfYKuFT%252Fdeno%2520mod%2520evaluate.png%3Falt%3Dmedia%26token%3D3d654b6c-6af2-4de5-900d-750d65553e21&width=768&dpr=4&quality=100&sign=25d1c34f&sv=2) Here is the source of mod\_evaluate(): Copy pub fn mod_evaluate( &self, isolate: &mut v8::Isolate, id: ModuleId, ) -> oneshot::Receiver> { let state_rc = self.0.state(); let module_map_rc = self.0.module_map(); let scope = &mut self.handle_scope(isolate); let tc_scope = &mut v8::TryCatch::new(scope); let module = module_map_rc .borrow() .get_handle(id) .map(|handle| v8::Local::new(tc_scope, handle)) .expect("ModuleInfo not found"); let mut status = module.get_status(); assert_eq!( status, v8::ModuleStatus::Instantiated, "{} {} ({})", if status == v8::ModuleStatus::Evaluated { "Module already evaluated. Perhaps you've re-provided a module or extension that was already included in the snapshot?" } else { "Module not instantiated" }, module_map_rc .borrow() .get_info_by_id(id) .unwrap() .name .as_str(), id, ); let (sender, receiver) = oneshot::channel(); { let mut state = state_rc.borrow_mut(); assert!( state.pending_mod_evaluate.is_none(), "There is already pending top level module evaluation" ); state.pending_mod_evaluate = Some(ModEvaluate { promise: None, has_evaluated: false, handled_promise_rejections: vec![], sender, }); } let maybe_value = module.evaluate(tc_scope); { let mut state = state_rc.borrow_mut(); let pending_mod_evaluate = state.pending_mod_evaluate.as_mut().unwrap(); pending_mod_evaluate.has_evaluated = true; } // Update status after evaluating. status = module.get_status(); let has_dispatched_exception = self .0 .runtime_state .borrow_mut() .dispatched_exception .is_some(); if has_dispatched_exception { // This will be overridden in `exception_to_err_result()`. let exception = v8::undefined(tc_scope).into(); let pending_mod_evaluate = { let mut state = state_rc.borrow_mut(); state.pending_mod_evaluate.take().unwrap() }; pending_mod_evaluate .sender .send(exception_to_err_result(tc_scope, exception, false)) .expect("Failed to send module evaluation error."); } else if let Some(value) = maybe_value { assert!( status == v8::ModuleStatus::Evaluated || status == v8::ModuleStatus::Errored ); let promise = v8::Local::::try_from(value) .expect("Expected to get promise as module evaluation result"); let promise_global = v8::Global::new(tc_scope, promise); let mut state = state_rc.borrow_mut(); { let pending_mod_evaluate = state.pending_mod_evaluate.as_ref().unwrap(); let pending_rejection_was_already_handled = pending_mod_evaluate .handled_promise_rejections .contains(&promise_global); if !pending_rejection_was_already_handled { state .pending_promise_rejections .retain(|(key, _)| key != &promise_global); } } let promise_global = v8::Global::new(tc_scope, promise); state.pending_mod_evaluate.as_mut().unwrap().promise = Some(promise_global); tc_scope.perform_microtask_checkpoint(); } else if tc_scope.has_terminated() || tc_scope.is_execution_terminating() { let pending_mod_evaluate = { let mut state = state_rc.borrow_mut(); state.pending_mod_evaluate.take().unwrap() }; pending_mod_evaluate.sender.send(Err( generic_error("Cannot evaluate module, because JavaScript execution has been terminated.") )).expect("Failed to send module evaluation error."); } else { assert!(status == v8::ModuleStatus::Errored); } receiver } Let's delve into the intricacies of the process, breaking it down into distinct steps: 1. **Obtaining the Module Handle**: To initiate the process, the first step involves acquiring the module handle using the assigned module identifier. This handle serves as a way to interact with the specific module under consideration. 2. **Invoking v8's Module Evaluate Function**: The subsequent action entails invoking the module evaluate function provided by v8. This function plays a pivotal role in evaluating and processing the module's contents. 3. **Verifying Immediate Evaluation Errors**: After the evaluation function is called, a crucial check is performed to identify any immediate errors that might have arisen during the evaluation process. This preliminary assessment ensures that the module is evaluated correctly and any errors are promptly identified. 4. **Fetching a Global Promise for Module's Scope**: As the evaluation progresses, a global promise is acquired to establish the scope of the current module. This promise encapsulates the asynchronous execution and encapsulation of the module's functionality. 5. **Storing the Global Promise**: The acquired global promise is then stored within a designated container known as "pending\_mod\_evaluate." This repository serves as a means to manage and keep track of promises associated with module evaluations. 6. **Providing a Return Path for Async Events**: Finally, to ensure a smooth flow of asynchronous events and facilitate the retrieval of asynchronous updates later, the process concludes by returning the receiver. This return path acts as a means to capture and process any events that may occur asynchronously during the module's evaluation. After submitting the module for evaluation, it's time to delve into the renowned event loop. Copy event_loop_result = self.run_event_loop(false) => { event_loop_result?; let maybe_result = receiver.await; maybe_result.expect("Module evaluation result not provided.") } [](https://choubey.gitbook.io/internals-of-deno/foundations/evaluate-module#event-loop) Event loop ------------------------------------------------------------------------------------------------------- Once you've sent the module for assessment, the JavaScript runtime steps into an asynchronous event loop. Think of this event loop as a busy worker that's responsible for managing different tasks. One of its important tasks is to keep an eye on the main promise, which is like a commitment made when you first introduced your code for evaluation. This event loop is made up of various components, each with its own role, but going into all the details right now might be a bit overwhelming. For our current focus, which is the simple "hello world" program, there are a couple of parts within the event loop that matter: 1. **Checking pending module evaluation:** This involves making sure that the module you submitted is being properly evaluated. The event loop ensures that the evaluation process is happening smoothly and without any hiccups. It's like a supervisor making sure the work is being done. 2. **Checking promise rejections:** In the world of programming, promises are used to represent actions that might take some time to complete. If something goes wrong during these actions, promises can be rejected. The event loop keeps a watchful eye on these promises to catch any rejections and handle them appropriately. It's like having someone who catches errors and helps fix them. So, you can think of the event loop as a diligent manager overseeing your code's progress, making sure evaluations are happening correctly and problems are taken care of. It's like the backstage crew of a show, ensuring everything runs smoothly for the main performance. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MPNyZNROAbp3lNYVVTJ%252F-MPNzHR_D9ubWXUs_v3x%252Fdeno%2520event%2520loop%2520pend%2520eval.png%3Falt%3Dmedia%26token%3Dbffeb9dc-c0f4-443d-87cd-e0b269904abf&width=768&dpr=4&quality=100&sign=411fd111&sv=2) The aspect of the event loop's functioning that involves other tasks remains mysterious at this point. This is primarily due to the simplicity of our current example. In this scenario, there are no tasks that involve asynchronous calls, intricate dynamic imports, operations (referred to as "ops"), event listeners, and similar complexities. As a result, the hello world program commences and concludes on its own. It doesn't rely on any external dependencies to maintain its execution. The global promise that is linked with the main module encapsulates the outcome of evaluating the hello world program. It's worth highlighting that the program's outcome isn't equivalent to its actual output. The visible output of the program is directed to a distinct location. Conversely, the assessment's outcome is fed back into the event loop via the global promise. The following is the code of a single tick of the event loop: Copy pub fn poll_event_loop( &mut self, cx: &mut Context, wait_for_inspector: bool, ) -> Poll> { let has_inspector: bool; { let state = self.inner.state.borrow(); has_inspector = state.inspector.is_some(); state.op_state.borrow().waker.register(cx.waker()); } if has_inspector { // We poll the inspector first. let _ = self.inspector().borrow().poll_sessions(Some(cx)).unwrap(); } self.pump_v8_message_loop()?; // Dynamic module loading - ie. modules loaded using "import()" { // Run in a loop so that dynamic imports that only depend on another // dynamic import can be resolved in this event loop iteration. // // For example, a dynamically imported module like the following can be // immediately resolved after `dependency.ts` is fully evaluated, but it // wouldn't if not for this loop. // // await delay(1000); // await import("./dependency.ts"); // console.log("test") // // These dynamic import dependencies can be cross-realm: // // await delay(1000); // await new ShadowRealm().importValue("./dependency.js", "default"); // loop { let mut has_evaluated = false; let state = self.inner.state.borrow(); if state.known_realms.len() == 1 { drop(state); // Try and resolve as many dynamic imports in each realm as possible // before moving to the next. let realm = self.inner.main_realm.as_ref().unwrap(); loop { let poll_imports = realm.prepare_dyn_imports(&mut self.inner.v8_isolate, cx)?; assert!(poll_imports.is_ready()); let poll_imports = realm.poll_dyn_imports(&mut self.inner.v8_isolate, cx)?; assert!(poll_imports.is_ready()); if realm.evaluate_dyn_imports(&mut self.inner.v8_isolate) { has_evaluated = true; } else { break; } } } else { // TODO(bartlomieju|mmastrac): Remove cloning in the runtime loop let realms = state.known_realms.clone(); drop(state); for inner_realm in realms { let realm = JsRealm::new(inner_realm); // Try and resolve as many dynamic imports in each realm as possible // before moving to the next. loop { let poll_imports = realm.prepare_dyn_imports(&mut self.inner.v8_isolate, cx)?; assert!(poll_imports.is_ready()); let poll_imports = realm.poll_dyn_imports(&mut self.inner.v8_isolate, cx)?; assert!(poll_imports.is_ready()); if realm.evaluate_dyn_imports(&mut self.inner.v8_isolate) { has_evaluated = true; } else { break; } } } } if !has_evaluated { break; } } } // Resolve async ops, run all next tick callbacks and macrotasks callbacks // and only then check for any promise exceptions (`unhandledrejection` // handlers are run in macrotasks callbacks so we need to let them run // first). let dispatched_ops = self.do_js_event_loop_tick(cx)?; self.check_promise_rejections()?; // Event loop middlewares let mut maybe_scheduling = false; { let op_state = self.inner.state.borrow().op_state.clone(); for f in &self.event_loop_middlewares { if f(op_state.clone(), cx) { maybe_scheduling = true; } } } // Top level module { let state = self.inner.state.borrow(); if state.known_realms.len() == 1 { drop(state); let realm = self.inner.main_realm.as_ref().unwrap(); realm.evaluate_pending_module(&mut self.inner.v8_isolate); } else { // TODO(bartlomieju|mmastrac): Remove cloning in the runtime loop let realms = state.known_realms.clone(); drop(state); for inner_realm in realms { let realm = JsRealm::new(inner_realm); realm.evaluate_pending_module(&mut self.inner.v8_isolate); } } } let pending_state = self.event_loop_pending_state(); if !pending_state.is_pending() && !maybe_scheduling { if has_inspector { let inspector = self.inspector(); let has_active_sessions = inspector.borrow().has_active_sessions(); let has_blocking_sessions = inspector.borrow().has_blocking_sessions(); if wait_for_inspector && has_active_sessions { // If there are no blocking sessions (eg. REPL) we can now notify // debugger that the program has finished running and we're ready // to exit the process once debugger disconnects. if !has_blocking_sessions { let context = self.main_context(); let scope = &mut self.handle_scope(); inspector.borrow_mut().context_destroyed(scope, context); println!("Program finished. Waiting for inspector to disconnect to exit the process..."); } return Poll::Pending; } } return Poll::Ready(Ok(())); } let state = self.inner.state.borrow(); // Check if more async ops have been dispatched // during this turn of event loop. // If there are any pending background tasks, we also wake the runtime to // make sure we don't miss them. // TODO(andreubotella) The event loop will spin as long as there are pending // background tasks. We should look into having V8 notify us when a // background task is done. if pending_state.has_pending_background_tasks || pending_state.has_tick_scheduled || maybe_scheduling { state.op_state.borrow().waker.wake(); } // If ops were dispatched we may have progress on pending modules that we should re-check if (pending_state.has_pending_module_evaluation || pending_state.has_pending_dyn_module_evaluation) && dispatched_ops { state.op_state.borrow().waker.wake(); } drop(state); if pending_state.has_pending_module_evaluation { if pending_state.has_pending_refed_ops || pending_state.has_pending_dyn_imports || pending_state.has_pending_dyn_module_evaluation || pending_state.has_pending_background_tasks || pending_state.has_tick_scheduled || maybe_scheduling { // pass, will be polled again } else { let known_realms = &self.inner.state.borrow().known_realms; return Poll::Ready(Err( find_and_report_stalled_level_await_in_any_realm( &mut self.inner.v8_isolate, known_realms, ), )); } } if pending_state.has_pending_dyn_module_evaluation { if pending_state.has_pending_refed_ops || pending_state.has_pending_dyn_imports || pending_state.has_pending_background_tasks || pending_state.has_tick_scheduled { // pass, will be polled again } else if self.inner.state.borrow().dyn_module_evaluate_idle_counter >= 1 { let known_realms = &self.inner.state.borrow().known_realms; return Poll::Ready(Err( find_and_report_stalled_level_await_in_any_realm( &mut self.inner.v8_isolate, known_realms, ), )); } else { let mut state = self.inner.state.borrow_mut(); // Delay the above error by one spin of the event loop. A dynamic import // evaluation may complete during this, in which case the counter will // reset. state.dyn_module_evaluate_idle_counter += 1; state.op_state.borrow().waker.wake(); } } Poll::Pending } fn event_loop_pending_state(&mut self) -> EventLoopPendingState { let mut scope = v8::HandleScope::new(self.inner.v8_isolate.as_mut()); EventLoopPendingState::new(&mut scope, &mut self.inner.state.borrow_mut()) } } In the Deno runtime, during each cycle of the event loop, numerous tasks are carried out. As we discussed earlier, in the context of the simple "hello world" program, the crucial aspect to note is the execution of the function call known as "evaluate\_pending\_module." When a module is being evaluated, there's a possibility of encountering errors. These evaluation errors manifest asynchronously, typically as promise rejections. These errors adhere to the standard JavaScript error types, such as ReferenceError or TypeError, among others. To manage this, in every iteration of the event loop, the global promise pool is examined until it transitions from a pending state. This transition might involve moving to either a resolved or a rejected state, depending on the outcome of the asynchronous operations associated with the module evaluations. Here is the code snippet responsible for the process of monitoring promises linked to pending evaluations: Copy pub(in crate::runtime) fn evaluate_pending_module( &self, isolate: &mut v8::Isolate, ) { let maybe_module_evaluation = self .0 .context_state .borrow_mut() .pending_mod_evaluate .take(); if maybe_module_evaluation.is_none() { return; } let mut module_evaluation = maybe_module_evaluation.unwrap(); let state_rc = self.0.state(); let scope = &mut self.handle_scope(isolate); let promise_global = module_evaluation.promise.clone().unwrap(); let promise = promise_global.open(scope); let promise_state = promise.state(); match promise_state { v8::PromiseState::Pending => { // NOTE: `poll_event_loop` will decide if // runtime would be woken soon state_rc.borrow_mut().pending_mod_evaluate = Some(module_evaluation); } v8::PromiseState::Fulfilled => { scope.perform_microtask_checkpoint(); // Receiver end might have been already dropped, ignore the result let _ = module_evaluation.sender.send(Ok(())); module_evaluation.handled_promise_rejections.clear(); } v8::PromiseState::Rejected => { let exception = promise.result(scope); scope.perform_microtask_checkpoint(); // Receiver end might have been already dropped, ignore the result if module_evaluation .handled_promise_rejections .contains(&promise_global) { let _ = module_evaluation.sender.send(Ok(())); module_evaluation.handled_promise_rejections.clear(); } else { let _ = module_evaluation .sender .send(exception_to_err_result(scope, exception, false)); } } } } Within Deno, the global promise goes through three distinct states: 1. **Pending** * At this stage, the promise is in a waiting state. * It's like a question that hasn't been answered yet. * We'll come back and see if there's an answer in the next cycle. 2. **Fulfilled** * When a promise reaches this state, it means the task assigned to it has been successfully completed. * Think of it as getting a response to your question, and that response is positive. * If you were waiting for some information, now you have it, and you can share it with others. 3. **Rejected** * Unfortunately, not all promises end up fulfilled. Some end up in the rejected state. * This is like getting an answer, but it's not the answer you wanted. Something went wrong. * Just as sometimes things don't go as planned, the task associated with the promise couldn't be accomplished successfully. * In this case, you get an error message instead of the expected result. ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MPRnNgLluKkcTtHmTfg%252F-MPRoDKrPESxsLAxHcbW%252Fdeno%2520global%2520promise%2520states.png%3Falt%3Dmedia%26token%3Da4657014-0a87-4c83-82b2-4a94b0592f20&width=768&dpr=4&quality=100&sign=1c20693a&sv=2) After the global promise has been either fulfilled or rejected, the evaluation of the module finishes. The event loop comes to an end when the following conditions are satisfied: * There are no pending operations left. * There are no pending dynamic imports. * There are no dynamic imports awaiting evaluation. * There are no top-level modules waiting to be evaluated. This signifies that the entire process of evaluating the module and handling asynchronous operations has been completed. The event loop, which is responsible for managing the execution flow, can gracefully conclude its operations once all these requirements are met. Copy if pending_state.has_pending_module_evaluation { if pending_state.has_pending_refed_ops || pending_state.has_pending_dyn_imports || pending_state.has_pending_dyn_module_evaluation || pending_state.has_pending_background_tasks || pending_state.has_tick_scheduled || maybe_scheduling { // pass, will be polled again } else { // Finish eventloop } } In brief, when all tasks have been completed, the event loop concludes its operations, leading to the program's termination. In essence, if no further tasks remain, both the event loop and the program itself come to an end. [](https://choubey.gitbook.io/internals-of-deno/foundations/evaluate-module#evaluation-of-hello-world) Evaluation of hello world ------------------------------------------------------------------------------------------------------------------------------------- Let's take a closer look at how the evaluation process functions in our uncomplicated "hello world" illustration. Just to refresh our memory, here's the JavaScript code we're working with: Copy "use strict"; function printNumber(input) { console.log(input); } function printString(input) { console.log(input); } printNumber(1); printString('One'); The current example we're looking at involves a situation where no external modules are being imported. In this case, we're dealing with a singular module located at the root or main level. This main module takes center stage for evaluation; no other modules are considered. Additionally, there are no asynchronous function calls or any other activities that would cause it to continue running. Let's get into the sequence of events when the process of evaluating our main module, which we'll refer to as "mod\_evaluate," takes place. This process unfolds as follows: 1. The V8 engine, which powers Deno, initiates the execution of the main module. 2. Within the V8 engine, the instruction to run `console.log` is executed. 3. V8, in turn, triggers an external reference named "op\_print" to display the number 1 on the standard output (stdout). 4. Moving on, V8 performs another `console.log` operation. 5. Once again, V8 employs the external reference "op\_print," this time to exhibit the word "One" on the standard output. 6. As the program lacks any further instructions, it reaches its conclusion. 7. The evaluation of the module by V8 comes to an end. 8. V8 designates the global promise of the module as fulfilled, indicating successful completion. 9. Given that there are no pending tasks or activities, the subsequent tick of the event loop transpires. 10. The global promise, having met its fulfillment, wraps up. 11. Consequently, the asynchronous polling concludes, culminating in the completion of the module's execution. This step-by-step breakdown illustrates how the main module's journey unfolds during the evaluation process within the Deno environment. **Output:** Copy 1 One After the module's execution has been completed, there remain additional tasks that the worker must attend to. Once the `execute_main_module` function wraps up its operations, it proceeds to the subsequent stages within the implementation of the run command: Copy self.worker.dispatch_unload_event(located_script_name!())?; if let Some(coverage_collector) = maybe_coverage_collector.as_mut() { self .worker .with_event_loop(coverage_collector.stop_collecting().boxed_local()) .await?; } Ok(self.worker.exit_code()) * If there are any listeners, the window load event is sent out. * The program enters the event loop. * However, it's important to note that the main module has already been executed. * The event loop closes immediately because there are no other tasks to perform or wait for. * The window unload event is sent out. * This marks the completion of the program. * Deno then shuts down. Here's what remains to be done: The tasks that are left will be completed right away since the primary module has already been executed. The program will come to an end once these four functions are finished. Even the process of running the event loop will conclude promptly, as there are no more tasks remaining. \-- That's fantastic! We have successfully executed our inaugural program. [Previous5.16 Instantiate module](https://choubey.gitbook.io/internals-of-deno/foundations/instantiate-module) [Next5.18 What's next](https://choubey.gitbook.io/internals-of-deno/foundations/4.18-whats-next) Last updated 1 year ago --- # 7.2 Local storage | The Internals of Deno By default, Deno provides access to local storage for persistently storing data that survives Deno runtime restarts. The data is stored in a file on the disk. The local storage implementation doesn't support saving data remotely. This problem has been solved by Deno KV, which will be topic of the next book revision. The local storage (and session storage too) is internally supported by SQLite. The users would never know or see SQLite. For them, it is a simple & local KV storage. The local storage mechanism is a simple KV store that reside on the disk. An overview of the local storage architecture is as follows: ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F-MJJDXLU1fV3Te4epBgE%252Fuploads%252FeWycrY03OzJCcBL7f0T8%252Fdeno%2520localstorage%2520arch.png%3Falt%3Dmedia%26token%3D5bda7788-258b-4ded-8c79-bebac351c423&width=768&dpr=4&quality=100&sign=e9e4c106&sv=2) The user applications call the local storage APIs provided by Deno. The local storage APIs in JS space are supported by the OPs in rust space. The local storage APIs in rust, in turn, calls the SQLite APIs, which works with the database file on the disk. In this section, we'll learn: * The location of the database * The queries used to work with the database For all the interactions with the SQLite database, Deno uses rusqlite crate. [](https://choubey.gitbook.io/internals-of-deno/chapter-7-local-and-session-storage/7.2-local-storage#location-of-the-database) Location of the database ------------------------------------------------------------------------------------------------------------------------------------------------------------- The default location of the SQLite database that would hold the data for local storage is: * On linux, the location of database is `$HOME/.cache/deno` * On Windows, the location of database is `%LOCALAPPDATA%/deno` * On Mac, the location of database is `$HOME/Library/Caches/deno` In short, the location of the database is inside the cache folder that is used by Deno for other purposes. Unless used at least once, Deno doesn't create any database file. There is no point in creating database files on the disk if the applications on that server have never or are never going to use it. Copy ~/Library/Caches/deno/location_data: ls -l total 0 As soon as any of the storage API is called, Deno ensures that a database exists. This is achieved through the following function (get\_webstorage): Copy fn get_webstorage( state: &mut OpState, persistent: bool, ) -> Result<&Connection, AnyError> { let conn = if persistent { if state.try_borrow::().is_none() { let path = state.try_borrow::().ok_or_else(|| { DomExceptionNotSupportedError::new( "LocalStorage is not supported in this context.", ) })?; std::fs::create_dir_all(&path.0)?; let conn = Connection::open(path.0.join("local_storage"))?; // Enable write-ahead-logging and tweak some other stuff. let initial_pragmas = " -- enable write-ahead-logging mode PRAGMA journal_mode=WAL; PRAGMA synchronous=NORMAL; PRAGMA temp_store=memory; PRAGMA page_size=4096; PRAGMA mmap_size=6000000; PRAGMA optimize; "; conn.execute_batch(initial_pragmas)?; conn.set_prepared_statement_cache_capacity(128); { let mut stmt = conn.prepare_cached( "CREATE TABLE IF NOT EXISTS data (key VARCHAR UNIQUE, value VARCHAR)", )?; stmt.execute(params![])?; } state.put(LocalStorage(conn)); } &state.borrow::().0 } else { if state.try_borrow::().is_none() { let conn = Connection::open_in_memory()?; { let mut stmt = conn.prepare_cached( "CREATE TABLE data (key VARCHAR UNIQUE, value VARCHAR)", )?; stmt.execute(params![])?; } state.put(SessionStorage(conn)); } &state.borrow::().0 }; Ok(conn) } This function takes the following actions: * Creates a database file * Establish a connection to the database * Apply some default settings * Create a table in the database The default name for the table is 'data'. This table contains exactly two columns: * key * value Both key and values are strings, as suggested by the web storage standards. The table is created only if it doesn't exist. If it exists, the table is left untouched. This ensures that the data present in the table remains intact. As soon as any of the local storage APIs are used, we'll see database files in the storage directories. If we use repl, there will be a storage directory for repl that gets used for all repl sessions. Copy $ deno Deno 1.40.2 exit using ctrl+d, ctrl+c, or close() REPL is running with all permissions allowed. To specify permissions, run `deno repl` with allow flags. > localStorage.getItem("abcd"); null Now, there is a directory that holds the database files (there are more than one). Copy ~/Library/Caches/deno/location_data: ls -ltr total 0 drwxr-xr-x 5 mayankc staff 160 Jan 27 18:11 92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14 ~/Library/Caches/deno/location_data: ls -l 92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14/ total 104 -rw-r--r-- 1 mayankc staff 4096 Jan 27 18:11 local_storage -rw-r--r-- 1 mayankc staff 32768 Jan 27 18:11 local_storage-shm -rw-r--r-- 1 mayankc staff 12392 Jan 27 18:11 local_storage-wal We can set some item in the local storage for repl, and try to get it in another repl session. Copy // REPL SESSION $ deno Deno 1.40.2 exit using ctrl+d, ctrl+c, or close() REPL is running with all permissions allowed. To specify permissions, run `deno repl` with allow flags. > localStorage.setItem("k1", "v1"); undefined > // STORAGE DIRECTORY ~/Library/Caches/deno/location_data: ls 92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14 // ANOTHER REPL SESSION $ deno Deno 1.40.2 exit using ctrl+d, ctrl+c, or close() REPL is running with all permissions allowed. To specify permissions, run `deno repl` with allow flags. > localStorage.getItem("k1"); "v1" > // STORAGE DIRECTORY ~/Library/Caches/deno/location_data: ls 92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14 [](https://choubey.gitbook.io/internals-of-deno/chapter-7-local-and-session-storage/7.2-local-storage#custom-databases) Custom databases --------------------------------------------------------------------------------------------------------------------------------------------- The local storage in the browser is defined as: _The_ `_**localStorage**_` _read-only property of the_ [`_window_`](https://developer.mozilla.org/en-US/docs/Web/API/Window) _interface allows you to access a_ [`_Storage_`](https://developer.mozilla.org/en-US/docs/Web/API/Storage) _object for the_ [`_Document_`](https://developer.mozilla.org/en-US/docs/Web/API/Document) _'s_ [_origin_](https://developer.mozilla.org/en-US/docs/Glossary/Origin) _; the stored data is saved across browser sessions._ This means that the local storage is tied to the origin of the URL in the browser's address bar. This also means that there is separate storage available for each origin. Deno takes the same concept forward and makes a different database available to different origins. However, there is no concept of address bar here. Therefore, Deno takes origin from two sources: * The main program name * The --location argument if specified This ensures that each application has their own database. Copy // storageTest.js const d = localStorage.getItem("k1"); console.log(d); // Test $ deno run storageTest.js null // Storage directory ~/Library/Caches/deno/location_data: ls -l total 0 drwxr-xr-x 3 mayankc staff 96 Jan 27 18:28 92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14 drwxr-xr-x 3 mayankc staff 96 Jan 27 18:29 ccd539c6c7447c6d3830b2816f029f5dda4a347e383a608fa7fef00e36432b29 As we can see, in addition to repl database, there is a new one specifically created for this application. A new database gets created for each application that attempts to use local storage. Copy // storageTest1.js const d = localStorage.getItem("k1"); console.log(d); // Test $ deno run storageTest1.js null // Storage directory ~/Library/Caches/deno/location_data: ls -l total 0 drwxr-xr-x 3 mayankc staff 96 Jan 27 18:31 845857c5ba27e9912cfc1de8b6f8261b86fd13e857aa12f044d59895ebac4ccd drwxr-xr-x 3 mayankc staff 96 Jan 27 18:28 92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14 drwxr-xr-x 3 mayankc staff 96 Jan 27 18:29 ccd539c6c7447c6d3830b2816f029f5dda4a347e383a608fa7fef00e36432b29 Within an application, there is a way to segregate data further by specifying the --location argument. This creates a separate database for each location, which is separate from the database that gets created if location is not specified. Each application can open and use a different database by utilizing the --location argument. Copy // Using location $ deno run --location https://deno.com storageTest.js null // Storage directory ~/Library/Caches/deno/location_data: ls -l total 0 drwxr-xr-x 3 mayankc staff 96 Jan 27 18:31 845857c5ba27e9912cfc1de8b6f8261b86fd13e857aa12f044d59895ebac4ccd drwxr-xr-x 3 mayankc staff 96 Jan 27 18:28 92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14 drwxr-xr-x 3 mayankc staff 96 Jan 27 18:29 ccd539c6c7447c6d3830b2816f029f5dda4a347e383a608fa7fef00e36432b29 drwxr-xr-x 3 mayankc staff 96 Jan 27 18:32 d1e195fe7f09a0f7760e7ebbe4e23ac702228747ddf77b771b3055c3e7e5d29f // Using a different location $ deno run --location https://denodeploy.com storageTest.js null // Storage directory ~/Library/Caches/deno/location_data: ls -l total 0 drwxr-xr-x 3 mayankc staff 96 Jan 27 18:31 845857c5ba27e9912cfc1de8b6f8261b86fd13e857aa12f044d59895ebac4ccd drwxr-xr-x 3 mayankc staff 96 Jan 27 18:32 90a9a993d66b0c1d647ea90e4039ebd4733927f43fa583c3636f5a338be65a3e drwxr-xr-x 3 mayankc staff 96 Jan 27 18:28 92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14 drwxr-xr-x 3 mayankc staff 96 Jan 27 18:29 ccd539c6c7447c6d3830b2816f029f5dda4a347e383a608fa7fef00e36432b29 drwxr-xr-x 3 mayankc staff 96 Jan 27 18:32 d1e195fe7f09a0f7760e7ebbe4e23ac702228747ddf77b771b3055c3e7e5d29f That's all about the database files. We'll now turn our attention to the internals of a couple of heavily used local storage APIs: * getItem * setItem [](https://choubey.gitbook.io/internals-of-deno/chapter-7-local-and-session-storage/7.2-local-storage#get-item-from-local-storage) Get item from local storage ------------------------------------------------------------------------------------------------------------------------------------------------------------------- Getting an item from local storage is one of the two most used local storage APIs. The other one is setItem, which we'll see in the next section. This is a simple API with only one input: Key. We'll trace the journey starting from the user space. **JS space** The getItem API code is as follows: Copy getItem(key) { webidl.assertBranded(this, StoragePrototype); const prefix = "Failed to execute 'getItem' on 'Storage'"; webidl.requiredArguments(arguments.length, 1, prefix); key = webidl.converters.DOMString(key, prefix, "Argument 1"); return op_webstorage_get(key, this[_persistent]); } This is a very simple function that * Validates the presence of input key * Defers the work to a rust OP called op\_webstorage\_get **Rust space** The OP function called _op\_webstorage\_get_ is a synchronous function, that runs a query in the SQLite database and sends the result back. Note that the getItem and setItem are synchronous functions. Copy #[op2] #[string] pub fn op_webstorage_get( state: &mut OpState, #[string] key_name: String, persistent: bool, ) -> Result, AnyError> { let conn = get_webstorage(state, persistent)?; let mut stmt = conn.prepare_cached("SELECT value FROM data WHERE key = ?")?; let val = stmt .query_row(params![key_name], |row| row.get(0)) .optional()?; Ok(val) } The code of op\_webstorage\_get is much simpler than our expectations. This function: * Opens a connection to database (if not already open) * Prepares a query * Executes the query * Return the results back The important part here is the query, which is: Copy SELECT value FROM data WHERE key = ? As simple as it looks, that's all about getting an item from local storage. Simply run a query in the SQLite data and return the result. This is no different from running a query directly (which we'll do shortly). [](https://choubey.gitbook.io/internals-of-deno/chapter-7-local-and-session-storage/7.2-local-storage#set-item-in-the-local-storage) Set item in the local storage ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- Setting an item into local storage is the other of the two most used local storage APIs.This is a simple API with two inputs: Key & Value. Again, we'll trace the journey starting from the user space. **JS space** The setItem API code is as follows: Copy setItem(key, value) { webidl.assertBranded(this, StoragePrototype); const prefix = "Failed to execute 'setItem' on 'Storage'"; webidl.requiredArguments(arguments.length, 2, prefix); key = webidl.converters.DOMString(key, prefix, "Argument 1"); value = webidl.converters.DOMString(value, prefix, "Argument 2"); op_webstorage_set(key, value, this[_persistent]); } This is a very simple function that * Validates the presence of input key & value * Defers the work to a rust OP called op\_webstorage\_set **Rust space** The OP function called _op\_webstorage\_set_ is a synchronous function, that runs a DML operation in the SQLite database. Note that the getItem and setItem are synchronous functions. Copy #[op2(fast)] pub fn op_webstorage_set( state: &mut OpState, #[string] key: &str, #[string] value: &str, persistent: bool, ) -> Result<(), AnyError> { let conn = get_webstorage(state, persistent)?; size_check(key.len() + value.len())?; let mut stmt = conn .prepare_cached("SELECT SUM(pgsize) FROM dbstat WHERE name = 'data'")?; let size: u32 = stmt.query_row(params![], |row| row.get(0))?; size_check(size as usize)?; let mut stmt = conn .prepare_cached("INSERT OR REPLACE INTO data (key, value) VALUES (?, ?)")?; stmt.execute(params![key, value])?; Ok(()) } The code of op\_webstorage\_set is also much simpler than our expectations, but slightly longer than the get function. There are multiple size related checks. Mirroring browser implementations, Deno enforces a 10MB size limit on local storage per origin. This constraint ensures database integrity and prevents excessive storage consumption. The `op_webstorage_set` function diligently safeguards database sanity by preventing size thresholds from being exceeded. This op\_webstorage\_set function: * Opens a connection to database (if not already open) * Checks if the length of key & value is not more than 10MB * Checks if current database size is less than 10MB * Executes the DML operation to insert or update record in the database The important part here is the DML statement, which is: Copy INSERT OR REPLACE INTO data (key, value) VALUES (?, ?) As simple as it looks, that's all about setting an item into local storage. This API returns nothing back. [](https://choubey.gitbook.io/internals-of-deno/chapter-7-local-and-session-storage/7.2-local-storage#other-apis) Other APIs --------------------------------------------------------------------------------------------------------------------------------- Before closing this section, we'll take a quick at a couple of other interesting APIs: * removeItem * clear The removeItem API removes the record from the database. It issues a DELETE command to make this happen. Copy removeItem(key) { webidl.assertBranded(this, StoragePrototype); const prefix = "Failed to execute 'removeItem' on 'Storage'"; webidl.requiredArguments(arguments.length, 1, prefix); key = webidl.converters.DOMString(key, prefix, "Argument 1"); op_webstorage_remove(key, this[_persistent]); } Copy #[op2(fast)] pub fn op_webstorage_remove( state: &mut OpState, #[string] key_name: &str, persistent: bool, ) -> Result<(), AnyError> { let conn = get_webstorage(state, persistent)?; let mut stmt = conn.prepare_cached("DELETE FROM data WHERE key = ?")?; stmt.execute(params![key_name])?; Ok(()) } Lastly, the clear API removes all records from the table. This API also issues the DELETE command without any inputs. Copy clear() { webidl.assertBranded(this, StoragePrototype); op_webstorage_clear(this[_persistent]); } Copy #[op2(fast)] pub fn op_webstorage_clear( state: &mut OpState, persistent: bool, ) -> Result<(), AnyError> { let conn = get_webstorage(state, persistent)?; let mut stmt = conn.prepare_cached("DELETE FROM data")?; stmt.execute(params![])?; Ok(()) } [](https://choubey.gitbook.io/internals-of-deno/chapter-7-local-and-session-storage/7.2-local-storage#database-file) Database file --------------------------------------------------------------------------------------------------------------------------------------- Fulfilling our earlier promise and satisfying our curiosity, we'll briefly examine the database file within the SQLite shell. The local\_storage file, residing within any database folder in the root database directory, can be directly opened using the SQLite shell. Copy ~/Library/Caches/deno/location_data: ls -ltr total 0 drwxr-xr-x 5 mayankc staff 160 Jan 27 21:33 92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14 ~/Library/Caches/deno/location_data: cd 92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14/ ~/Library/Caches/deno/location_data/92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14: ls -ltr total 136 -rw-r--r-- 1 mayankc staff 4096 Jan 27 21:33 local_storage -rw-r--r-- 1 mayankc staff 32768 Jan 27 21:33 local_storage-shm -rw-r--r-- 1 mayankc staff 28872 Jan 27 21:33 local_storage-wal First, we'll set a couple of keys into the local storage using deno repl: Copy $ deno Deno 1.40.2 exit using ctrl+d, ctrl+c, or close() REPL is running with all permissions allowed. To specify permissions, run `deno repl` with allow flags. > localStorage.setItem('k1', 'v1'); undefined > localStorage.setItem('k2', 'v2'); undefined The file named local\_storage can be opened directly inside the SQLite shell: Copy ~/Library/Caches/deno/location_data/92dc35f73a8a47b9f4df517ad0744dd596ae537e9608d33461eb99891fd28f14: sqlite3 local_storage SQLite version 3.39.5 2022-10-14 20:58:05 Enter ".help" for usage hints. sqlite> .headers on sqlite> .schema data CREATE TABLE data (key VARCHAR UNIQUE, value VARCHAR); sqlite> select * from data; key|value k1|v1 k2|v2 sqlite> As you can see, this is the same type of SQLite database file that you may have used in the past. We've verified that a table named **"data"** exists with two keys that we set earlier. We've also verified that the table contains two rows matching the data we set through the Deno REPL. * * * This concludes our exploration of local storage internals in Deno. The next section is about session storage, building upon the detailed background we've established. The focus on session storage will be concise, leveraging your newly acquired understanding of local storage mechanisms. [Previous7.1 Introduction](https://choubey.gitbook.io/internals-of-deno/chapter-7-local-and-session-storage/7.1-introduction) [Next7.3 Session storage](https://choubey.gitbook.io/internals-of-deno/chapter-7-local-and-session-storage/7.3-session-storage) Last updated 1 year ago --- # 6.5 Registration and instantiation | The Internals of Deno [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/registration-and-instantiation#overview) Overview --------------------------------------------------------------------------------------------------------------------- All modules are prepared to enter v8. This occurs in two steps: 1. Compiling and registering the module. 2. Instantiating the module. Finally, they are loaded into v8, ready for execution or evaluation. [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/registration-and-instantiation#registration) Registration ----------------------------------------------------------------------------------------------------------------------------- Let's review how registration works because we have some imports in this example. This time, the process will be a bit longer since the imports also need to be loaded into V8. Here's a detailed explanation: 1. **load\_main\_module** The first step involves loading the main module. 2. **Recursive Loading and Preparation** After that, the process performs recursive loading and prepares the modules for registration. 3. **Loop Over Pending List and Module Registration** Next, there is a loop that goes through a list of pending modules and registers them. 4. **register\_and\_recurse** Within the registration process, there's a loop that handles both registration and recursion. 5. **Loop Over Imports** Another loop iterates over all the imports of a module. 6. **Adding Imports to Pending List** During this iteration, if an import is not yet registered, it is added to the pending list for later processing. The "load\_module" loop continues registering modules until the pending list becomes empty. Here is an abbreviated code of all the three functions (only import specific code is shown, rest is suppressed): Copy pub async fn load_main_module( &self, isolate: &mut v8::Isolate, specifier: &ModuleSpecifier, code: Option, ) -> Result { let module_map_rc = self.0.module_map(); if let Some(code) = code { let specifier = specifier.as_str().to_owned().into(); let scope = &mut self.handle_scope(isolate); // true for main module module_map_rc .borrow_mut() .new_es_module(scope, true, specifier, code, false) .map_err(|e| match e { ModuleError::Exception(exception) => { let exception = v8::Local::new(scope, exception); exception_to_err_result::<()>(scope, exception, false).unwrap_err() } ModuleError::Other(error) => error, })?; } let mut load = ModuleMap::load_main(module_map_rc.clone(), &specifier).await?; while let Some(load_result) = load.next().await { let (request, info) = load_result?; let scope = &mut self.handle_scope(isolate); load.register_and_recurse(scope, &request, info).map_err( |e| match e { ModuleError::Exception(exception) => { let exception = v8::Local::new(scope, exception); exception_to_err_result::<()>(scope, exception, false).unwrap_err() } ModuleError::Other(error) => error, }, )?; } let root_id = load.root_module_id.expect("Root module should be loaded"); self.instantiate_module(isolate, root_id).map_err(|e| { let scope = &mut self.handle_scope(isolate); let exception = v8::Local::new(scope, e); exception_to_err_result::<()>(scope, exception, false).unwrap_err() })?; Ok(root_id) } pub(crate) fn register_and_recurse( &mut self, scope: &mut v8::HandleScope, module_request: &ModuleRequest, module_source: ModuleSource, ) -> Result<(), ModuleError> { let expected_asserted_module_type = module_source.module_type.into(); let module_url_found = module_source.module_url_found; let module_url_specified = module_source.module_url_specified; if module_request.asserted_module_type != expected_asserted_module_type { return Err(ModuleError::Other(generic_error(format!( "Expected a \"{}\" module but loaded a \"{}\" module.", module_request.asserted_module_type, module_source.module_type, )))); } // Register the module in the module map unless it's already there. If the // specified URL and the "true" URL are different, register the alias. let module_url_found = if let Some(module_url_found) = module_url_found { let (module_url_found1, module_url_found2) = module_url_found.into_cheap_copy(); self.module_map_rc.borrow_mut().alias( module_url_specified, expected_asserted_module_type, module_url_found1, ); module_url_found2 } else { module_url_specified }; let maybe_module_id = self .module_map_rc .borrow() .get_id(&module_url_found, expected_asserted_module_type); let module_id = match maybe_module_id { Some(id) => { debug!( "Already-registered module fetched again: {:?}", module_url_found ); id } None => match module_source.module_type { ModuleType::JavaScript => { self.module_map_rc.borrow_mut().new_es_module( scope, self.is_currently_loading_main_module(), module_url_found, module_source.code, self.is_dynamic_import(), )? } ModuleType::Json => self.module_map_rc.borrow_mut().new_json_module( scope, module_url_found, module_source.code, )?, }, }; // Recurse the module's imports. There are two cases for each import: // 1. If the module is not in the module map, start a new load for it in // `self.pending`. The result of that load should eventually be passed to // this function for recursion. // 2. If the module is already in the module map, queue it up to be // recursed synchronously here. // This robustly ensures that the whole graph is in the module map before // `LoadState::Done` is set. let mut already_registered = VecDeque::new(); already_registered.push_back((module_id, module_request.clone())); self.visited.insert(module_request.clone()); while let Some((module_id, module_request)) = already_registered.pop_front() { let referrer = ModuleSpecifier::parse(&module_request.specifier).unwrap(); let imports = self .module_map_rc .borrow() .get_requested_modules(module_id) .unwrap() .clone(); for module_request in imports { if !self.visited.contains(&module_request) && !self .visited_as_alias .borrow() .contains(&module_request.specifier) { if let Some(module_id) = self.module_map_rc.borrow().get_id( module_request.specifier.as_str(), module_request.asserted_module_type, ) { already_registered.push_back((module_id, module_request.clone())); } else { let request = module_request.clone(); let specifier = ModuleSpecifier::parse(&module_request.specifier).unwrap(); let visited_as_alias = self.visited_as_alias.clone(); let referrer = referrer.clone(); let loader = self.loader.clone(); let is_dynamic_import = self.is_dynamic_import(); let fut = async move { // `visited_as_alias` unlike `visited` is checked as late as // possible because it can only be populated after completed // loads, meaning a duplicate load future may have already been // dispatched before we know it's a duplicate. if visited_as_alias.borrow().contains(specifier.as_str()) { return Ok(None); } let load_result = loader .load(&specifier, Some(&referrer), is_dynamic_import) .await; if let Ok(source) = &load_result { if let Some(found_specifier) = &source.module_url_found { visited_as_alias .borrow_mut() .insert(found_specifier.as_str().to_string()); } } load_result.map(|s| Some((request, s))) }; self.pending.push(fut.boxed_local()); } self.visited.insert(module_request); } } } // Update `self.state` however applicable. if self.state == LoadState::LoadingRoot { self.root_module_id = Some(module_id); self.root_asserted_module_type = Some(module_source.module_type.into()); self.state = LoadState::LoadingImports; } if self.pending.is_empty() { self.state = LoadState::Done; } Ok(()) } } The process is pretty simple. It's similar to creating the module graph. But, to ensure everything is covered, we'll review all the steps that lead to registering modules for the hello world v2 program. #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/registration-and-instantiation#step-1) Step 1 Compile and register module `file:///Users/mayankc/Work/source/denoExamples/helloV2.ts`. ID Module 1 file:///Users/mayankc/Work/source/denoExamples/helloLogV2.ts #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/registration-and-instantiation#step-2) Step 2 Go through imports and add them to the pending list * add `https://deno.land/x/machine_id/mod.ts` to the pending list * add `npm:nanoid` to the pending list #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/registration-and-instantiation#step-3) Step 3 Compile and register module `https://deno.land/x/machine_id/mod.ts`. ID Module 2 https://deno.land/x/machine\_id/mod.ts There are further external dependencies from machine\_id module. #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/registration-and-instantiation#step-4) Step 4 Compile and register module `https://deno.land/x/machine_id/mod.ts`. ID Module 3 npm:nanoid or file:///Users/mayankc/Work/source/denoExamples/node\_modules/.deno/nanoid@4.0.2/node\_modules/nanoid/index.js #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/registration-and-instantiation#step-5) Step 5 Go through imports and add them to the pending list * add file:///Users/mayankc/Work/source/denoExamples/node\_modules/.deno/nanoid@4.0.2/node\_modules/nanoid/url-alphabet/index.js to the pending list #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/registration-and-instantiation#step-6) Step 6 Compile and register module file:///Users/mayankc/Work/source/denoExamples/node\_modules/.deno/nanoid@4.0.2/node\_modules/nanoid/url-alphabet/index.js. ID Module 4 file:///Users/mayankc/Work/source/denoExamples/node\_modules/.deno/nanoid@4.0.2/node\_modules/nanoid/url-alphabet/index.js #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/registration-and-instantiation#step-7) Step 7 There are no further imports, nothing gets added to the pending list. #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/registration-and-instantiation#step-8) Step 8 Since the pending list is empty, the registration process is complete. All the modules that were imported from the main module have been compiled and registered. It's crucial to understand that these modules have been loaded separately into V8, without any connection between them for now. The actual connection will occur when they are instantiated. [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/registration-and-instantiation#instantiation) Instantiation ------------------------------------------------------------------------------------------------------------------------------- Regardless of how many imports or modules there are, instantiation only occurs in the root or main module. To remind you, here is the relevant code where "instantiate" is called. Copy let root_id = load.root_module_id.expect("Root module should be loaded"); self.instantiate_module(isolate, root_id).map_err(|e| { let scope = &mut self.handle_scope(isolate); let exception = v8::Local::new(scope, e); exception_to_err_result::<()>(scope, exception, false).unwrap_err() })?; Ok(root_id) In the previous chapter, we learned about the mod\_instantiate function. This function helps set up a module in v8, which is like a place where JavaScript code runs. In the last chapter's hello world program, we didn't need any callbacks because that program didn't use any external code. But things are a bit different now. In this chapter, we're working with modules that have imports. An import is like a way to bring in code from another module. When we have imports, the v8 engine wants us to provide a callback to help it figure out where to find the imported modules. Here's how it works: 1. We start by creating the main module. 2. For each import in each module: * V8 asks us for a module handle using a callback. * In this callback, V8 tells us which module it wants to find and the module that's asking for it. This helps V8 connect the dots correctly. * This process continues, going deeper into each module, until all the modules are taken care of. It's like solving a puzzle one piece at a time. Each module is a piece, and V8 needs us to tell it where to find the pieces it's missing. This whole process happens in a loop, like a chain reaction, until everything is set up. Remember, it's a looping process, so we don't have to set up each module individually. We just start with the main module, and V8 takes care of the rest, step by step. Now, let's see how all of this works with our updated hello world v2 program. #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/registration-and-instantiation#step-1-1) Step 1 Instantiate main module id=1. #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/registration-and-instantiation#step-2-1) Step 2 Receive callback to resolve: Resolve Referrer npm:nanoid file:///Users/mayankc/Work/source/denoExamples/helloV2.ts #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/registration-and-instantiation#step-3-1) Step 3 Receive callback to resolve: Resolve Referrer https://deno.land/x/machine\_id/mod.ts file:///Users/mayankc/Work/source/denoExamples/helloV2.ts #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/registration-and-instantiation#step-4-1) Step 4 Receive callback to resolve: Resolve Referrer crypto https://deno.land/x/machine\_id/mod.ts Yes, even built-in modules like crypto needs a resolve callback. They're built-in for us, but not for V8. #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/registration-and-instantiation#step-5-1) Step 5 Receive callback to resolve: Resolve Referrer crypto https://deno.land/x/machine\_id/mod.ts #### [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/registration-and-instantiation#step-6-1) Step 6 Receive callback to resolve: Resolve Referrer ./url-alphabet/index.js npm:nanoid or file:///Users/mayankc/Work/source/denoExamples/node\_modules/.deno/nanoid@4.0.2/node\_modules/nanoid/index.js \-- After all the import callbacks are successfully resolved, v8 completes the module setup. Both the main module and its imports are prepared and set up inside v8 through a recursive process. They are now set to be evaluated. Before we move on to evaluation, let's understand how operations (ops) are registered. This is important because our hello world v2 program includes both synchronous and asynchronous ops. We've touched upon the ops registration briefly in a previous chapter. Before we proceed to execute the program, let's take a closer look at this registration process. [Previous6.4 Transpile](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.4-transpile) [Next6.6 Registration of ops](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.6-registration-of-ops) Last updated 1 year ago --- # 6.6 Registration of ops | The Internals of Deno Operations, often referred to as "Ops," play a crucial role in Deno. Ops act as the primary connectors between JavaScript and Rust, effectively forming a bridge between the two. This interaction happens back and forth, allowing seamless communication. Ops are essential because V8 strictly adheres to the ECMAScript specification, the standard for JavaScript. Any functionality that goes beyond this specification must be provided as an external reference. Even fundamental actions like using `console.log` require an external op call, as the ECMAScript specification doesn't encompass console output. In the previous chapter, we explored how external references are registered with V8. We will now quickly revisit that concept. Additionally, we will look into built-in and ext ops. This section will specifically focus on the registration process for ops. Later on, we will examine how ops are triggered during runtime and how the system manages their responses. [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.6-registration-of-ops#overview) Overview -------------------------------------------------------------------------------------------------------------- In short: _**Ops are low-level functions that are implemented in Rust to support the high-level functions in JS.**_ For example, `crypto.randomUUID` is a high-level function, and the corresponding Rust op is implemented by `op_crypto_random_uuid`. The high-level JS function is; Copy randomUUID() { webidl.assertBranded(this, CryptoPrototype); return ops.op_crypto_random_uuid(); } The corresponding low-level OP is: Copy pub fn op_crypto_random_uuid(state: &mut OpState) -> Result { let maybe_seeded_rng = state.try_borrow_mut::(); let uuid = if let Some(seeded_rng) = maybe_seeded_rng { let mut bytes = [0u8; 16]; seeded_rng.fill(&mut bytes); uuid::Builder::from_bytes(bytes) .with_version(uuid::Version::Random) .into_uuid() } else { uuid::Uuid::new_v4() }; Ok(uuid.to_string()) } In Deno, operations (OPs) typically have a one-to-one mapping. This means that one JavaScript API directly corresponds to one Rust operation (OP). All the operations (OPs) follow a consistent pattern, from the initial setup to processing the result. The following outlines the typical phases of an operation (OP): ![](https://choubey.gitbook.io/internals-of-deno/~gitbook/image?url=https%3A%2F%2F4136075195-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-legacy-files%2Fo%2Fassets%252F-MJJDXLU1fV3Te4epBgE%252F-MPav7XJwz3aI_P27nnq%252F-MPawLEO0a-yGyzmgkiY%252Fdeno%2520op%2520phases.png%3Falt%3Dmedia%26token%3Dfdb6b2e5-6ea6-4fe0-8070-f6f3079fef98&width=768&dpr=4&quality=100&sign=3da0297f&sv=2) Deno operations progress through various stages. The diagram above illustrates the standard operation state machine. Let's examine each state: 1. **Register handler** * In this step, an operation handler is registered. 2. **Waiting to get called** * This state resembles an idle condition where the operation awaits a call. 3. **Dispatch** * Dispatch involves moving across the bridge from V8 to Deno. 4. **Execute** * Deno executes the operation's handler. 5. **Process result** * The outcome is sent back to V8, where it undergoes processing in JavaScript. * This step is akin to returning from the bridge. The process of dispatching and result processing happens during runtime. We will discuss the details later. For now, our focus is on registering operations. [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.6-registration-of-ops#types-of-ops) Types of Ops ---------------------------------------------------------------------------------------------------------------------- All the operations can be divided into two main categories: 1. _Sync Operations_: * In sync operations, the request and response are processed one after the other, in a step-by-step manner. * During the execution of sync operations, the process is blocked, meaning it waits for each step to complete before moving forward. 2. _Async Operations_: * Async operations, on the other hand, handle requests and responses asynchronously, which means they can process multiple tasks concurrently. * The execution of async operations is not blocked, allowing the program to continue with other tasks while waiting for certain operations to finish. Certain operations fall distinctly into either of these categories, while some operations can belong to both. For instance, operations like getting and setting environment variables are categorized as sync operations. Another example involves reading a file, which can involve both sync and async operations, depending on the specific operation and the user's choice. The suitability of async operations varies based on the nature of the operation. Some operations are better suited for sync processing, and thus only have sync variants. On the other hand, for operations like reading a file, the decision to use sync or async mode is up to the user. If dealing with a particularly large file, the user might opt for async reading to improve efficiency. [](https://choubey.gitbook.io/internals-of-deno/import-and-ops/5.6-registration-of-ops#registration) Registration ---------------------------------------------------------------------------------------------------------------------- Op registration happens at startup. Deno registers all of its low-level ops right at the process startup. There two sources of OPs: * _Built-in OPs_: These OPs are part of Deno core. These cover the most basic & fundamental OPs. Some examples are - op\_print, op\_resources, op\_read, op\_write, op\_encode, op\_decode, etc. * _EXT OPs_: These OPs come from Deno's ext modules. As we know that the ext modules are a close combination of JS APIs alongwith their low-level OPs. The OPs coming from EXT usually have the 1:1 mapping from JS API to the OP. Some examples are - op\_fetch, op\_crypto\_sign\_key, op\_crypto\_get\_random\_values, etc. The code in the main worker below registers all the OPs with V8 as external references: Copy pub fn from_options( main_module: ModuleSpecifier, permissions: PermissionsContainer, mut options: WorkerOptions, ) -> Self { deno_core::extension!(deno_permissions_worker, options = { permissions: PermissionsContainer, unstable: bool, enable_testing_features: bool, }, state = |state, options| { state.put::(options.permissions); state.put(ops::UnstableChecker { unstable: options.unstable }); state.put(ops::TestingFeaturesEnabled(options.enable_testing_features)); }, ); // Permissions: many ops depend on this let unstable = options.bootstrap.unstable; let enable_testing_features = options.bootstrap.enable_testing_features; let exit_code = ExitCode(Arc::new(AtomicI32::new(0))); let create_cache = options.cache_storage_dir.map(|storage_dir| { let create_cache_fn = move || SqliteBackedCache::new(storage_dir.clone()); CreateCache(Arc::new(create_cache_fn)) }); // NOTE(bartlomieju): ordering is important here, keep it in sync with // `runtime/build.rs`, `runtime/web_worker.rs` and `cli/build.rs`! let mut extensions = vec![\ // Web APIs\ deno_webidl::deno_webidl::init_ops_and_esm(),\ deno_console::deno_console::init_ops_and_esm(),\ deno_url::deno_url::init_ops_and_esm(),\ deno_web::deno_web::init_ops_and_esm::(\ options.blob_store.clone(),\ options.bootstrap.location.clone(),\ ),\ deno_fetch::deno_fetch::init_ops_and_esm::(\ deno_fetch::Options {\ user_agent: options.bootstrap.user_agent.clone(),\ root_cert_store_provider: options.root_cert_store_provider.clone(),\ unsafely_ignore_certificate_errors: options\ .unsafely_ignore_certificate_errors\ .clone(),\ file_fetch_handler: Rc::new(deno_fetch::FsFetchHandler),\ ..Default::default()\ },\ ),\ deno_cache::deno_cache::init_ops_and_esm::(\ create_cache,\ ),\ deno_websocket::deno_websocket::init_ops_and_esm::(\ options.bootstrap.user_agent.clone(),\ options.root_cert_store_provider.clone(),\ options.unsafely_ignore_certificate_errors.clone(),\ ),\ deno_webstorage::deno_webstorage::init_ops_and_esm(\ options.origin_storage_dir.clone(),\ ),\ deno_crypto::deno_crypto::init_ops_and_esm(options.seed),\ deno_broadcast_channel::deno_broadcast_channel::init_ops_and_esm(\ options.broadcast_channel.clone(),\ unstable,\ ),\ deno_ffi::deno_ffi::init_ops_and_esm::(unstable),\ deno_net::deno_net::init_ops_and_esm::(\ options.root_cert_store_provider.clone(),\ unstable,\ options.unsafely_ignore_certificate_errors.clone(),\ ),\ deno_tls::deno_tls::init_ops_and_esm(),\ deno_kv::deno_kv::init_ops_and_esm(\ MultiBackendDbHandler::remote_or_sqlite::(\ options.origin_storage_dir.clone(),\ ),\ unstable,\ ),\ deno_napi::deno_napi::init_ops_and_esm::(),\ deno_http::deno_http::init_ops_and_esm::(),\ deno_io::deno_io::init_ops_and_esm(Some(options.stdio)),\ deno_fs::deno_fs::init_ops_and_esm::(\ unstable,\ options.fs.clone(),\ ),\ deno_node::deno_node::init_ops_and_esm::(\ options.npm_resolver,\ options.fs,\ ),\ // Ops from this crate\ ops::runtime::deno_runtime::init_ops_and_esm(main_module.clone()),\ ops::worker_host::deno_worker_host::init_ops_and_esm(\ options.create_web_worker_cb.clone(),\ options.format_js_error_fn.clone(),\ ),\ ops::fs_events::deno_fs_events::init_ops_and_esm(),\ ops::os::deno_os::init_ops_and_esm(exit_code.clone()),\ ops::permissions::deno_permissions::init_ops_and_esm(),\ ops::process::deno_process::init_ops_and_esm(),\ ops::signal::deno_signal::init_ops_and_esm(),\ ops::tty::deno_tty::init_ops_and_esm(),\ ops::http::deno_http_runtime::init_ops_and_esm(),\ deno_permissions_worker::init_ops_and_esm(\ permissions,\ unstable,\ enable_testing_features,\ ),\ runtime::init_ops_and_esm(),\ ]; for extension in &mut extensions { #[cfg(not(feature = "__runtime_js_sources"))] { extension.js_files = std::borrow::Cow::Borrowed(&[]); extension.esm_files = std::borrow::Cow::Borrowed(&[]); extension.esm_entry_point = None; } #[cfg(feature = "__runtime_js_sources")] { for source in extension.esm_files.to_mut() { maybe_transpile_source(source).unwrap(); } for source in extension.js_files.to_mut() { maybe_transpile_source(source).unwrap(); } } } extensions.extend(std::mem::take(&mut options.extensions)); #[cfg(all(feature = "include_js_files_for_snapshotting", feature = "dont_create_runtime_snapshot", not(feature = "__runtime_js_sources")))] options.startup_snapshot.as_ref().expect("Sources are not embedded, snapshotting was disabled and a user snapshot was not provided."); // Clear extension modules from the module map, except preserve `node:*` // modules. let preserve_snapshotted_modules = Some(SUPPORTED_BUILTIN_NODE_MODULES_WITH_PREFIX); let mut js_runtime = JsRuntime::new(RuntimeOptions { module_loader: Some(options.module_loader.clone()), startup_snapshot: options .startup_snapshot .or_else(crate::js::deno_isolate_init), create_params: options.create_params, source_map_getter: options.source_map_getter, get_error_class_fn: options.get_error_class_fn, shared_array_buffer_store: options.shared_array_buffer_store.clone(), compiled_wasm_module_store: options.compiled_wasm_module_store.clone(), extensions, preserve_snapshotted_modules, inspector: options.maybe_inspector_server.is_some(), is_main: true, ..Default::default() }); if let Some(server) = options.maybe_inspector_server.clone() { server.register_inspector( main_module.to_string(), &mut js_runtime, options.should_break_on_first_statement || options.should_wait_for_inspector_session, ); // Put inspector handle into the op state so we can put a breakpoint when // executing a CJS entrypoint. let op_state = js_runtime.op_state(); let inspector = js_runtime.inspector(); op_state.borrow_mut().put(inspector); } let bootstrap_fn_global = { let context = js_runtime.main_context(); let scope = &mut js_runtime.handle_scope(); let context_local = v8::Local::new(scope, context); let global_obj = context_local.global(scope); let bootstrap_str = v8::String::new_external_onebyte_static(scope, b"bootstrap").unwrap(); let bootstrap_ns: v8::Local = global_obj .get(scope, bootstrap_str.into()) .unwrap() .try_into() .unwrap(); let main_runtime_str = v8::String::new_external_onebyte_static(scope, b"mainRuntime").unwrap(); let bootstrap_fn = bootstrap_ns.get(scope, main_runtime_str.into()).unwrap(); let bootstrap_fn = v8::Local::::try_from(bootstrap_fn).unwrap(); v8::Global::new(scope, bootstrap_fn) }; Self { js_runtime, should_break_on_first_statement: options.should_break_on_first_statement, should_wait_for_inspector_session: options .should_wait_for_inspector_session, exit_code, bootstrap_fn_global: Some(bootstrap_fn_global), } } Let's look at the master function that register all the Ops with V8: Copy pub(crate) fn external_references( ops: &[OpCtx], additional_references: &[v8::ExternalReference], ) -> v8::ExternalReferences { // Overallocate a bit, it's better than having to resize the vector. let mut references = Vec::with_capacity(4 + (ops.len() * 4) + additional_references.len()); references.push(v8::ExternalReference { function: call_console.map_fn_to(), }); references.push(v8::ExternalReference { function: import_meta_resolve.map_fn_to(), }); references.push(v8::ExternalReference { function: catch_dynamic_import_promise_error.map_fn_to(), }); references.push(v8::ExternalReference { function: empty_fn.map_fn_to(), }); for ctx in ops { let ctx_ptr = ctx as *const OpCtx as _; references.push(v8::ExternalReference { pointer: ctx_ptr }); references.push(v8::ExternalReference { function: ctx.decl.v8_fn_ptr, }); if let Some(fast_fn) = &ctx.decl.fast_fn { references.push(v8::ExternalReference { pointer: fast_fn.function as _, }); references.push(v8::ExternalReference { pointer: ctx.fast_fn_c_info.unwrap().as_ptr() as _, }); } } references.extend_from_slice(additional_references); let refs = v8::ExternalReferences::new(&references); // Leak, V8 takes ownership of the references. std::mem::forget(references); refs } Although the code of OPs is in Rust, they're invoked from JS space. Therefore, each OP has a corresponding registration function in JS too. This also comes from Deno core. Copy Deno.__op__registerOp = function (isAsync, op, opName) { const core = Deno.core; if (isAsync) { if (core.ops[opName] !== undefined) { return; } core.asyncOps[opName] = op; const fn = function (...args) { if (this !== core.ops) { // deno-lint-ignore prefer-primordials throw new Error( "An async stub cannot be separated from Deno.core.ops. Use ???", ); } return core.asyncStub(opName, args); }; fn.name = opName; core.ops[opName] = fn; } else { core.ops[opName] = op; } }; The call to registerOp called from Rust code: Copy pub(crate) fn initialize_context<'s>( scope: &mut v8::HandleScope<'s>, context: v8::Local<'s, v8::Context>, op_ctxs: &[OpCtx], init_mode: InitMode, ) -> v8::Local<'s, v8::Context> { let global = context.global(scope); let mut codegen = String::with_capacity(op_ctxs.len() * 200); codegen.push_str(include_str!("bindings.js")); _ = writeln!( codegen, "Deno.__op__ = function(opFns, callConsole, console) {{" ); if init_mode == InitMode::New { _ = writeln!(codegen, "Deno.__op__console(callConsole, console);"); } for op_ctx in op_ctxs { if op_ctx.decl.enabled { _ = writeln!( codegen, "Deno.__op__registerOp({}, opFns[{}], \"{}\");", op_ctx.decl.is_async, op_ctx.id, op_ctx.decl.name ); } else { _ = writeln!( codegen, "Deno.__op__unregisterOp({}, \"{}\");", op_ctx.decl.is_async, op_ctx.decl.name ); } } codegen.push_str("Deno.__op__cleanup();"); _ = writeln!(codegen, "}}"); let script = v8::String::new_from_one_byte( scope, codegen.as_bytes(), v8::NewStringType::Normal, ) .unwrap(); let script = v8::Script::compile(scope, script, None).unwrap(); script.run(scope); let deno = get(scope, global, b"Deno", "Deno"); let op_fn: v8::Local = get(scope, deno, b"__op__", "Deno.__op__"); let recv = v8::undefined(scope); let op_fns = v8::Array::new(scope, op_ctxs.len() as i32); for op_ctx in op_ctxs { let op_fn = op_ctx_function(scope, op_ctx); op_fns.set_index(scope, op_ctx.id as u32, op_fn.into()); } if init_mode == InitMode::FromSnapshot { op_fn.call(scope, recv.into(), &[op_fns.into()]); } else { // Bind functions to Deno.core.* let call_console_fn = v8::Function::new(scope, call_console).unwrap(); // Bind v8 console object to Deno.core.console let extra_binding_obj = context.get_extras_binding_object(scope); let console_obj: v8::Local = get( scope, extra_binding_obj, b"console", "ExtrasBindingObject.console", ); op_fn.call( scope, recv.into(), &[op_fns.into(), call_console_fn.into(), console_obj.into()], ); } context } At the time of writing, there are 478 ops that get registered. Here is the complete list: Copy op_close op_try_close op_print op_resources op_wasm_streaming_feed op_wasm_streaming_set_url op_void_sync op_error_async op_error_async_deferred op_void_async op_void_async_deferred op_add op_add_async op_read op_read_all op_write op_read_sync op_write_sync op_write_all op_write_type_error op_shutdown op_metrics op_format_file_name op_is_proxy op_str_byte_length op_ref_op op_unref_op op_set_promise_reject_callback op_run_microtasks op_has_tick_scheduled op_set_has_tick_scheduled op_eval_context op_queue_microtask op_create_host_object op_encode op_decode op_serialize op_deserialize op_set_promise_hooks op_get_promise_details op_get_proxy_details op_get_non_index_property_names op_get_constructor_name op_memory_usage op_set_wasm_streaming_callback op_abort_wasm_streaming op_destructure_error op_dispatch_exception op_op_names op_apply_source_map op_set_format_exception_callback op_event_loop_has_more_work op_store_pending_promise_rejection op_remove_pending_promise_rejection op_has_pending_promise_rejection op_arraybuffer_was_detached op_url_reparse op_url_parse op_url_get_serialization op_url_parse_with_base op_url_parse_search_params op_url_stringify_search_params op_urlpattern_parse op_urlpattern_process_match_input op_base64_decode op_base64_encode op_base64_atob op_base64_btoa op_encoding_normalize_label op_encoding_decode_single op_encoding_decode_utf8 op_encoding_new_decoder op_encoding_decode op_encoding_encode_into op_encode_binary_string op_blob_create_part op_blob_slice_part op_blob_read_part op_blob_remove_part op_blob_create_object_url op_blob_revoke_object_url op_blob_from_object_url op_message_port_create_entangled op_message_port_post_message op_message_port_recv_message op_compression_new op_compression_write op_compression_finish op_now op_timer_handle op_cancel_handle op_sleep op_transfer_arraybuffer op_readable_stream_resource_allocate op_readable_stream_resource_get_sink op_readable_stream_resource_write_error op_readable_stream_resource_write_buf op_readable_stream_resource_close op_readable_stream_resource_await_close op_fetch op_fetch_send op_fetch_response_upgrade op_fetch_custom_client op_cache_storage_open op_cache_storage_has op_cache_storage_delete op_cache_put op_cache_put_finish op_cache_match op_cache_delete op_ws_check_permission_and_cancel_handle op_ws_create op_ws_close op_ws_next_event op_ws_get_buffer op_ws_get_buffer_as_string op_ws_get_error op_ws_send_binary op_ws_send_text op_ws_send_binary_async op_ws_send_text_async op_ws_send_ping op_ws_send_pong op_ws_get_buffered_amount op_webstorage_length op_webstorage_key op_webstorage_set op_webstorage_get op_webstorage_remove op_webstorage_clear op_webstorage_iterate_keys op_crypto_get_random_values op_crypto_generate_key op_crypto_sign_key op_crypto_verify_key op_crypto_derive_bits op_crypto_import_key op_crypto_export_key op_crypto_encrypt op_crypto_decrypt op_crypto_subtle_digest op_crypto_random_uuid op_crypto_wrap_key op_crypto_unwrap_key op_crypto_base64url_decode op_crypto_base64url_encode op_crypto_generate_x25519_keypair op_crypto_derive_bits_x25519 op_crypto_import_spki_x25519 op_crypto_import_pkcs8_x25519 op_crypto_generate_ed25519_keypair op_crypto_import_spki_ed25519 op_crypto_import_pkcs8_ed25519 op_crypto_sign_ed25519 op_crypto_verify_ed25519 op_crypto_export_spki_ed25519 op_crypto_export_pkcs8_ed25519 op_crypto_jwk_x_ed25519 op_crypto_export_spki_x25519 op_crypto_export_pkcs8_x25519 op_broadcast_subscribe op_broadcast_unsubscribe op_broadcast_send op_broadcast_recv op_ffi_load op_ffi_get_static op_ffi_call_nonblocking op_ffi_call_ptr op_ffi_call_ptr_nonblocking op_ffi_ptr_create op_ffi_ptr_equals op_ffi_ptr_of op_ffi_ptr_offset op_ffi_ptr_value op_ffi_get_buf op_ffi_buf_copy_into op_ffi_cstr_read op_ffi_read_bool op_ffi_read_u8 op_ffi_read_i8 op_ffi_read_u16 op_ffi_read_i16 op_ffi_read_u32 op_ffi_read_i32 op_ffi_read_u64 op_ffi_read_i64 op_ffi_read_f32 op_ffi_read_f64 op_ffi_read_ptr op_ffi_unsafe_callback_create op_ffi_unsafe_callback_close op_ffi_unsafe_callback_ref op_net_accept_tcp op_net_connect_tcp op_net_listen_tcp op_net_listen_udp op_node_unstable_net_listen_udp op_net_recv_udp op_net_send_udp op_net_join_multi_v4_udp op_net_join_multi_v6_udp op_net_leave_multi_v4_udp op_net_leave_multi_v6_udp op_net_set_multi_loopback_udp op_net_set_multi_ttl_udp op_dns_resolve op_set_nodelay op_set_keepalive op_tls_start op_net_connect_tls op_net_listen_tls op_net_accept_tls op_tls_handshake op_net_accept_unix op_net_connect_unix op_net_listen_unix op_net_listen_unixpacket op_node_unstable_net_listen_unixpacket op_net_recv_unixpacket op_net_send_unixpacket op_kv_database_open op_kv_snapshot_read op_kv_atomic_write op_kv_encode_cursor op_kv_dequeue_next_message op_kv_finish_dequeued_message op_napi_open op_http_accept op_http_headers op_http_shutdown op_http_upgrade_websocket op_http_websocket_accept_header op_http_write_headers op_http_write_resource op_http_write op_http_get_request_header op_http_get_request_headers op_http_get_request_method_and_url op_http_read_request_body op_http_serve_on op_http_serve op_http_set_promise_complete op_http_set_response_body_bytes op_http_set_response_body_resource op_http_set_response_body_text op_http_set_response_header op_http_set_response_headers op_http_set_response_trailers op_http_track op_http_upgrade_websocket_next op_http_upgrade_raw op_raw_write_vectored op_can_write_vectored op_http_try_wait op_http_wait op_fs_cwd op_fs_umask op_fs_chdir op_fs_open_sync op_fs_open_async op_fs_mkdir_sync op_fs_mkdir_async op_fs_chmod_sync op_fs_chmod_async op_fs_chown_sync op_fs_chown_async op_fs_remove_sync op_fs_remove_async op_fs_copy_file_sync op_fs_copy_file_async op_fs_stat_sync op_fs_stat_async op_fs_lstat_sync op_fs_lstat_async op_fs_realpath_sync op_fs_realpath_async op_fs_read_dir_sync op_fs_read_dir_async op_fs_rename_sync op_fs_rename_async op_fs_link_sync op_fs_link_async op_fs_symlink_sync op_fs_symlink_async op_fs_read_link_sync op_fs_read_link_async op_fs_truncate_sync op_fs_truncate_async op_fs_utime_sync op_fs_utime_async op_fs_make_temp_dir_sync op_fs_make_temp_dir_async op_fs_make_temp_file_sync op_fs_make_temp_file_async op_fs_write_file_sync op_fs_write_file_async op_fs_read_file_sync op_fs_read_file_async op_fs_read_file_text_sync op_fs_read_file_text_async op_fs_seek_sync op_fs_seek_async op_fs_fdatasync_sync op_fs_fdatasync_async op_fs_fsync_sync op_fs_fsync_async op_fs_fstat_sync op_fs_fstat_async op_fs_flock_sync op_fs_flock_async op_fs_funlock_sync op_fs_funlock_async op_fs_ftruncate_sync op_fs_ftruncate_async op_fs_futime_sync op_fs_futime_async op_node_create_decipheriv op_node_cipheriv_encrypt op_node_cipheriv_final op_node_create_cipheriv op_node_create_hash op_node_get_hashes op_node_decipheriv_decrypt op_node_decipheriv_final op_node_hash_update op_node_hash_update_str op_node_hash_digest op_node_hash_digest_hex op_node_hash_clone op_node_private_encrypt op_node_private_decrypt op_node_public_encrypt op_node_check_prime op_node_check_prime_async op_node_check_prime_bytes op_node_check_prime_bytes_async op_node_gen_prime op_node_gen_prime_async op_node_pbkdf2 op_node_pbkdf2_async op_node_hkdf op_node_hkdf_async op_node_generate_secret op_node_generate_secret_async op_node_sign op_node_generate_rsa op_node_generate_rsa_async op_node_dsa_generate op_node_dsa_generate_async op_node_ec_generate op_node_ec_generate_async op_node_ed25519_generate op_node_ed25519_generate_async op_node_x25519_generate op_node_x25519_generate_async op_node_dh_generate_group op_node_dh_generate_group_async op_node_dh_generate op_node_dh_generate2 op_node_dh_compute_secret op_node_dh_generate_async op_node_verify op_node_random_int op_node_scrypt_sync op_node_scrypt_async op_node_ecdh_generate_keys op_node_ecdh_compute_secret op_node_ecdh_compute_public_key op_node_x509_parse op_node_x509_ca op_node_x509_check_email op_node_x509_fingerprint op_node_x509_fingerprint256 op_node_x509_fingerprint512 op_node_x509_get_issuer op_node_x509_get_subject op_node_x509_get_valid_from op_node_x509_get_valid_to op_node_x509_get_serial_number op_node_x509_key_usage op_node_sys_to_uv_error op_v8_cached_data_version_tag op_v8_get_heap_statistics op_node_idna_domain_to_ascii op_node_idna_domain_to_unicode op_node_idna_punycode_decode op_node_idna_punycode_encode op_zlib_new op_zlib_close op_zlib_close_if_pending op_zlib_write op_zlib_write_async op_zlib_init op_zlib_reset op_brotli_compress op_brotli_compress_async op_create_brotli_compress op_brotli_compress_stream op_brotli_compress_stream_end op_brotli_decompress op_brotli_decompress_async op_create_brotli_decompress op_brotli_decompress_stream op_brotli_decompress_stream_end op_node_http_request op_node_os_get_priority op_node_os_set_priority op_node_os_username op_node_build_os op_is_any_arraybuffer op_node_is_promise_rejected op_require_init_paths op_require_node_module_paths op_require_proxy_path op_require_is_deno_dir_package op_require_resolve_deno_dir op_require_is_request_relative op_require_resolve_lookup_paths op_require_try_self_parent_path op_require_try_self op_require_real_path op_require_path_is_absolute op_require_path_dirname op_require_stat op_require_path_resolve op_require_path_basename op_require_read_file op_require_as_file_path op_require_resolve_exports op_require_read_closest_package_json op_require_read_package_scope op_require_package_imports_resolve op_require_break_on_next_statement op_main_module op_ppid op_create_worker op_host_terminate_worker op_host_post_message op_host_recv_ctrl op_host_recv_message op_fs_events_open op_fs_events_poll op_env op_exec_path op_exit op_delete_env op_get_env op_gid op_hostname op_loadavg op_network_interfaces op_os_release op_os_uptime op_node_unstable_os_uptime op_set_env op_set_exit_code op_system_memory_info op_uid op_runtime_memory_usage op_query_permission op_revoke_permission op_request_permission op_spawn_child op_spawn_wait op_spawn_sync op_spawn_kill op_run op_run_status op_kill op_signal_bind op_signal_unbind op_signal_poll op_stdin_set_raw op_isatty op_console_size op_http_start op_http_upgrade op_npm_process_state Once again, the call to OP happens like this: Copy randomUUID() { webidl.assertBranded(this, CryptoPrototype); return ops.op_crypto_random_uuid(); } \-- That was all about ops registration. Let's move on to module evaluation. [Previous6.5 Registration and instantiation](https://choubey.gitbook.io/internals-of-deno/import-and-ops/registration-and-instantiation) [Next6.7 Evaluate module](https://choubey.gitbook.io/internals-of-deno/import-and-ops/evaluate-module) Last updated 1 year ago --- # 1.0 Cover page | The Internals of Deno 1.0 Cover page | The Internals of Deno --- # 2.0 Cover page | The Internals of Deno 2.0 Cover page | The Internals of Deno --- # 7.0 Cover page | The Internals of Deno 7.0 Cover page | The Internals of Deno --- # 5.0 Cover page | The Internals of Deno 5.0 Cover page | The Internals of Deno --- # 4.0 Cover page | The Internals of Deno 4.0 Cover page | The Internals of Deno --- # 3.0 Cover page | The Internals of Deno 3.0 Cover page | The Internals of Deno --- # 6.0 Cover page | The Internals of Deno 6.0 Cover page | The Internals of Deno --- # Afterword | The Internals of Deno Afterword | The Internals of Deno ---