# Table of Contents
- [Turnkey Whitepaper](#turnkey-whitepaper)
- [Turnkey Whitepaper](#turnkey-whitepaper)
- [Turnkey Whitepaper](#turnkey-whitepaper)
- [Turnkey Whitepaper](#turnkey-whitepaper)
- [Turnkey Whitepaper](#turnkey-whitepaper)
---
# Turnkey Whitepaper
* [Index](https://whitepaper.turnkey.com/)
* [Principles](https://whitepaper.turnkey.com/principles)
* [Foundations](https://whitepaper.turnkey.com/foundations)
* [Architecture](https://whitepaper.turnkey.com/architecture)
* [Applications](https://whitepaper.turnkey.com/applications)
Turnkey: a Verifiable Key Management Solution
=============================================
Turnkey Team
January 2025
Turnkey is the first verifiable key management system of its kind. This live system has operated for 2+ years and secures millions of wallets and private keys for a wide variety of use cases including embedded wallets, smart contract deployments, payments, treasury management, and AI deployments.
Structure of this whitepaper
----------------------------
There is a lot to explain so we've split the content in multiple documents to make it easier to consume.
#### [Key Management Re-imagined from First Principles](https://whitepaper.turnkey.com/principles/)
This document details Turnkey's drastically different vision. After explaining why we think now is the time to re-imagine key management, we talk about our ambitious security model, our bet on asymmetric cryptography for authentication, and our belief that curve-level operations are the correct building blocks for modern crypto-asset operations. By the end of this document you'll understand the motivations behind Turnkey and should be ready to jump to our technical design with that context in mind.
#### [Verifiable Foundations](https://whitepaper.turnkey.com/foundations/)
This document explains the foundations on which Turnkey is built. We'll see what Trusted Execution Environments (“TEEs”) are and how we use them. We'll introduce QuorumOS (“QOS”), a new minimal, open-source operating system engineered for verifiability which runs inside of all Turnkey enclaves. We also introduce StageX, a new Linux distro which solves the reproducibility problem and supports all secure builds at Turnkey today. Finally we'll see how these components prove the software running inside of TEEs all the way down to the application source code.
#### [Turnkey's Architecture](https://whitepaper.turnkey.com/architecture/)
We offer an engineering-focused tour of our system as it is running today, with a focus on the enclave applications Turnkey runs internally. We'll see the challenges that emerge from running on top of TEEs, the main one being: TEEs do not store state. We'll also talk about our API design and digital signature scheme for authentication. By the end of this document you'll have a full understanding of Turnkey's design and the crucial role that enclave applications play within it.
#### [Applications Beyond Key Management](https://whitepaper.turnkey.com/applications/)
In this last document we focus on what's possible to build today and tomorrow. While Turnkey has chosen key management as the first class of applications on top of its verifiable foundations, we envision it to be relevant well beyond that. We'll explain why we think QuorumOS is a valuable platform to run any application, provisioned with centralized or decentralized quorum sets, held by humans, servers or AI agents.
Acknowledgements
----------------
I’d like to sincerely thank Jack Kearney, Bryce Ferguson, Hannah Arnold, Michael Avrukin, Sarah Lu, Zane Kharitonov, Carolyn Philip, Raheel Ahmed, Samuel Ebstein, Hao Su, Andrew Min, Mark Nesbitt, Brian Esler, Lance Vick, Seán McCord, Robin Arenson, Mohammed Odeh, and Amir Cheikh for their invaluable feedback, thoughtful insights, and unwavering support throughout the writing of this whitepaper. Their suggestions have significantly enhanced its clarity, precision, and depth.
Beyond those named, I am especially grateful for the big ideas that exist out in the open, waiting to be captured, shaped, and implemented. Thank you to the anonymous and generous minds who spend their time and energy sharing knowledge publicly. That spirit of openness and collaboration has been instrumental in bringing Turnkey, and this whitepaper, to life. I hope others are inspired to build upon what is presented here in the same way.
_Arnaud Brousseau, Founding Engineer, Turnkey_
---
# Turnkey Whitepaper
* [Index](https://whitepaper.turnkey.com/)
* [Principles](https://whitepaper.turnkey.com/principles)
* [Foundations](https://whitepaper.turnkey.com/foundations)
* [Architecture](https://whitepaper.turnkey.com/architecture)
* [Applications](https://whitepaper.turnkey.com/applications)
Applications beyond Key Management
==================================
Turnkey Team
January 2025
[](https://whitepaper.turnkey.com/applications/#abstract "Copy link to this section")
Abstract
------------------------------------------------------------------------------------------------------------------------------------------------
While Turnkey has chosen key management as the first application on top of its verifiable foundations, we envision it to be relevant well beyond that.
First, we highlight some compelling use-cases and products built on top of Turnkey as it stands today and explain why they benefit from the foundations we provide.
Next, we imagine what could be unlocked by adding to or enhancing Turnkey. We look at what happens when QuorumOS can run on any secure hardware equipped with a TPM, when our policy engine is expanded a few steps beyond where it is today, when the flexibility of Quorum Sets is leveraged to its fullest extent, or when builders can write their own enclave applications running on Turnkey's [Verifiable Foundation](https://whitepaper.turnkey.com/foundations/)
.
Finally we discuss what decentralizing Turnkey could look like. We envision that tasks such as reviewing critical source code and its dependencies, reproducing secure builds, and providing compute resources can be incentivized with common crypto-native mechanisms. A lot of public goods can come out of this. As an industry we must demand better tools to thwart supply chain risks. We believe a decentralized, censorship-resistant Turnkey will result in safer foundations to build critical software going forward, in or out of crypto. Security is a universal, growing need for all software developers who care about their end-users.
[](https://whitepaper.turnkey.com/applications/#what-turnkey-solves-today "Copy link to this section")
What Turnkey solves today
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
In this section we zoom out to explain what Turnkey solves today. We conclude this section with a list of products built on top of Turnkey and why these applications benefit from Turnkey's offering.
### [](https://whitepaper.turnkey.com/applications/#offchain-smart-contract-like-authorization "Copy link to this section")
Offchain smart-contract-like authorization
One under-appreciated aspect of the Turnkey Policy Engine is its chain-agnostic language. For example, requiring two or more approvers for Turnkey activities is similar to an on-chain multi-sig setup where multiple keys share ownership of blockchain funds, except the functionality is chain-agnostic. If there are no smart contracts or multisig capabilities available (or if they are prohibitively expensive to use), Turnkey fills this gap.
There are also privacy and upgradability advantages to using Turnkey: modifying Turnkey policies (to require three approvers instead of two, for example) does not require any on-chain action. It is cheap and completely private. A Turnkey wallet looks like any other wallet on-chain, even though it may be controlled by multiple parties underneath. This concept is talked about at length in [Liquefaction](https://arxiv.org/abs/2412.02634)
, where the authors introduce the concept of “key encumbrance” to describe the enforcement of “an access control policy over the key’s full lifecycle”. This is possible with Turnkey given the flexibility of its Policy Engine. A User or set of Users can be granted granular access to a wallet without having full knowledge of the underlying private key.
This off-chain authorization composes very well with existing on-chain primitives. For example, Ethereum Account Abstraction Wallets (using [EIP 4337](https://www.erc4337.io/)
, also known as AA Wallets) can designate Externally Owned Accounts (EOAs) as signers. We see Turnkey customers use Turnkey-controlled EOAs for this purpose to minimize the number of on-chain actions and enhance privacy when users need to add or remove authenticators. Another big advantage of a Turnkey-controlled EOA is that it can be updated once, within Turnkey, and used across many Ethereum L2s. Creating AA wallets across many L2s would otherwise require many on-chain actions upon credential update [1](https://whitepaper.turnkey.com/applications/#1)
(one per AA wallet)
### [](https://whitepaper.turnkey.com/applications/#cross-chain-tooling-escrow-swaps-and-more "Copy link to this section")
Cross-chain tooling: escrow, swaps, and more
Concrete use-cases that benefit from flexible and cross-chain authorization are escrow and swaps platforms. With Turnkey it is possible to build a robust off-chain swap workflow. For example:
* Configure a new Turnkey Organization with two Root Users (Root Quorum configured with 2-out-of-2) and two distinct new wallets. These two users are the users who want to swap assets.
* Both parties use consensus to create two policies which allow their individual User to unilaterally sign transactions for their own wallet. This is safe to do: the wallets are empty.
* Users send funds to their designated wallets. This is safe because they can claw back their funds if something goes wrong, thanks to the policies introduced in the previous step.
* Using consensus, users create 4 new policies such that they lose access to their original wallet and gain access to their new (swapped) wallet at the same time:
* The first two policies explicitly `DENY` signing access to the original owners, negating the policies already in place (`DENY` takes precedence over `ALLOW`)
* The other two policies `ALLOW` unilateral signing access for the new owners
* This action is atomic: `CREATE_POLICIES` is a single activity on Turnkey which either succeeds in creating all 4 policies at once, or fails.
* The two parties then transfer funds out of Turnkey at their own convenience. They know that no further critical action can be taken on this Turnkey Organization because of the Root Quorum setting.
This swap workflow doesn't depend on chain-specific functionality. As long as Turnkey supports the underlying cryptographic curve, swapping is possible. One can also envision a verifiable bridge built on Turnkey using these types of primitives. Because Turnkey is programmatic, the parties swapping assets do not have to be humans. Machines or agents can use it in the same way.
### [](https://whitepaper.turnkey.com/applications/#non-custodial-wallets-and-keys "Copy link to this section")
Non-custodial wallets and keys
Non-custodial wallets and keys are a special case of offchain authorization, but deserve to be highlighted because it is such a common use-case: how can a business create wallets or keys for its end-users without being exposed to the private keys themselves or be in a position to unilaterally use them? Turnkey solves this problem with Sub-Organizations and the flexibility of the Policy Engine. As outlined in [Sub-Organizations as Wallets](https://docs.turnkey.com/embedded-wallets/sub-organizations-as-wallets)
, a Sub-Organization can be structured with exclusive access (the end-user is the only Root User, ensuring complete control and ownership) or joint access (the business and the end-user have control over one Root User each, and they each must sign Turnkey Activities for them to be processed). Of course, Turnkey also allows for traditional custodial setups where a business creates and controls Sub-Organizations on behalf of their end-users programmatically.
### [](https://whitepaper.turnkey.com/applications/#scoping-permissions-for-ai-agents "Copy link to this section")
Scoping permissions for AI agents
As with every credential, developers should follow the [Principle of Least Privilege](https://en.wikipedia.org/wiki/Principle_of_least_privilege)
when building applications powered by agents — in most cases, an agent should be capable of taking specific, limited actions with a crypto wallet instead of having complete control. Turnkey's policy engine is a perfect tool for this.
Using Turnkey, an application developer can generate an API key that has scoped permissions. Specifically, an API key can be permissioned to only initiate transactions that interact with specific contracts or trade against certain asset pairs. Furthermore, the policy engine can be used to require additional approvals on a signature request. These approvals can come from humans or other agents.
Additionally, as our policy engine functionality continues to grow, it can be leveraged as a defensive mechanism against an AI Agent taking action beyond its allowed risk scope, for example limiting the total number of transactions in a time span or account value as stop-gaps.
As of the time of writing, Turnkey has broad support for EVM- and Solana-specific operations. We plan to expand functionality as necessary within these ecosystems and into many others.
### [](https://whitepaper.turnkey.com/applications/#flexibility "Copy link to this section")
Flexibility
Our data model doesn't make assumptions about the relationship between resources (Wallets, Private Keys) and Users. Policies are the link between the two and as a result this relationship can take the shapes you need. As evidenced with the previous swap example, there are a lot of workflows unlocked by this flexibility.
We've broken down Turnkey's functionality in atomic Activities. These Activities have an “action” type as well as a “resource”, similar to REST semantics (see [this breakdown](https://docs.turnkey.com/concepts/policies/language#activity-breakdown)
). This allows the creation of more succinct policies. For example:
* A group of Users can create resources (Wallets, Private Keys) but not delete or use them
* Another group of Users (which could be machines or agents) is exclusively granted privilege to sign payloads with existing Wallets and Private Keys
* A third group is responsible for “disaster recovery”-style capabilities and is part of the Root Quorum of the organization
A Turnkey Organization can be home to:
* a sophisticated VC fund securing millions of dollars of crypto assets with a strict Root Quorum and many redundant authenticators for each Root User,
* a degen AI agent trading many low value memecoins as the sole Root User,
* or a DAO coordinating on-chain voting or upgrading smart contracts.
Your imagination is the limit.
### [](https://whitepaper.turnkey.com/applications/#verifiable-transaction-metadata-available-in-policies "Copy link to this section")
Verifiable transaction metadata available in Policies
For several ecosystems, Turnkey can parse unsigned transactions into metadata that can be leveraged inside of our policy engine (see [Ecosystem Integrations](https://docs.turnkey.com/documentation/ecosystem-integrations/)
). This is relevant for authorization when it requires context about the to-be-signed transaction.
A couple of examples where this is relevant:
* An organization wants to authorize a user or set of users to sign a transaction only if its amount is less than 10 SOL.
* An organization wants to authorize a trader to sign a transaction so long as it is a `swap` call to Uniswap.
* An organization wants to deny any transaction going to a list of known bad actors.
This need is most acute for payments integrations and wallet use-cases.
### [](https://whitepaper.turnkey.com/applications/#best-in-class-performance-and-scalability "Copy link to this section")
Best-in-class performance and scalability
As detailed in our [Architecture](https://whitepaper.turnkey.com/architecture/)
, the Signer enclave is responsible for producing signatures. When a fleet of signers is provisioned, they can process many signatures in parallel, at near-native speed, because the key is reconstructed in enclave memory. We thus unlock use-cases which would otherwise be hard or impossible to build with current on-chain or MPC solutions. Thanks to the verifiable nature of Turnkey's enclaves, security and performance aren't at odds with each other.
Turnkey is capable of scaling to millions of wallets and keys because it is horizontally scalable: a bigger fleet of Signer enclaves can process more signing Activities. It is possible to process signing Activities in parallel because signatures do not mutate organization data, as discussed in [Architecture](https://whitepaper.turnkey.com/architecture/)
. This is crucial to scale Turnkey going forward when we look to more complex setups where Turnkey runs in multiple geographies and jurisdictions.
### [](https://whitepaper.turnkey.com/applications/#import-and-export "Copy link to this section")
Import and Export
We've chosen to offer secure key import and export functionality in Turnkey because we believe that vendor lock-in is predatory. Both are implemented as Turnkey activities and secured with asymmetric cryptography: we perform a Diffie-Hellman key exchange between secure enclaves and user-held keys (following HPKE aka [RFC 9180](https://datatracker.ietf.org/doc/rfc9180/)
). This is critical for users who come in with existing wallets, or users who wish to take their wallets with them. More about this in our documentation: [Import](https://docs.turnkey.com/wallets/import-wallets)
and [Export](https://docs.turnkey.com/wallets/export-wallets)
.
### [](https://whitepaper.turnkey.com/applications/#built-with-turnkey "Copy link to this section")
Built with Turnkey
The table below contains a list of verticals and a few products using Turnkey. We highlight the main reason why they picked Turnkey as their foundation.
| | | |
| --- | --- | --- |
| **Vertical** | **Customers** | **What Turnkey solves** |
| Consumer Social | [Droppp](https://droppp.io/)
, [dub](http://crypto.social/) | Flexibility, performance (throughput) |
| Defi | [Infinex](https://infinex.xyz/)
, [Trojan](https://trojan.com/)
, [Moonshot](https://moonshot.money/)
, [Tria](https://www.tria.so/)
, [Azura](https://azura.xyz/) | Smooth UX with passkey auth, performance, cross-chain. |
| Dev tooling | [Alchemy](https://www.alchemy.com/)
, [Dynamic](https://www.dynamic.xyz/)
, [ZeroDev](https://zerodev.app/)
, [OneBalance](https://www.onebalance.io/) | Low-level abstractions, performance |
| Gaming | [Parallel](https://parallel.life/)
, [Faraway](https://faraway.com/) | Auth flexibility, API performance. |
| Music / NFTs | [Medallion](https://www.medallion.fm/)
, [Magic Eden](https://magiceden.us/?gr) | Flexibility |
| Payments | [Bridge](https://www.bridge.xyz/)
, [Mural](https://www.muralpay.com/)
, [Squads](https://squads.so/) | Security (policy engine), API performance and reliability |
| Protocols | [Mysten Labs](https://www.mystenlabs.com/)
, [Aptos](https://aptosfoundation.org/) | Security (policy engine) |
| DePIN | [DIMO](https://dimo.org/) | Flexible auth, security |
| RWAs | [Superstate](https://superstate.co/)
, [Heron Finance](https://www.heronfinance.com/)
, [Superform](https://www.superform.xyz/) | Security |
| Wallets | [TipLink](https://tiplink.io/)
, [Beranames](https://www.beranames.com/)
, [Brillion](https://www.brillion.finance/) | Multi-chain UX and security (policy engine) |
| AI agents | [Spectral](https://www.spectrallabs.xyz/) | Flexible API and authentication, bot-friendly, autonomous deployment |
| Prediction markets | [Polymarket](https://polymarket.com/) | Security (policy engine) |
[](https://whitepaper.turnkey.com/applications/#sensitive-workloads-beyond-key-management "Copy link to this section")
Sensitive workloads beyond key management
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
In this section we highlight what becomes possible with a few upgrades to Turnkey. Strap in!
### [](https://whitepaper.turnkey.com/applications/#verifiable-compute-platform "Copy link to this section")
Verifiable compute platform
Turnkey runs [QuorumOS](https://github.com/tkhq/qos)
internally as a solution to run applications in secure enclaves. Today we do not let users run their own applications. Only Turnkey-authored applications (Policy Engine, Signer, Notarizer, Parser, TLS Fetcher) are exposed to our customers.
We have plans to open up our foundations and let anyone provide StageX-built binaries and run them. This opens up the design space for builders massively: the key-management-specific applications remain usable, but others can be built. This goes beyond key management.
Some examples of sensitive workloads that could benefit from moving to attestable environments are in the table below.
| | |
| --- | --- |
| **Critical software** | **Advantages to verifiability** |
| AI Inference | Avoid [“Wizard of OZ”](https://x.com/hosseeb/status/1874288532686295058)
agents (who are human actors pretending to be real AI agents). |
| Chain abstraction | Trustworthy cosigners for cross-chain resource locks (e.g. [OneBalance](https://www.onebalance.io/)
). |
| Transaction construction | Users know that the unsigned transaction bytes are legitimate. |
| Transaction parsing | Provide accurate metadata about the effects of a transaction. This is critical for trusted wallet UX. |
| Oracles (data fetching) | Leverage external data without the overhead of full decentralization, on-chain or off-chain. Replace economic incentives with verifiability. |
| Blockchain nodes | Allow for private balance lookups, verifiable mempool inclusion, and more. |
| Blockchain L2 sequencers | Prove correct behavior and eliminate the need for challenge periods and economic incentives around them. |
| Identity verification | Prove no identity is leaked as part of the verification process. |
| VPN nodes | Guarantees privacy by proving that forwarded traffic isn't logged anywhere. |
| Exchanges | Ensure no malicious behavior (frontrunning), and create verifiable order books. |
| Web2 bridges | Prove the state of web2 (exchange balance, credit scores, X follower count) in web3. |
| PII Processing | Prove that processing does not leak or misuse PII (Personally Identifiable Information). |
### [](https://whitepaper.turnkey.com/applications/#tpm-2-0-support "Copy link to this section")
TPM 2.0 support
The Turnkey secure application stack is currently dependent on Nitro Enclaves but will soon target other TEEs. Many TPM 2.0 devices have direct network access, which means applications running on this type of hardware will have the ability to connect to remote hosts out-of-the-box. Any executable that targets Linux and can be built reproducibly will be able to be run directly into an attestable environment without modification.
With this in place, virtually all applications on the Internet would benefit from the increased transparency and verifiability afforded by the Turnkey stack.
### [](https://whitepaper.turnkey.com/applications/#expansion-of-the-policy-engine "Copy link to this section")
Expansion of the policy engine
The Turnkey policy engine is already best-in-class but we are planning to expand beyond where we are today to unlock new use cases and product features:
* Ability to fetch and expose external data such as price: enables notional value policies.
* Time locks: enables dead-man switch policies.
* Ability to provide a WASM blob run by the policy engine instead of writing in Turnkey's language: enables builders to leverage code they have already written.
* Deeper ecosystem integrations: today our Parser can parse EVM and Solana transactions. We will expand the number of ecosystems over time.
### [](https://whitepaper.turnkey.com/applications/#organization-specific-provisioning "Copy link to this section")
Organization-specific provisioning
Using the flexibility of Quorum Sets, control over enclave application deployment and provisioning can be distributed across geographies and jurisdiction. Turnkey is well-positioned to enable complex customer-specific setups in the near future.
We've seen in [Foundations](https://whitepaper.turnkey.com/foundations/)
that QuorumOS applications are deployed with configuration: each application has a QOS Manifest which describes it fully. There are two important Quorum Sets in a QOS Manifest:
* The Manifest Set is a set of parties who must approve QOS manifests.
* The Share Set is a set of parties who must post their share into a booted enclave, to provision it with its Quorum Key. The Quorum Public Key is also part of the QOS Manifest.
Using the flexibility offered by these quorum settings one can picture a wide variety of setups with different trade-offs:
* Individual customers could choose to be part of the Manifest Set to ensure they review each QOS Manifest before it is deployed. We envision this to be a good fit for e.g. well-known code auditors or parties who attest to the build.
* Individual customers could choose to be part of the Share Set to be part of the provisioning process. We envision this to be a good fit for parties who already have experience holding key material securely.
* Multiple independent parties could participate in the process of approval or provisioning. This decentralization can happen across geographies, jurisdictions, and span multiple archetypes of parties (companies, individual developers, DAOs). Similar to Security Councils built around multisig wallets, a Turnkey application can distribute control because of the flexibility of Quorum Sets.
Using Quorum Sets to their fullest extent is only one way to distribute control. Another idea to accomplish customer-specific ciphertext is to do it at a level higher. Our Signer application could encrypt key material to an external key known to the customer only. This key material would have to be encrypted and injected into the Signer enclave once provisioned. The advantage of this scheme is its simplicity: one signer enclave can serve multiple customers. We essentially create a customer-specific provisioning process which can be verified by all.
### [](https://whitepaper.turnkey.com/applications/#maximal-crypto-toolkit "Copy link to this section")
Maximal crypto toolkit
As crypto matures and moves towards post-quantum crypto, more primitives will become available and adopted. We envision Turnkey to be a complete cryptography toolkit providing support for new MPC protocols, ZK proving, encryption, hash-based signatures, and more.
Execution in TEEs nicely complements on-chain primitives: on-chain protocols can verify TEE attestations, and TEEs can provide compute services to on-chain protocols. This symbiotic relationship unlocks an experimental mindset where builders can unblock themselves and try new things using TEEs first, then enshrine these primitives in distributed protocols if they are met with broad adoption. We're excited to provide this missing critical playground for the industry.
[](https://whitepaper.turnkey.com/applications/#thwarting-supply-chain-risks-once-and-for-all "Copy link to this section")
Thwarting supply chain risks once and for all
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Several portions of the Turnkey stack should be distributed across a network of decentralized parties over time. Each of these is aimed at improving the security of the software supply chain, from code review, to builds, to runtime. Our end-game: thwart supply-chain risk once and for all.
### [](https://whitepaper.turnkey.com/applications/#reviewing-source-code-and-dependencies "Copy link to this section")
Reviewing source code and dependencies
Modern application developers depend on external software dependencies to accelerate the development process and use well-vetted & secure methods of performing sensitive tasks. Unfortunately, there are simply too many dependencies to review well. As a result, most companies either:
* Accept serious risks and don’t properly review their dependencies, or
* Slow down their development by limiting the number of new dependencies that can be taken.
Fortunately, many organizations have significant overlap in the dependencies they rely on. As a result, reviews can (and should!) be shared widely & performed in public. We think that incentivizing this behavior via a decentralized network is a critical initiative for the security of crypto at large.
Code review of internal applications is another critical behavior. Verifiable software (via remote attestation) is critical to ensuring transparent, high-security applications. But if those applications themselves simply contain malicious or accidental security issues, what good is verification?
It seems foolish to continue with the current model where every critical piece of software incurs a heavy tax (both in terms of money and time) to be “trusted”. Very often this comes in the form of expensive certification processes and lengthy audits. We can cut the middleman, improve efficiency, and provide a lot more transparency.
### [](https://whitepaper.turnkey.com/applications/#reproducing-builds "Copy link to this section")
Reproducing builds
We envision a network of builders who are incentivized to compile and sign artifacts which are eventually executed on a verifiable stack. A diverse array of builders attesting that a particular bundle of source code (and its dependencies) yields an artifact with a particular hash will be an important vote of confidence, and critical to the verifiability of Turnkey-deployed applications.
### [](https://whitepaper.turnkey.com/applications/#providing-compute-resources "Copy link to this section")
Providing compute resources
Individual consumers of verifiable applications likely have different opinions on:
* Which geographies and jurisdictions they want to be deployed in.
* Which types of TEEs or hardware providers they trust.
* Which operators they trust.
For this reason, we expect to have a wide array of compute providers running one or more verifiable applications on a specific kind of hardware in a specific location. These providers are one side of a market; the other side is composed of builders looking to use verifiable compute resources.
[](https://whitepaper.turnkey.com/applications/#conclusion "Copy link to this section")
Conclusion
----------------------------------------------------------------------------------------------------------------------------------------------------
Unlike most whitepapers, we are not talking about hypotheticals. Turnkey has been operating for 2+ years, securing millions of wallets. Ready to build? [Sign up for Turnkey](https://app.turnkey.com/dashboard/auth/initial)
, check out [our documentation](https://docs.turnkey.com/)
, and give us a whirl.
If you want to say hello, that's okay too: [hello@turnkey.com](mailto:hello@turnkey.com)
* * *
1
While not yet practical at the time of writing (Feb 2025), solutions to the problem of cross-L2 credential updates exist in prototype form. See Vitalik's description of [keystores](https://vitalik.eth.limo/general/2023/06/20/deeperdive.html)
. One can imagine a rollup dedicated to credential management (see [Dedicated minimal rollup for keystores](https://notes.ethereum.org/@vbuterin/minimal_keystore_rollup)
and [key.space](https://docs.key.space/)
) or a dedicated keystore contract on L1 read by other L2s (see [`L1SLOAD`](https://ethereum-magicians.org/t/rip-7728-l1sload-precompile/20388)
). These solutions require deployment and wide adoption so they're not practical today, but may become practical soon given the fast developments in the interoperability space.[↩](https://whitepaper.turnkey.com/applications/#ref:1)
---
# Turnkey Whitepaper
* [Index](https://whitepaper.turnkey.com/)
* [Principles](https://whitepaper.turnkey.com/principles)
* [Foundations](https://whitepaper.turnkey.com/foundations)
* [Architecture](https://whitepaper.turnkey.com/architecture)
* [Applications](https://whitepaper.turnkey.com/applications)
Turnkey's Architecture
======================
Turnkey Team
January 2025
[](https://whitepaper.turnkey.com/architecture/#abstract "Copy link to this section")
Abstract
------------------------------------------------------------------------------------------------------------------------------------------------
In [Verifiable Foundations](https://whitepaper.turnkey.com/foundations/)
we explained how Turnkey runs binaries inside of Trusted Execution Environments (TEEs). StageX provides reproducible builds to go from source code to stable binaries, and QuorumOS provides a base secure OS to provision StageX binaries into cloud TEEs. We now take an engineering-focused tour of Turnkey as it is running today. We explain the roles and responsibilities of the different applications running inside secure enclaves, as well as what's needed _around_ these applications to run them at scale and expose them to the outside world securely.
Below we detail our ambitious threat model: verifiable components are trusted, everything else isn't. We've built Turnkey to be a provably secure key management system, where anything touching user private key material is implemented within secure enclaves, in trusted space, hence verifiable. Our production infrastructure runs five main types of enclave applications (jump to [our architecture diagram](https://whitepaper.turnkey.com/architecture/#complete-architecture-diagram)
if you want to skip to the final picture):
* The Policy Engine enclave is responsible for end-user request parsing, authentication, and authorization. For maximum flexibility we've implemented authorization with a policy engine and a simple yet expressive language.
* The Notarizer[1](https://whitepaper.turnkey.com/architecture/#1)
enclave guarantees the integrity of critical user data.
* The Signer creates cryptographic keys and signs transactions.
* The Parser extracts metadata from unsigned transactions.
* The TLS Fetcher fetches content securely via TLS and produces portable proofs such that other enclaves can trust the TLS responses. The TLS Fetcher is the only enclave with access to external connectivity (at layer 4), because it uses [`qos_net`](https://github.com/tkhq/qos/tree/main/src/qos_net)
.
By the end of this document you should understand how Turnkey implements key management in practice, from bottom to top, and why its verifiable foundations are so vital to its design. With this in mind you'll be able to jump to [Applications Beyond Key Management](https://whitepaper.turnkey.com/applications/)
where we explain where Turnkey goes from where it is today from a technology and product perspective.
[](https://whitepaper.turnkey.com/architecture/#threat-model "Copy link to this section")
Threat model
--------------------------------------------------------------------------------------------------------------------------------------------------------
Turnkey was built with a wickedly simple threat model:
* We consider enclave applications and their Quorum Sets **trusted**.
* Everything else is considered **untrusted**.
This threat model minimizes our Trusted Computing Base ([TCB](https://en.wikipedia.org/wiki/Trusted_computing_base)
): the less we have to trust the easier it is to reason about security, and the easier it is to strengthen over time.
Note that it _does not mean untrusted components are insecure_. For example, we consider our database untrusted. As a result, even a malicious AWS admin or Turnkey admin modifying data doesn't break our security model! Another example is our DDoS protection measures: they are enforced outside of our secure enclaves, at the ingress layer, which we consider untrusted. This means Turnkey is secure even without these protections. We have them in place because [Defense in Depth](https://en.wikipedia.org/wiki/Defense_in_depth_(computing))
is a critical strategy in any security system, but do not rely on them to guarantee the security of enclave-held private keys.
The main difference between code running within enclave applications and code running outside is the provable nature of it. By leveraging remote attestations, anyone can verify that security measures (in the form of software checks) are indeed in place where they should be when they are implemented within secure enclaves. This isn't the case when they are implemented in software running outside of secure enclaves: one must rely on formal security audits and insiders to get an accurate picture of the system and its inner workings. Another major difference between enclave code and non-enclave code is the set of constraints around it: enclave applications are self-contained binaries with no outside dependencies. We choose to write all enclave applications in Rust. This helps reduce their surface dramatically.
So how do we choose what must and must not be run within secure enclaves? We've architected Turnkey to be a provably secure _key management_ system. Our security north star: if something can potentially impact user private keys, it must be implemented within secure enclaves, in **trusted** space. Otherwise, it can be done outside, in **untrusted** space.
[](https://whitepaper.turnkey.com/architecture/#system-topology "Copy link to this section")
System Topology
--------------------------------------------------------------------------------------------------------------------------------------------------------------
Turnkey is an API product[2](https://whitepaper.turnkey.com/architecture/#2)
. When clients send requests to Turnkey they hit an API Gateway component over a secure connection. Requests to Turnkey are standard HTTP requests.
Once a request ingresses into Turnkey infrastructure the rest of the components use gRPC to communicate internally: the API Gateway converts HTTP requests into gRPC requests, and the Coordinator routes requests to enclave applications. If multiple enclaves need to be called, the coordinator handles this as well, by calling the first enclave, then the second enclave. Enclaves cannot send requests to other enclaves. Topologically they're leaves of the call graph.
Turnkey topology with enclaves as leaves
The Coordinator is also responsible for DB interactions: it can read and write to a datastore as needed.
Enclave apps are highlighted in green to show that they're the only verifiable (and thus, trusted) components in this diagram. At this point you may be wondering:
* How can user requests travel through untrusted components? Can't they be modified by the API Gateway or the coordinator before reaching enclaves?
* Is it safe to persist data in an untrusted component? How can we prevent data from being modified by an administrator for example?
We answer these questions and more in the rest of this document.
[](https://whitepaper.turnkey.com/architecture/#organization "Copy link to this section")
Organization
--------------------------------------------------------------------------------------------------------------------------------------------------------
Organizations are data containers encapsulating signing resources (Wallets and Private keys) as well as data relevant for [authentication and authorization](https://whitepaper.turnkey.com/architecture/#authentication-and-authorization)
of their usage (e.g. Users, Policies). Concretely, Organizations are objects with the following top-level fields:
* **version**: organization data schema version. This version changes when the organization data schema changes. For example, adding an “email” field to each user.
* **uuid**: a unique identifier (we use [UUIDv4](https://datatracker.ietf.org/doc/html/rfc9562#name-uuid-version-4)
)
* **name**: name of the organization
* **users**: a list of Users. Each User has one or more ways to authenticate into the organization. See [Authentication](https://whitepaper.turnkey.com/architecture/#authentication)
below.
* **rootQuorum**: a list of user IDs and a threshold which defines the Root Quorum. See [Root Users and Quorum](https://whitepaper.turnkey.com/architecture/#root-users-and-quorum)
.
* **invitations**: a list of pending Invitations into the organization.
* **policies**: a list of Policies.
* **tags**: a list of Tags. Tags can apply to Private Keys, Wallets, or Users and are a convenient way to group resources to be able to refer to them consistently in Policies.
* **privateKeys**: collection of Private Keys.
* **wallets**: collection of Wallets. These are HD wallets compliant with BIP39[3](https://whitepaper.turnkey.com/architecture/#3)
.
When an Organization is stored we serialize it to JSON[4](https://whitepaper.turnkey.com/architecture/#4)
and it looks like the following:
{
"version": "15.0",
"uuid": "d12b...7f6b",
"name": "My Organization",
"users": [...],
"rootQuorum": {...},
"invitations": [...],
"policies": [...],
"tags": [...],
"privateKeys": [...],
"wallets": [...]
}
_The rest of this document will reference Organizations and their data fields with a capital letter to distinguish them from the common nouns. For example, if you read about Wallets, we're referring to Turnkey Wallets defined inside of Organization data. The term “wallet” without a capital letter can then refer to an external user wallet such as Metamask, or a physical wallet holding cash or cards._
[](https://whitepaper.turnkey.com/architecture/#activities-and-queries "Copy link to this section")
Activities and Queries
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Now that we've introduced the concept of an Organization, let's talk about the two classes of requests supported by the Turnkey API: Activities and Queries.
Queries are read-only requests made to Turnkey. They are considered non-critical because they do not mutate the state of the organization, and do not allow usage of Wallets or Private Keys. All queries can be executed outside of secure enclaves. Examples of queries are “get user details” or “list policies”.
Activity requests represent critical actions within an Organization. There are roughly three categories to consider:
* Genesis: creation of a new organization.
* Resource management: creation, update, or deletion of Users, Policies, Tags, Private Keys or Wallets.
* Signing resource usage: produce new signature, import/export Private Keys, import/export Wallets.
Activities define atomic operations with typed inputs (we often talk about activity “params”), and outputs (activity “results”). To allow safe upgrades over time in either params or result shapes, Activities are versioned[5](https://whitepaper.turnkey.com/architecture/#5)
. Because of how critical activities are, they must be verifiably processed, in trusted space, within secure enclaves. We explain how we pull this off below.
[](https://whitepaper.turnkey.com/architecture/#authentication-and-authorization "Copy link to this section")
Authentication and Authorization
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Request Authentication (and Authorization) are core to the security of Turnkey. Request **Authentication** answers the question “who made this request?” while **Authorization** answers the question “is this authenticated User authorized to perform this operation?”.
### [](https://whitepaper.turnkey.com/architecture/#authentication "Copy link to this section")
Authentication
The threats we want to defend against in the context of request **authentication** are:
* MITM attack: an attacker with control of an untrusted component modifies authenticated requests bodies before they reach enclaves
* Request forgery: an attacker with control of an untrusted component creates new requests from within Turnkey's infrastructure
* Replay attacks: an attacker captures and replays previous legitimate requests
To defend against MITM attacks and request forgery, requests to Turnkey are **signed by user-held key pairs**. These key pairs can be pure P256 key pairs (we call these “API keys”) or webauthn authenticators in the form of passkeys or hardware security keys (we call these “authenticators”) or secp256k1 key pairs associated with a crypto address (we call these “external wallets”). These signatures sign over the full request body and are attached to HTTP requests as HTTP headers. We refer to these signatures as “stamps”.
To defend against replay attacks we mandate that each request to Turnkey must contain a timestamp as part of the signed payload. Enclaves check this timestamp against their internal secure source of time and reject requests if their timestamp is too old or in the future.
In summary HTTP requests to Turnkey look like the following:
POST /api/v1/submit/create_users
Host: api.turnkey.com
X-Stamp:
{
"timestampMs": "",
"organizationId": "",
"type": "ACTIVITY_TYPE_CREATE_USERS",
"params": {
}
}
We're left with an important question: how do enclaves determine which key pairs are user-held?
The answer to this question is our concept of Organization which we've introduced above. Turnkey Organizations hold the definition of Users and Policy. Given a request and the correct Organization data (which can be loaded by ID, since the request contains an Organization ID), an enclave can independently determine who the request is from by verifying the request signature and looking up the associated public key inside of Organization data. If the public key is unknown, the request is denied. If the public key is attached to an existing organization User, the request is authenticated successfully.
### [](https://whitepaper.turnkey.com/architecture/#authorization "Copy link to this section")
Authorization
Once a request is authenticated, we have answered “who requested this?”. The answer comes in the form of a User ID within a Turnkey Organization. Once we've established that a particular User has placed a request, authorization needs to take place and answer the question: “can this user perform this action?”. The answer needs to be a definite “yes” or “no”. This is where Policies come in[6](https://whitepaper.turnkey.com/architecture/#6)
. We've designed Turnkey to be flexible enough to support granular, programmatic permissions.
#### Policies
Turnkey Policies are part of Organization data. Concretely a policy looks like the following:
{
"policyName": "Allow user MY_USER to create wallets",
"condition": "activity.resource == 'WALLET' && activity.action == 'CREATE'",
"consensus": "approvers.any(user, user.id == 'MY_USER')",
"effect": "EFFECT_ALLOW"
}
* `policyName` is a field with a human-readable description of what the policy does or why it was put in place.
* `Condition` is an expression to codify the scope of the policy. In other words: when it applies. If this expression evaluates to true, the policy applies. In the example above, the policy applies when the activity is a “create wallet” activity.
* `consensus` is an expression codifying “who” must sign an activity for the policy to apply. In the example above the policy applies if the user `MY_USER` approves the activity.
* `effect` indicates the effect of the policy. `EFFECT_ALLOW` will allow the activity to be processed, `EFFECT_DENY` will reject it.
The expression language we chose for Turnkey policies (`condition` and `consensus` fields) is a simple yet expressive language based on [Google's CEL](https://github.com/google/cel-spec)
to provide the convenience of common infix operators such as `<`, `>`, or `==`. A full list of operators and grammar functions can be found [in our documentation](https://docs.turnkey.com/concepts/policies/language#grammar)
.
Policies can be managed programmatically: creating, updating or deleting a policy is done with a Turnkey Activity. All policies are Organization resources and all of them are evaluated when a request is processed. Note that Policy security is extremely important because it is an easy attack vector for Users of an Organization. If strict policies can be weakened or removed, the integrity of authorization falls apart. We'll see in later sections how verifiable Activity processing is architected and ensures the integrity of Policies and other Organization resources.
#### Combining policies
Because we have multiple types of outcomes (`ALLOW`, `DENY`) it is possible for two policies to evaluate to different, conflicting outcomes. For example:
* `ALLOW` policy with condition `activity.resource == 'WALLET' && activity.action == 'SIGN'`
* `DENY` policy with condition `wallet.id == 'MY_WALLET'`
If a request to sign with `MY_WALLET` is evaluated, both policies will trigger: the first one results in `ALLOW`, the second in `DENY`.
To resolve conflicts we have the following rules in place:
* An Activity is not allowed unless explicitly `ALLOW`ed by one or more Policies
* Any explicit `DENY` outcome is final and results in the denial of the Activity, even if it is explicitly `ALLOW`ed by other policies in the Organization. In other words, `DENY` takes precedence over `ALLOW`.
In the example above, an attempt to sign with any wallet would succeed (because the first policy `ALLOW`s it), but fails when the wallet ID is `MY_WALLET` (the two policies conflict, and `DENY` takes precedence).
#### Consensus
Consensus is an important feature of the policy engine because it enables multiple parties who do not trust each other to collaborate on critical actions. For example, a user can give permission to a trading bot, via policies, to perform signing activities. Without consensus this is potentially risky because the trading bot gains unilateral control of the signing resource. If it goes rogue, funds can be lost. A safer setup is a consensus setup where both the bot and the user have to agree on a signing action before it executes.
As its name indicates the `consensus` field is built to support consensus use-cases:
* `approvers.any(user, user.id == 'HUMAN_USER_ID') && approvers.any(user, user.id == 'BOT_USER_ID'`) would fulfill the trading bot use-case above
* `approvers.count() > 2` is a looser policy which triggers when any 2 distinct users approve an activity
To approve an activity, a user can either sign the activity payload directly, or use the `APPROVE_ACTIVITY` activity, which references an activity by its fingerprint (unique sha256 digest). Both methods are equivalent from a consensus point of view, and count as a single approval.
#### Root Users and Quorum
Organizations are initialized with a single User at creation time. This user must be able to perform activities within the organization to set it up correctly: invite other Users, add API Keys, create new Wallets, and so on. Given we deny activities unless explicitly allowed, we need a solution to facilitate the initial setup. This solution is the concept of Root Quorum. When initialized, the organization starts with a Root Quorum of exactly 1 user, the first User:
"rootQuorum": {
"threshold": 1,
"userIds": ["a241...4277"]
}
When Root Quorum is reached, the policy engine is bypassed (Organization policies are not evaluated) and the activity is automatically allowed. With Root Quorum approvals, any Turnkey activity can be performed, similar to a root user on a Unix system.
As the organization grows and matures the Root Quorum should evolve to be a multi-party quorum. Below, an example of a 2-out-of-3 Root Quorum.
"rootQuorum": {
"threshold": 2,
"userIds": ["a241...4277", "078d...14ea", "4af2...98cb"]
}
Two Users must approve unilateral actions, which is a much safer setup because it removes any single-point-of-failure: if a single User goes rogue, they're not able to take action, and if a single User loses access to their API keys or authenticators, the Organization still has enough Users left to approve critical activities, including activities to update the Root Quorum itself.
### [](https://whitepaper.turnkey.com/architecture/#the-policy-engine-enclave "Copy link to this section")
The Policy Engine Enclave
The critical nature of Authentication and Authorization is obvious: if something goes wrong there, the security of Turnkey is compromised. It should be no surprise that we build this in trusted space, within secure enclaves. More specifically, authentication and authorization logic is contained within our Policy Engine. Its only responsibility is a simple but crucial one: given a user request and a snapshot of Organization data, return a “yes” or “no” answer. This is where Policies are evaluated.
Policy Engine enclave
In the diagram above we depict an Activity request (with its signature, or “Stamp”), and an Organization (with its resources: Users, Policies, Wallets, Private Keys, etc) being fed as input to the Policy Engine, which responds with a signed decision. We call this decision a **Ruling**. The signature is produced with the Policy Engine's Quorum Key. A Ruling contains:
* The organization ID and digest
* The decision itself (`ALLOW` or `DENY`)
* The Activity request (type, params)
* A timestamp
A Ruling is tied to a particular point-in-time snapshot of an organization: indeed if a policy is later removed or added, or if a User rotates their API Key, the Ruling might change! Everything needed to make a decision is injected as input, which means the Policy Engine itself can be stateless[7](https://whitepaper.turnkey.com/architecture/#7)
and run within a secure enclave.
Because Rulings are signed by a stable Quorum Key (the Policy Engine's), other components can hardcode the Policy Engine's Quorum Public Key and rely on Rulings and their signatures as proof that a request was successfully authenticated and authorized. This is a powerful building block: we've isolated the problem of “auth” to a single stateless computation which produces portable proofs. We'll see in the following sections how Rulings are used as inputs to other secure enclaves.
[](https://whitepaper.turnkey.com/architecture/#organization-data-notarization "Copy link to this section")
Organization Data Notarization
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
So far we've assumed that Organization data is fed to the Policy Engine faithfully. This isn't a correct assumption given both the database (which stores Organization data) and the Coordinator (which loads data from the database and calls enclave applications) are assumed “untrusted” in our threat model. In other words, we should consider the following attack scenarios:
* Data tampering: a database admin modifies organization data directly
* Downgrade attack: a database admin rolls back organization data to a previous point-in-time
Carried out successfully, these attacks would completely undermine authentication and authorization. With a successful tampering attack, an admin would be able to arbitrarily insert their public key in the organization data, gaining signing access to its resources (Wallets, Private Keys). With a downgrade attack, revoked API Keys or deleted Users get their previous level of access back.
To guarantee the integrity of organization data, an enclave must sign it. We call these signatures **Notarizations**. To guarantee that data is recent and thwart downgrade attacks we periodically re-sign Organization data to generate fresh Notarizations. Any old Notarization is considered invalid when it is older than a static threshold.
The enclave application creating, updating, and signing organization data is called the **Notarizer**. Requests to the Notarizer are authorized with Policy Engine Rulings:
* Create and Update operations: The Notarizer enclave expects a valid Ruling to create or update organization data and produce new Notarization.
* Refresh operation: The Notarizer expects a valid, existing Notarization when generating new Notarization.
Summarizing this section with a few diagrams, the Notarizer has three operations: Create, Update, and Refresh.
Notarizer Enclave (create operation)
The “Create” operation produces a new Organization and its associated Notarization.
Notarizer Enclave (update operation)
The “Update” operation is also authorized with a Policy Engine Ruling. Note the sha256 update: this means something changed inside of the organization data. In the example above a new User is added.
Notarizer Enclave (refresh operation)
Finally, the “Refresh” operation takes as input:
* Organization data
* A valid past Notarization (labelled “Old”) for this data. Remember: a notarization is a signature of the Organization data digest which includes a timestamp.
The output of the “Refresh” operation is the same, unmodified Organization data (note the sha256 does not change) and a new valid Notarization, labelled “New” because its timestamp is the current timestamp.
Refresh operations need to happen periodically for all inactive Organizations. Indeed, unless an Organization mutates its data periodically (resulting in updated Organization data and Notarization), the Notarization associated with an Organization becomes stale, and thus invalid once it is past our replay protection threshold.
The number of Refresh operations scale linearly with the number of Organizations in our system, which is unsustainable at scale. To address this challenge we have designed a Merkle-tree based validation system for Notarizations and repudiation of old data. See [Appendix: scaling Verifiable Data](https://whitepaper.turnkey.com/architecture/#appendix-scaling-verifiable-data)
for more details.
[](https://whitepaper.turnkey.com/architecture/#tackling-organization-data-growth "Copy link to this section")
Tackling organization data growth
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Because Organization data is a monolithic object and used as input/output value in/out of enclave applications, an Organization cannot grow indefinitely (see [Resource Limits](https://docs.turnkey.com/getting-started/resource-limits)
).
To allow Organizations to scale to millions of Private Keys and Wallets we've introduced the concept of **Sub-Organization**. A Sub-Organization is just like an Organization: it's an independent data container with the same set of fields as a normal Organization. The only difference is a `parentOrganizationUuid` field which links the Sub-Organization to its parent Organization[8](https://whitepaper.turnkey.com/architecture/#8)
. This is a useful primitive to model end-users: when a business uses Turnkey as a wallet provider, each end-user can be provisioned with its own Sub-Organization. The business can scale to millions of users and wallets without continuously growing its parent Organization data size. Another crucial aspect of this design: end-users are guaranteed that their Private Keys and Wallets are self-contained, kept intact, and isolated from the business or other end-users. See [wallet.tx.xyz](https://wallet.tx.xyz/)
for a live demonstration.
[](https://whitepaper.turnkey.com/architecture/#verifiable-activity-processing "Copy link to this section")
Verifiable Activity processing
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
We now have the necessary knowledge to fully grasp Activity processing: it is based on a bi-directional dependency[9](https://whitepaper.turnkey.com/architecture/#9)
between the Notarizer and Policy Engine enclaves.
Verifiable activity processing
* The Policy Engine relies on the Notarizer to ensure the integrity of organization data it uses as input. A valid and current Notarization is a portable proof that Organization data is intact and current.
* The Notarizer relies on the Policy Engine for authentication and authorization. A valid and current Ruling is a portable proof that an Activity was authenticated and authorized at a given point in time, for a particular snapshot of an Organization's data.
* Finally, the Notarizer relies on itself as well: it requires a valid and current Notarization to process activities.
In the picture below we show the sequence of events from “genesis” of an Organization: the first activity is a “Create Org” activity, resulting in “Org State 0”. A new activity (“Activity 0”) is then requested and results in “Org State 1”. And so on: “Activity N” and "Org State N" result in “Org State N+1” after processing.
Organization state transition through activities, from genesis
It's worth noting that all of the data in the diagram above is signed by trusted entities[10](https://whitepaper.turnkey.com/architecture/#10)
:
* Organization data is signed by the Notarizer
* Rulings are signed by the Policy Engine
* Activities are signed by known User public keys, which we trust because they are part of signed Organization data.
By running the Notarizer and Policy Engine enclaves in trusted space we have verifiable authentication, verifiable authorization, and verifiable Organization data. We now introduce a third, very important enclave: the Signer.
[](https://whitepaper.turnkey.com/architecture/#the-signer-enclave "Copy link to this section")
The Signer enclave
--------------------------------------------------------------------------------------------------------------------------------------------------------------------
Turnkey is a key management solution and so far we have not talked about Wallets or Private Keys much. We have mentioned that they are part of Organization data, and that should scare you: are we storing private keys and wallet seeds in plaintext inside of Organization data? How can this be safe?
The Signer enclave creates new Private Keys and Wallets, and produces signatures. These actions are implemented as Activities (create wallet, create private key, sign payload), secured using the previously introduced enclaves.
Key generation or usage requires a valid Policy Engine Ruling and a valid, notarized Organization. This ensures all Signer actions are user-initiated and based on valid and recent snapshot of an Organization.
### [](https://whitepaper.turnkey.com/architecture/#key-generation "Copy link to this section")
Key Generation
Key generation requires the use of the Nitro NSM[11](https://whitepaper.turnkey.com/architecture/#11)
to use secure entropy. Once key material is generated (in the form of a raw Private Key, or a Wallet seed), it is **encrypted to the Signer's Quorum Key** and returned. Because the Signer Quorum Key is never reconstructed outside secure enclaves, the key material never exists in plaintext form outside our trusted, verifiable boundaries. The Notarizer inserts this encrypted key material inside of Organization data following its normal requirements: a valid Ruling and Organization are required. Once Organization data is updated, the key material is ready to be used.
Key generation is crucial and thus verifiable. Because the Signer runs on top of Turnkey's [Verifiable Foundations](https://whitepaper.turnkey.com/foundations/)
, it is possible to walk back from a [Boot Proof](https://whitepaper.turnkey.com/foundations/#boot-proofs-and-app-proofs)
all the way to the source code to verify entropy sourcing and encryption are performed correctly.
A visual diagram of key generation is given below, showing the signer producing key material, and the Notarizer using this signed output to update organization data.
Signer enclave (key generation)
It's worth noting that because all enclave outputs and inputs are signed with Quorum Keys, it's safe for e.g. “Encrypted Key Material” to transit in untrusted space. The Notarizer enclave will reject encrypted key material with an invalid Signer signature, or if the key material wasn't created for the correct Ruling.
### [](https://whitepaper.turnkey.com/architecture/#key-usage-signing-payloads "Copy link to this section")
Key Usage (signing payloads)
Once an Organization contains encrypted key material, producing signed payloads is straightforward. The Signer is given a Ruling and an Organization containing the encrypted key material. The Signer verifies the validity of the Policy Engine Signature and Notarization, and decrypts the key material. The Policy Engine Ruling contains the activity params and thus, the payload to sign. The signed payload is produced and returned to the user.
Signer enclave (signing)
The output (“Signed Payload”) is not signed by the Signer Quorum Key because it is self validating: end-users can trivially verify the validity of signature against the message (provided as input) and the expected public key (previously generated by the Signer Enclave).
### [](https://whitepaper.turnkey.com/architecture/#key-import-and-export "Copy link to this section")
Key Import and Export
Because no untrusted component in our system can see imported or exported keys in plaintext, we've designed these flows with one-time encryption keys derived with HPKE ([RFC 9180](https://datatracker.ietf.org/doc/rfc9180/)
).
* To import a key, the Signer creates a new one-shot encryption keypair, to which the user encrypts the imported key material.
* To export a key, the reverse happens: the user creates a one-shot encryption keypair client-side[12](https://whitepaper.turnkey.com/architecture/#12)
, to which the Signer encrypts the exported key material.
See our documentation on [import](https://docs.turnkey.com/wallets/import-wallets)
and [export](https://docs.turnkey.com/wallets/export-wallets)
for more details.
### [](https://whitepaper.turnkey.com/architecture/#key-deletion "Copy link to this section")
Key Deletion
Deletion is a critical action which modifies Organization data. It is implemented as a standard Turnkey activity and requires the same authentication and authorization checks as any other critical actions.
[](https://whitepaper.turnkey.com/architecture/#transaction-aware-policies "Copy link to this section")
Transaction-aware Policies
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
We've seen that our Policy language supports the ability to restrict what Users can and can't do by activity type. Our “sign payload” activity can thus be broadly allowed or denied based on policies. But what if more granularity is needed? Common examples are deny-list based on sanctioned addresses (`DENY` if transaction targets a sanctioned address, otherwise `ALLOW`), or amount-based policies to reduce risk (`ALLOW` if transaction amount is under an arbitrary threshold, otherwise `DENY`).
Because we want to keep enclave applications focused and minimal, we decided to introduce a new enclave to solve this particular problem: the Parser enclave.
### [](https://whitepaper.turnkey.com/architecture/#the-parser-enclave "Copy link to this section")
The Parser enclave
The Parser enclave is a simple enclave to parse unsigned transactions and extract important metadata. We currently support Solana and EVM transaction parsing.
The Parser Quorum Key signs the resulting metadata, which can then transit to the Policy Engine to inform its decisions. The signed transaction metadata returned by the Parser becomes extra “context” data which can be used inside of the Policy Engine.
Parser enclave (parse operation)
Organization Policies can use this metadata through dedicated namespaces to build transaction-aware policies. A namespace is nothing more than a unique keyword, associated with data inside of it. We have chosen **eth.tx** and **sol.tx** as the top-level namespaces containing metadata for Ethereum and Solana parsed transactions, respectively. Refer to [our Policy Engine language reference](https://docs.turnkey.com/concepts/policies/language#struct)
for a complete inventory of the available metadata inside of each of them.
A concrete example: an `ALLOW` policy with the condition `eth.tx.chain_id == 11155111` would allow transactions as long as they're [Sepolia](https://sepolia.etherscan.io/)
transactions. For more policy examples, see [our documentation](https://docs.turnkey.com/concepts/policies/examples#signing-control)
.
### [](https://whitepaper.turnkey.com/architecture/#flexibility-and-extensibility-of-the-policy-language "Copy link to this section")
Flexibility and extensibility of the policy language
We've seen above that our Parser enclave provides extra verifiable metadata to the Policy Engine, in the form of another namespace which policies can use. We envision many other types of data to be relevant and useful to write fine grained policies. We detail this in [Applications beyond Key Management](https://whitepaper.turnkey.com/applications/)
.
[](https://whitepaper.turnkey.com/architecture/#verifiable-tls "Copy link to this section")
Verifiable TLS
------------------------------------------------------------------------------------------------------------------------------------------------------------
### [](https://whitepaper.turnkey.com/architecture/#solving-the-external-connectivity-problem "Copy link to this section")
Solving the external connectivity problem
Secure enclaves do not have the ability to contact the outside world directly. In [Verifiable Foundations](https://whitepaper.turnkey.com/foundations/)
we've seen that enclaves are connected to their host by a [`VSOCK`](https://man7.org/linux/man-pages/man7/vsock.7.html)
interface, but have not explained what `VSOCK` is, really. Put simply, a `VSOCK` is similar to a UNIX domain socket ([UDS](https://en.wikipedia.org/wiki/Unix_domain_socket)
) but is used to communicate between hosts and virtual machines. A `VSOCK` connection has a context ID and a port. The context ID is analogous to an IP address in TCP/IP, and ports work as you would expect.
Enclave `VSOCK` connection
Typically an enclave application binds to its context ID and a chosen port, listening for host connections and requests. The host client forwards requests it receives from the network to the enclave application by connecting to the right context ID and port.
To make outbound requests from inside an enclave application we need to do this in reverse: the host has to listen for requests made by the enclave application. When the enclave makes a request to open a connection, the host-side proxy can connect to the right target because it has network access. Similarly, read or write requests can be made by the enclave, and the proxy will act on the enclave's behalf and call read or write on the real TCP connections it holds.
Enclave `VSOCK` proxy
The proxy component in the diagram above provides external connectivity **at the TCP level only** (aka “layer 4” or [“transport layer”](https://osi-model.com/transport-layer/)
). More exactly, the proxy’s interface is composed of three operations:
* **Open** connection: Open a new TCP connection to a target IP address
* **Read** from connection: Read N bytes from an existing TCP connection
* **Write** to connection: Write N bytes to an existing TCP connection
### [](https://whitepaper.turnkey.com/architecture/#tls-on-top-of-tcp "Copy link to this section")
TLS on top of TCP
All enclave applications at Turnkey are written in Rust. In Rust, `Read` and `Write` are standard library traits: [`std::io::Read`](https://doc.rust-lang.org/std/io/trait.Read.html)
, [`std::io::Write`](https://doc.rust-lang.org/std/io/trait.Write.html)
. The most popular pure-Rust TLS crate, [Rustls](https://github.com/rustls/rustls)
, works with these traits to implement TLS: users of the library provide a connection object which implements `Read` and `Write` traits, and Rustls uses that connection object to implement TLS handshakes, request encryption, and response decryption on top. This is explained in more detail [in their documentation](https://docs.rs/rustls/latest/rustls/index.html#rustls-provides-encrypted-pipes)
. We use this to our advantage by implementing these traits with a custom struct: upon receiving a call to `read` or `write` our trait-implementing struct calls the host-side proxy to read or write from already-established TCP connections.
If the proxy isn’t honest (this is in our threat model because the proxy runs outside of the secure enclave, in untrusted space), the TCP packets could be routed to the wrong remote host, but that would cause TLS certificate verification to fail. The proxy can also choose to censor and refuse to forward packets. This will be detected by the enclave as well. Finally, we’ve already seen that the proxy cannot decrypt or change TCP packets because TLS encrypts traffic with session keys, and these session keys are created and kept in our secure enclave.
### [](https://whitepaper.turnkey.com/architecture/#the-tls-fetcher-enclave "Copy link to this section")
The TLS fetcher enclave
We've named the secure enclave capable of making verifiable TLS requests “TLS fetcher”. Its interface is very minimal: given a method (`POST`, `GET`) a host, a path, a collection of headers, and a request body, return a response (status, body, headers) and a timestamp.
TLS Fetcher enclave
The response is signed by the TLS Fetcher Quorum Key. It is a portable proof that a given URL returned some content at a specific time, which neatly resolves the long-standing challenge of providing non-repudiation to TLS responses[13](https://whitepaper.turnkey.com/architecture/#13)
.
We use the TLS Fetcher to verifiably fetch OIDC configuration (see [this blog post](https://quorum.tkhq.xyz/posts/tls-sessions-within-tees/)
) and envision it to be an important building block for our future roadmap, which we talk about in [Applications Beyond Key Management](https://whitepaper.turnkey.com/applications/)
.
[](https://whitepaper.turnkey.com/architecture/#complete-architecture-diagram "Copy link to this section")
Complete Architecture Diagram
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
We've now completed our tour and introduced all the components inside of the trusted boundary. In this section we work up to a diagram showing all enclave applications and their connections in one picture, as well as a few new components which are outside of the trusted boundary.
### [](https://whitepaper.turnkey.com/architecture/#enclave-trust-relationships "Copy link to this section")
Enclave trust relationships
We've seen in the previous sections that enclaves are provisioned with their own Quorum Key, and a common pattern for enclave-to-enclave communication is to return signed structures which can be used as portable proofs fed to other components, including other enclaves. Below we summarize the trust relationships between enclaves:
* The Notarizer trusts the Policy Engine's “yes” or “no” decisions as well as key material generated by the Signer.
* The Policy Engine trusts the Notarizer to create and update Organization data. It also trusts metadata coming from the Parser enclave, and TLS responses coming from the TLS Fetcher.
* The Signer trusts the Policy Engine (for its decisions) and the Notarizer (for its notarizations of Organization data).
These trust relationships are only symbolic and do not mean that enclaves contact each other directly. Rather, it means that enclave applications hardcode each other's Quorum public keys, in order to ensure signed payloads are intact and authentic. For example, the Policy Engine's decisions come in the form of signed Ruling payloads. The Notarizer, by hardcoding the Policy Engine's Quorum public key, can check these Ruling and only accept the legitimate ones. It guarantees that a component in untrusted space such as the Coordinator can't modify a Ruling payload (it would invalidate the signature) or create a fake Ruling (the coordinator doesn't have access to the Policy Engine's Quorum Key).
These relationships are summarized in the diagram below:
Enclave trust relationships
### [](https://whitepaper.turnkey.com/architecture/#other-important-components "Copy link to this section")
Other important components
#### Notifier
This service powers our webhooks feature. See [our Activity webhook documentation](https://docs.turnkey.com/developer-reference/webhooks)
.
#### Heartbeat
This service schedules Organizations for a “refresh” operation when their notarization gets stale. The Heartbeat service enqueues “refresh tasks” in Redis, and the Updater picks them up.
#### Redis
A fast, lightweight datastore used to contain the set of refresh tasks from the heartbeat service. Tasks are dequeued by the updater. We use a redis `ZSET` ([sorted set](https://redis.io/glossary/redis-sorted-sets/)
) to atomically add and pop refresh tasks.
#### Updater
This service is connected to trusted enclaves in the same way that the Coordinator is. It polls from SQS in case there are Activities to retry, and executes refresh tasks pushed to Redis by the Heartbeat service.
### [](https://whitepaper.turnkey.com/architecture/#complete-diagram "Copy link to this section")
Complete diagram
Turnkey Architecture (full diagram)
[](https://whitepaper.turnkey.com/architecture/#utility-of-app-proofs "Copy link to this section")
Utility of App Proofs
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------
In [Foundations](https://whitepaper.turnkey.com/foundations/#boot-proofs-and-app-proofs)
we've introduced the concept of Boot Proofs and App Proofs. Now that we've introduced the enclaves Turnkey has built we can talk about use-cases for App Proofs.
Note that at the time of writing (2025-02-03), Turnkey uses Boot Proofs internally to power enclave provisioning, but they are not exposed to external customers yet. We plan to produce and expose App Proofs in the near future. Please [get in touch with us](https://www.turnkey.com/contact-us)
if you'd like to be a part of this effort or help us beta test this important feature.
In the table below we lay out some proof types along with the originating enclave and their purpose. This list isn't exhaustive!
| | | |
| --- | --- | --- |
| **Proof type** | **Originating Enclave** | **Purpose** |
| `SIGNATURE` | Signer | Prove that a signature was performed within Turnkey. |
| `ADDRESS_DERIVATION` | Signer | Prove that an address originates from within Turnkey. Also known as “provenance proof”. |
| `IMPORT_BUNDLE` | Signer | Prove that an import bundle originates from a legitimate signer. |
| `EXPORT_BUNDLE` | Signer | Prove that an export bundle originates from a legitimate signer. |
| `POLICY_OUTCOME` | Policy Engine | Prove that an activity request was authorized or denied by Turnkey's Policy Engine. |
| `NOTARIZATION` | Notarizer | Prove that our notarizer enclave produced a particular snapshot of Organization data. |
| `FETCH` | TLS Fetcher | Prove that our TLS fetcher enclave fetched content from the given URL. |
| `TRANSACTION_PARSING` | Parser | Prove that a transaction was parsed into a given set of metadata. |
[](https://whitepaper.turnkey.com/architecture/#conclusion "Copy link to this section")
Conclusion
----------------------------------------------------------------------------------------------------------------------------------------------------
Turnkey represents a significant advancement in key management. For the first time, the critical security claims of a key management provider are verifiable without audits or intermediaries. This is possible because Turnkey is built on verifiable foundations (see [Verifiable Foundations](https://whitepaper.turnkey.com/foundations/)
). We've chosen the strictest threat model possible: anything that can touch funds stored on user key material needs to be built in trusted space. We trust what is verifiable, and do not trust what can't be.
This overarching principle led us to design key management around enclave applications responsible for well-defined functionality:
* The Policy Engine authenticates and authorizes user requests
* The Notarizer produces and modifies Organization data
* The Signer creates and uses Private Keys and Wallets
* The Parser extracts metadata from unsigned transactions
* The TLS Fetcher makes secure requests to remote hosts
Each enclave application contributes to a unified system where authentication, authorization, and data integrity are not just ensured but verifiable by design. We now discuss the transformative possibilities of this architecture and its underlying foundations in [Applications Beyond Key Management](https://whitepaper.turnkey.com/applications/)
.
[](https://whitepaper.turnkey.com/architecture/#appendix-scaling-verifiable-data "Copy link to this section")
Appendix: Scaling Verifiable Data
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Because Turnkey stores millions of Organizations, the load on the Notarizer enclave is significant: this enclave application (and the system around it, namely: the heartbeat service, Redis, and the Updater) must periodically refresh the Organizations which have not mutated their data recently.
To address this challenge we've developed a better system based on Merkle trees. Organization notarizations are part of a Merkle tree, and the root of this Merkle tree is signed (with a timestamp) periodically.
An organization can prove it is valid when it presents a recent-enough Notarization, or a notarization which is included into a fresh enough Merkle tree (with a fresh-enough root).
When an organization goes through long periods of inactivity, its membership into the Merkle tree remains valid, which means we do not need to periodically refresh this org's notarization.
For normally-active organizations, nothing changes: activity processing will result in updated Organization data and fresh notarizations. In other words, the Merkle tree is “additive” and is simply here to provide another, more efficient way to prove organization data freshness for inactive organizations.
More formally, the logic to check Organization data freshness has two steps:
1. **Notarization Freshness Check** - If the latest Notarization for an Organization is sufficiently fresh, we are done, the data is considered fresh.
2. **Merkle Tree Inclusion & Freshness Check** - If the Notarization Freshness Check fails, then the following check is performed:
1. Check if the Merkle Tree Inclusion Proof is valid. This proves the Notarization is part of the Merkle Tree
2. Check the signature payload of the root node of the Merkle tree, and ensure its timestamp is fresh enough.
The key advantage of this system is that these two mechanisms exhibit a symbiotic relationship, wherein the burden of one is greatly eased by the role of the other:
* Introducing the Merkle Tree as an archival mechanism for stale Notarizations negates the need to perform continuous refreshes. Only the Merkle tree root node needs to be periodically timestamped and signed.
* Keeping the Notarization Freshness check as a “first-pass check” allows to shield the Merkle Tree from the responsibility of accommodating Organizations which mutate at a very frequent rate. This works much like a [low-pass filter](https://en.wikipedia.org/wiki/Low-pass_filter)
in electronics, which filters out high-frequency signals.
* * *
1
We named this enclave application “Notarizer” because it acts as the software version of what a human notarizer does. It reviews, and provides a seal of authenticity verifiable by other parties.[↩](https://whitepaper.turnkey.com/architecture/#ref:1)
2
We have a dashboard for admin purposes, hosted at app.turnkey.com. But this isn't where most of the production traffic flows.[↩](https://whitepaper.turnkey.com/architecture/#ref:2)
3
You're probably wondering where wallet accounts are stored. Because wallet accounts are deterministically derived from seed, we only store the seed in organization data and re-derive accounts pre-signing.[↩](https://whitepaper.turnkey.com/architecture/#ref:3)
4
We chose to use JSON for human-readability but we had to deal with non-determism: because of field ordering and whitespace, there are multiple valid JSON strings for a single in-memory JSON object. This is okay in most cases: as long as the signed serialized JSON can be parsed, two components can load JSON strings, verify signatures, and extract data. When deterministic serialization is needed internally we use [Borsh](https://borsh.io/)
instead of JSON.[↩](https://whitepaper.turnkey.com/architecture/#ref:4)
5
For example, `ACTIVITY_TYPE_CREATE_USERS_V2` is the third version of our `CREATE_USERS` activity. The previous activity types were `ACTIVITY_TYPE_CREATE_USERS` and `ACTIVITY_TYPE_CREATE_USERS_V1` and had a different structure to their parameters or results. We support many activity versions into the past, maintaining a significant level of backwards compatibility.[↩](https://whitepaper.turnkey.com/architecture/#ref:5)
6
Policies only apply to Activities. For read-only Queries, which are non-critical requests, authorization is much simpler: they're allowed as long as Authentication succeeds, which means any user within an organization has read-only access to its data. Over time we intend to create RBAC-style roles to scope visibility of organization resources when required.[↩](https://whitepaper.turnkey.com/architecture/#ref:6)
7
Time is an exception: the Policy Engine must use its secure source of time to reject expired Activity requests (we consider any request older than 1hr expired). Injecting time as input would be insecure because we do not trust the coordinator in our threat model. Time needs to come from the NSM.[↩](https://whitepaper.turnkey.com/architecture/#ref:7)
8
There are other non-data differences at the authorization level: (1) parent orgs can have read-only access to sub-org, (2) parent orgs can initiate authentication or recovery activities for their sub-organizations, (3) parent orgs are allowed to create sub-orgs while sub-orgs can't create sub-orgs (the organization “tree” has max depth of exactly one)[↩](https://whitepaper.turnkey.com/architecture/#ref:8)
9
This “dependency” is implicit in the system design, rather than an outright API, or library type dependency. In this section we say “dependency” when we mean “trust relationship”. The actual mechanism for this is pinning of the Quorum public key: an enclave A trusts another enclave B when it pins B's Quorum public key and uses it to verify its responses. From an infrastructure or system point of view, enclaves cannot communicate with one another: enclave B's response has to be injected into requests for A by the coordinator.[↩](https://whitepaper.turnkey.com/architecture/#ref:9)
10
The only exception to this is the initial Create Organization activity. It is signed by an untrusted key because it's signed by the Organization's first Root user (new by definition). This is a standard Trust-On-First-Use (TOFU) solution where we assume the initial User to be legitimate. This is safe because an Organization starts completely empty. If the first activity is indeed malicious, the attacker does not gain anything aside from access to brand new, empty Organization.[↩](https://whitepaper.turnkey.com/architecture/#ref:10)
11
NSM stands for Nitro Secure Module. We built our initial implementation on top of AWS Nitro but hope to move towards being able to execute enclaves in any TPM 2.0 environment. TPM 2.0 has a well-defined API to source entropy inside of the secure environment: [`getrandom`](https://github.com/tpm2-software/tpm2-tools/blob/master/man/tpm2_getrandom.1.md)
.[↩](https://whitepaper.turnkey.com/architecture/#ref:11)
12
This can be done entirely with vanilla Javascript: these key pairs are standard P-256 key pairs. We also offer [SDK](https://github.com/tkhq/sdk)
abstractions to do this for apps and users who do not want to write code themselves.[↩](https://whitepaper.turnkey.com/architecture/#ref:12)
13
Providing non-repudiation via other means has been tried before but has failed thus far. See [TLS Evidence](https://datatracker.ietf.org/doc/draft-housley-evidence-extns/)
and [TLS sign](https://datatracker.ietf.org/doc/draft-hajjeh-tls-sign/)
.[↩](https://whitepaper.turnkey.com/architecture/#ref:13)
---
# Turnkey Whitepaper
* [Index](https://whitepaper.turnkey.com/)
* [Principles](https://whitepaper.turnkey.com/principles)
* [Foundations](https://whitepaper.turnkey.com/foundations)
* [Architecture](https://whitepaper.turnkey.com/architecture)
* [Applications](https://whitepaper.turnkey.com/applications)
Key Management Re-Imagined from First Principles
================================================
Turnkey Team
January 2025
[](https://whitepaper.turnkey.com/principles/#abstract "Copy link to this section")
Abstract
----------------------------------------------------------------------------------------------------------------------------------------------
Every transaction in crypto starts and ends with a private key. Creating, storing, managing, and using private keys is an extremely hard problem to solve. Since digital asset ownership and usage is all rooted in public key cryptography, solving this problem is an essential need for the industry.
Turnkey is a novel approach to key management built for the next wave of crypto applications and users. Our team is composed of industry veterans[1](https://whitepaper.turnkey.com/principles/#1)
who spent years in the trenches managing keys through multiple bull and bear cycles, securing hundreds of billions of dollars worth of crypto. The pain of struggling with non-optimal solutions has given our team the grit and hunger needed to invent a better solution for the future. We are now building the foundation we wish we had during those years.
Turnkey is low-level key management infrastructure that can be used by a business to automate transactions at scale, or to provide non-custodial wallets for end users. This includes everything from high-throughput payments, to smart contract management, to cross-chain consumer wallets, and more. The foundations we've laid are useful even beyond crypto. From the outside, this may seem like a simple product, but the power of Turnkey is in the details. We've built Turnkey to avoid many of the pitfalls of existing key management solutions. In this document we lay out the principles we've used to architect it:
1. **Meet users where they are**: Most insiders pride themselves on being able to deal with the friction of crypto UX, as a badge of honor to distinguish themselves from the mainstream. We're betting on familiar authentication standards like passkeys, email auth, and OAuth to meet users where they are and drive the next wave of adoption.
2. **Build on shared cryptographic primitives**: Transactions and assets are the wrong level of abstraction to start at in modern crypto. We've chosen to bet on curve-level operations and signing schemes because this is where the shared cryptographic fundamentals live.
3. **Engineer for speed and scale**: High-throughput payments, cross-chain abstractions, and AI agents interacting with crypto demand readily available keys and lightning fast signing. We've designed Turnkey as a pure key management product to avoid doing anything superfluous. We deliver sub-100ms signing latency and scale to millions of wallets.
4. **Assume everything is compromised until proven secure**: All sensitive actions run within our trusted, verifiable environment. We are securing the entire perimeter of the signing process rather than just key material. All requests to modify data are signed by a user-held authenticator, verified and processed entirely within secure enclaves.
5. **Don't trust, verify**: We divided Turnkey's infrastructure into “trusted” and “untrusted” spaces. The Trusted space, where all critical software runs, is verifiable with remote attestations. This is an industry first.
6. **Be pragmatic, not dogmatic**: Decentralizing portions of Turnkey will make sense in the long run. In the meantime, our priority is builders and their users.
7. **Build a library, not a framework**: We built Turnkey as a library full of modular building blocks, rather than a restrictive framework. It is the Unix philosophy applied to key management.
By the end of this document you should be able to understand the motivations behind Turnkey's [Verifiable Foundations](https://whitepaper.turnkey.com/foundations/)
and [Architecture](https://whitepaper.turnkey.com/architecture/)
, and be ready to jump to our technical designs with that context in mind.
[](https://whitepaper.turnkey.com/principles/#1-meet-users-where-they-are "Copy link to this section")
1\. Meet users where they are
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
We will not onboard the next billion users or trillion machines with seed phrases. Despite improvements to wallets over the years, the cryptocurrency space is fundamentally limited by its ability to onboard new users[2](https://whitepaper.turnkey.com/principles/#2)
. Both novice and experienced users encounter issues with wallet creation and private key material management, in raw or mnemonic form. These challenges can lead to errors and unacceptable financial losses. Key management is anxiety-inducing for experienced users who are aware of the security pitfalls, and a source of friction and unknown risks for novice users.
Onboarding flow for a new Metamask user
While most insiders agree that crypto onboarding is too hard, they pride themselves on being able to deal with that friction. It's a badge of honor to distinguish themselves from the mainstream. Crypto veterans can bear the burden; others can't. Crypto veterans can reason about self-custody and security, others can't. Dealing with the stress and fear is just the “price to pay” to do business in crypto. Now is the time to evolve and get rid of this piece of our identity. Crypto shouldn't be hard to use — let's bring crypto mainstream.
For a long time the crypto industry struggled to scale. Transaction fees were an issue because of scarce block space, and apps struggled to offer consistent user experiences as a result. We now have robust decentralized infrastructure for transaction processing, and plenty of block space to use. Throughput has gotten better as well (Solana, ETH L2s). The next thing to scale is crypto's adoption.
Turnkey meets users where they are: we're betting on familiar authentication standards like [passkeys](https://docs.turnkey.com/passkeys/introduction)
, [email auth](https://docs.turnkey.com/features/email-auth)
, and [OAuth](https://docs.turnkey.com/features/oauth)
. We're leaving passwords out for security reasons. Turnkey, as a principle, does not store any user authentication secret. Our authentication methods rely on public/private key cryptography. We store public keys only, and authentication happens with cryptographic signatures verified against them. To bridge traditional authentication protocols like email and OAuth to strong cryptographic key pairs we leverage Trusted Execution Environments (TEEs) to perform a key exchange. More on this in [Architecture](https://whitepaper.turnkey.com/architecture/)
.
Turnkey is built to onboard users and machines alike. Passkeys, email and Oauth are designed for human end-users, but API keys are a great fit for machines and agents. Over time we expect servers, AI agents, and other kinds of infrastructure to authenticate to Turnkey and adopt crypto as their primary rails.
[](https://whitepaper.turnkey.com/principles/#2-build-on-shared-cryptographic-primitives "Copy link to this section")
2\. Build on shared cryptographic primitives
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Traditional key management products struggle to keep up with new chains, ecosystems, and use-cases because they have overfit their APIs to asset-specific, transaction-level primitives. As a result, builders are hobbled by their custodians or key management providers. Although these may be simple abstractions for developers, it isn't possible to keep up with the explosion of ecosystems and use-cases without crumbling under technical debt and maintenance burden.
Our diagnosis: transactions and assets are the wrong level of abstraction to start at in modern crypto. A system designed specifically for Bitcoin might work well for handling Bitcoin transactions but will require significant rework to support Ethereum or Cosmos because of the differences in how transactions are structured and encoded. Instead, Turnkey operates at the curve level. Since Bitcoin, Ethereum, and Cosmos all use the Secp256k1 curve, Turnkey can support key generation and signing for these ecosystems without requiring asset-specific integrations.
We've chosen to bet on curve-level operations and signing schemes because this is where the shared cryptographic fundamentals live. Bitcoin, Ethereum, and Cosmos sign with Secp256k1. Solana, Polkadot, Stellar, Sei and Sui all sign using Ed25519. With two curves we cover the vast majority of crypto assets. We have the ability to easily add new curves, and we can support virtually all of them with fewer than 10 distinct curves total.
That said, to make integration easier, builders need more than curve-level operations to offer applications to their users. We approach asset and ecosystem-specific work with a [tiered approach](https://docs.turnkey.com/documentation/ecosystem-integrations/)
to make popular integrations easier. Higher tiers provide more robust abstractions, allowing for easier integration.
* **Tier 1**: Curve-level support. Cryptographic curves are our fundamental primitive, allowing Turnkey private keys to store and sign for any cryptocurrency that uses a supported curve. We currently support Secp256k1 and Ed25519 curves.
* **Tier 2**: Address derivation. Turnkey abstracts address generation, automatically deriving addresses for supported cryptocurrencies.
* **Tier 3**: Client-side SDK for transaction construction and signing. Our SDK provides tools and scripts to help in constructing and signing basic transactions, enabling an even smoother integration.
* **Tier 4**: Transaction parsing and policy creation. At our highest level of support, Turnkey offers the ability to parse transactions and define custom policies based on transaction parameters.
Curve-level support lets our customers move fast; you'll never be hard-blocked by Turnkey’s engineering team. Turnkey is built as extension-friendly infrastructure. We're betting on safe, broadly applicable cryptography first. We'll continuously go after more primitives as they become available and mature over time.
[](https://whitepaper.turnkey.com/principles/#3-engineer-for-speed-and-scale "Copy link to this section")
3\. Engineer for speed and scale
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Key management providers have historically cornered themselves because they’ve designed for low-throughput use cases. The emergence of high-throughput payments, cross-chain abstractions, and AI agents interacting with crypto demands readily available keys and lightning fast signing. Cold storage won't cut it. The largely “buy and hold” era of crypto is over.
We've designed Turnkey as a pure key management product to avoid doing anything superfluous. User private keys are generated and used inside attestable secure enclaves running [QuorumOS](https://github.com/tkhq/qos)
, a minimal Linux kernel (to read more about our foundations, see [Verifiable Foundations](https://whitepaper.turnkey.com/foundations/)
). Our enclave applications are written in Rust, and deployed on modern, secure hardware. They run unhampered, as close to native performance as it gets. As a result, we deliver sub-100ms signing latency, and this number includes all the ingress networking overhead, going in and out of our edge infrastructure.
Turnkey was built on modern infrastructure to horizontally scale to the needs of our customers. We designed key generation to be a low overhead, synchronous operation. Keys are created and ready to use in 100-200ms, end-to-end. As a result Turnkey is able to onboard end-users or machines at breakneck speeds and scales to hundreds of millions of wallets. This is what modern crypto infrastructure should feel like.
[](https://whitepaper.turnkey.com/principles/#4-assume-everything-is-compromised-until-proven-secure "Copy link to this section")
4\. Assume everything is compromised until proven secure
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Turnkey does more than just protect from private key theft. Aside from the flexibility and crypto-friendly design of our APIs, we offer markedly better security. The overwhelming majority of platforms that use words like “HSM”, “Enclave”, or “TEEs” are using off-the-shelf services like [SafeNet HSMs](https://thalesdocs.com/gphsm/luna/6.3/docs/network/Content/overview/product_line/sa_about_luna_sa.htm)
, [AWS KMS](https://aws.amazon.com/kms/)
or [Google KMS](https://cloud.google.com/kms/docs/key-management-service)
. These offer decryption or signing as a service without any validation or context about what is being decrypted or signed. A traditional Linux server makes requests to these secure services, telling them what to decrypt or sign, and they will always decrypt or sign. Hence an attacker doesn't need to compromise these secure services: compromising the traditional server is enough!
Modern crypto key security requires secure key storage of course, but we must extend security more broadly to key **usage**. Attackers do not need to steal private keys to steal crypto assets: malicious authentication, transaction parsing, or policy evaluation are enough to drain wallets.
While these off-the-shelf services will most likely stop private keys from being exfiltrated, they will not stop wallets from being drained. Once the funds are gone, the key being private no longer matters. It's too late: no use in closing the barn door after the horse has already run off.
One of our core beliefs is that the trusted environment must be smart enough to say “yes” or “no” with the full context of the end-user request. Otherwise we've simply shifted trust from the machine holding the key to a machine that communicates with it. And unfortunately, attackers always go after the weakest link.
Turnkey secures all sensitive actions with the same level of paranoia: the services generating private keys, producing signatures, parsing transactions, or evaluating policies all run within our trusted, verifiable environment (our [Trusted Computing Base](https://en.wikipedia.org/wiki/Trusted_computing_base)
, or TCB, see next section for details). We are securing the entire **perimeter** of the signing process. An enclave verifies passkey signatures on user requests. An enclave parses unsigned transactions to extract metadata. An enclave enforces policies. An enclave signs with your private key. Data defining organizations, users, policies, wallets, and private keys are cryptographically signed so they cannot be modified or rolled back to a previous point in time. All requests to modify data (Turnkey “activities”) are signed by a user-held authenticator, verified and processed entirely within secure enclaves.
Our top-level goal is to guarantee no private key or wallet can be accessed maliciously. Software outside of our TCB must not be able to modify data, and must not be able to modify or trigger user activity requests. We assume it can be compromised.
We have engineered Turnkey organization data and activities to be unforgeable and immutable. We'll see in [Turnkey's Architecture](https://whitepaper.turnkey.com/architecture/)
how this all comes together in greater detail.
[](https://whitepaper.turnkey.com/principles/#5-don-t-trust-verify "Copy link to this section")
5\. Don't trust, verify
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------
The crypto industry and the key management space in particular is full of wild and unverifiable security claims. We built Turnkey with transparency in mind: our security claims are real, tangible, and verifiable. We can prove we have nothing up our sleeves.
The standard approach to key management is to generate and manage keys hidden behind redundant layers of defense. Key management software and its operators are typically shrouded in secrecy and unaccountable for their actions.
Turnkey sets itself apart with an ambitious threat model which considers everything potentially compromised **by default**. We decide to trust only components that can be externally verified, and place our trust in **quorums** of operators to eliminate classic single-of-point-of-failure risks, all too common in our industry.
Trusted vs. Untrusted spaces
In order to achieve this, we divided Turnkey's infrastructure into “trusted” (Verifiable) and “untrusted” (Unverifiable) spaces.
“Untrusted” space exists as a bridge from the outside world to the trusted space. Our threat model assumes anything within this space (software, hardware, individuals) can be compromised. This includes edge infrastructure, ingress services, databases, configuration services, CI pipeline, deployment tooling, and so on. Untrusted space is secured in the traditional way: with best practices and defense-in-depth. Auditors and insiders can verify it runs correctly and safely with the right level of access, but regular users can't.
“Trusted” space contains critical software to create keys, authorize key usage, and use keys. Application binaries running in this space must be compiled with trusted tools, from trusted code, and approved by a quorum of operators. We secure the trusted domain with Trusted Execution Environments (“TEEs” aka “secure enclaves”), and anyone can verify what runs in Turnkey's secure enclaves through the process of remote attestation. Remote attestations are signed measurements provided by the underlying hardware. This allows anyone with an attestation to know with confidence that a specific piece of software is running on a specific machine. It is the basic building block for **verifiability**. We cover this in greater detail in [Verifiable Foundations](https://whitepaper.turnkey.com/foundations/)
.
This “trusted” space is Turnkey's [Trusted Computing Base](https://en.wikipedia.org/wiki/Trusted_computing_base)
(TCB). The ambitious bet we're making with Turnkey is to make our TCB open and verifiable. This is an industry first: the goal is for anyone to be able to verify key generation and usage independently. When in doubt, don’t trust—verify.
[](https://whitepaper.turnkey.com/principles/#6-be-pragmatic-not-dogmatic "Copy link to this section")
6\. Be pragmatic, not dogmatic
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Crypto moves slowly in the name of decentralization. Consensus needs to form across people and teams to alter how blockchains work at the protocol level. This is often the right process: enshrining something at the base layer requires careful design. In many ways it's irreversible, akin to etching a circuit during chip manufacturing.
Turnkey is a bet on verifiability as a supplement to full decentralization. A great example of this is authentication into Ethereum: Smart Wallets are now native ([EIP4337](https://www.erc4337.io/)
), but it'll take a while before passkey verification ([RIP7212](https://github.com/ethereum/RIPs/blob/master/RIPS/rip-7212.md)
) is supported natively across all Ethereum rollups and its base layer. And how long before passkey signature verification is available across Bitcoin, Solana, and others? In the meantime, Turnkey unlocks passkey authentication in a chain-agnostic way and powers crypto apps which would otherwise not exist.
The need for decentralization is a spectrum. It is low at the start and grows steadily as usage and value accrues.
We're pragmatic and not dogmatic about this. Decentralizing portions of Turnkey will make sense over time. And we track native primitives to offer the best integration possible where it makes sense (see our [AA guide](https://docs.turnkey.com/reference/aa-wallets)
for example). In the meantime, our priority is builders and their users. We hope Turnkey can be leveraged as a lab for the crypto industry to gauge which primitives are the most useful to end-users, and thus worthy of the effort to enshrine them in base protocols.
[](https://whitepaper.turnkey.com/principles/#7-build-a-library-not-a-framework "Copy link to this section")
7\. Build a library, not a framework
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Turnkey is engineered to be usable infrastructure. We're operators, not custodians. We allow developers to create and manage cryptographic private keys, rather than opening accounts and managing funds for them.
Only the person or machine in possession of authenticators (in the form of passkeys or bare API keys) can sign Turnkey activities and use the associated wallets or private keys. This setup is flexible enough to create experiences where:
* End-users hold authenticators directly and are in full control of their wallets (non-custodial).
* A business creates and holds authenticators on behalf of its end-users or itself (custodial setup).
* Business and end-users must cooperate on signing activities, leveraging the consensus features of our policy engine.
We do not make assumptions about who holds credentials and how they use Turnkey. Our goal is to provide safe, flexible, and performant foundations to build in crypto.
In terms of approach, we built Turnkey as a library full of modular building blocks, rather than an all-encompassing framework you have to fit into. You can use Turnkey standalone, or combine Turnkey with other ecosystem primitives, or even other providers. Turnkey can be combined with AA wallets, Gnosis safes, or a Bitcoin multisig. Turnkey can be used to deploy smart contracts. It can be used to sponsor gas, bridge funds, or safeguard domain names.
We provide the ability to [import](https://docs.turnkey.com/wallets/import-wallets)
and [export](https://docs.turnkey.com/wallets/export-wallets)
your keys so you're free to come and go anytime. We envision Turnkey to be useful, but we don't mandate it to be used the way we like. You can use Turnkey the way you intend to, using the parts of our API that make the most sense for your use case.
Another way to think about Turnkey: it is the Unix tool philosophy applied to key management. We've gone to great lengths to solve key management the best we can, and can't wait to see what gets built on top, crypto or not: The verifiable foundations we've laid to build Turnkey are useful infrastructure even beyond crypto. More on this in [Applications Beyond Key Management](https://whitepaper.turnkey.com/applications/)
.
[](https://whitepaper.turnkey.com/principles/#conclusion "Copy link to this section")
Conclusion
--------------------------------------------------------------------------------------------------------------------------------------------------
We started by looking at the UX problem in crypto and saw why now is the time to re-imagine key management with end-users and machines in mind. We've placed our bet on asymmetric cryptography, passkeys, email and OAuth to meet users where they are, removing decade-old friction and stigma associated with crypto onboarding.
We've detailed our ambitious security model and explained our belief that transparency and multi-party controls are vital to secure the critical portions of Turnkey. We've also seen why we've chosen TEEs as building blocks: the process of remote attestation is how you can keep us accountable.
Design-wise we believe curve-level operations are the correct building blocks for modern crypto-asset operations. Turnkey is built with low-latency and high-throughput in mind to enable bleeding-edge crypto apps that would otherwise not exist.
Turnkey operates on top of, and composes with, existing decentralized infrastructure. Our loyalty lives with builders and their users. We hope Turnkey can be used as a lab to determine which primitives are worth enshrining in existing decentralized protocols.
Finally, we're an infrastructure platform, not a custodian. Turnkey is a sharp, focused set of tools to build in crypto. Think library, not framework.
Retrofitting our vision into existing key management systems is near-impossible. This is why we're on this journey. With that, let's dive into the nitty-gritty details of our foundations with [Verifiable Foundations](https://whitepaper.turnkey.com/foundations/)
.
* * *
1
The Turnkey team includes people who have worked at Coinbase, Kraken, BitGo, Fireblocks, and within the US government defense industry.[↩](https://whitepaper.turnkey.com/principles/#ref:1)
2
See [this](https://dl.acm.org/doi/fullHtml/10.1145/3411764.3445407)
, [this](https://dl.acm.org/doi/fullHtml/10.1145/3613904.3642534)
, or [this](https://ieeexplore.ieee.org/document/9315193)
for academic papers on the topic.[↩](https://whitepaper.turnkey.com/principles/#ref:2)
---
# Turnkey Whitepaper
* [Index](https://whitepaper.turnkey.com/)
* [Principles](https://whitepaper.turnkey.com/principles)
* [Foundations](https://whitepaper.turnkey.com/foundations)
* [Architecture](https://whitepaper.turnkey.com/architecture)
* [Applications](https://whitepaper.turnkey.com/applications)
Verifiable Foundations
======================
Turnkey Team
January 2025
[](https://whitepaper.turnkey.com/foundations/#abstract "Copy link to this section")
Abstract
-----------------------------------------------------------------------------------------------------------------------------------------------
Here we dive straight into the depth of Turnkey's foundations and how they allow for Turnkey applications to be independently verifiable.
We briefly explain what Trusted Execution Environments (“TEEs”) are and how we use them. On top of strong isolation and confidentiality guarantees, TEEs can prove the software they run through remote attestations. These attestations contain signed measurements from the underlying platform provider (“Platform Configuration Registers”, or “PCRs”).
We introduce [QuorumOS](https://github.com/tkhq/qos)
(“QOS”), a new minimal, open-source operating system engineered for verifiability. QuorumOS is the operating system run by TEEs, acting as the glue between provider-specific hardware and provider-agnostic applications. QuorumOS proves that a TEE is running a specific application by providing QOS Manifests to attestations. This lets anyone verify that an enclave is running the correct software (QuorumOS itself), and that QuorumOS is running the correct application binary (as specified in the QOS Manifest).
We explain why remote attestations require reproducible builds and introduce [StageX](https://codeberg.org/stagex/stagex)
, a new Linux distro which powers all secure builds at Turnkey today. StageX provides reproducible builds to ensure application binaries can be reproduced, agreed upon, and certified by multiple parties. StageX guarantees a 1-to-1, immutable relationship between human-readable source code and the resulting machine-executable artifacts running inside of QuorumOS.
Finally we introduce the concept of Boot Proofs and App Proofs and explain how remote attestations, QuorumOS, and StageX combined together yield full verifiability of the software running inside TEEs. The entire operating system (QuorumOS itself) as well as applications within it are verifiable all the way down to the exact source code.
What we present here is a major step forward compared to the current opaque status quo where the security of critical software components can't be proved and relies instead on huge audit and compliance industries.
[](https://whitepaper.turnkey.com/foundations/#tees-and-aws-nitro-enclaves "Copy link to this section")
TEEs and AWS Nitro enclaves
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
In this section we explain what Trusted Execution Environments (“TEEs”) are and how Turnkey uses them. On top of strong isolation and confidentiality guarantees, TEEs provide access to secure sources of time and entropy. The most interesting feature of TEEs is Remote Attestations, which we introduce in a dedicated section. Remote attestations are how TEEs can prove the software they run, with signed measurements from the underlying platform provider (“Platform Configuration Registers”, or “PCRs”).
### [](https://whitepaper.turnkey.com/foundations/#overview "Copy link to this section")
Overview
Trusted Execution Environments are dedicated areas within a device or system to provide extra confidentiality and integrity guarantees when running secure workloads. TEEs are paired with a “host” system which calls into the secure area to invoke functionality hosted within it. TEEs are designed to protect against malware and OS-level vulnerabilities on the host system, which means data and computation in TEEs remain confidential and intact.
Historically TEEs have been leveraged for many use cases and they come in a wide variety of hardware. They are governed by many different standards ([TPM](https://trustedcomputinggroup.org/resource/tpm-main-specification)
, [GlobalPlatform](https://globalplatform.org/specs-library)
, [JavaCard](https://www.oracle.com/java/technologies/javacard-specs-downloads.html)
, [FIPS](https://csrc.nist.gov/pubs/fips/140-2/upd2/final)
). Mobile device authentication on iOS (with the [Apple Secure Enclave](https://support.apple.com/guide/security/secure-enclave-sec59b0b31ff/web)
) or Android (with [Trusty](https://source.android.com/docs/security/features/trusty)
) is the most well-known use case for TEEs because it underpins fingerprint and FaceID authentication, used daily by hundreds of millions of end-users. A good survey of TEEs and their classification is available in [Trusted Execution Environments (Shepherd, Markantonakis)](https://link.springer.com/book/10.1007/978-3-031-55561-9)
.
Turnkey uses [AWS Nitro Enclaves](https://aws.amazon.com/ec2/nitro/nitro-enclaves/)
as its primary platform today[1](https://whitepaper.turnkey.com/foundations/#1)
. Over time we'll expand our deployment capabilities to any TPM 2.0 compatible providers such as Google Cloud Platform ([Shielded VMs](https://cloud.google.com/security/products/shielded-vm)
), Azure ([Confidential Computing](https://learn.microsoft.com/en-us/azure/confidential-computing/application-development)
), or on-prem deployments. For the rest of this document we'll use AWS-specific language to make the explanation more concrete, but the concepts carry over to other vendors.
AWS Nitro enclaves can be visualized in the following diagram:
AWS Nitro enclave components
On the left we have the “network”, which is to say: the outside world. Other applications or humans connect to the Nitro Host over TCP or UDP connections. A Nitro Host is “just” another EC2 instance!
The Nitro Host runs a Host VM, depicted in blue, running its own operating system. What makes a Nitro Host special is its access to the [Nitro Controller and Nitro Hypervisor](https://docs.aws.amazon.com/whitepapers/latest/security-design-of-aws-nitro-system/the-components-of-the-nitro-system.html#the-nitro-hypervisor)
. The Nitro Controller lets the Nitro Host issue commands like “start” or “stop” to boot and terminate Nitro enclaves[2](https://whitepaper.turnkey.com/foundations/#2)
. New nitro enclaves are created from an Enclave Image File (EIF)[3](https://whitepaper.turnkey.com/foundations/#3)
. Booting a new Nitro enclave results in a new isolated VM with its own resources (RAM, CPU), running the specified EIF, on the same physical host. We label the resulting VM “Enclave VM”, in green.
Running heavy workloads in TEEs is generally a challenge because of the physical limitations of secure elements. However, Nitro enclaves can be provisioned with an arbitrary amount of CPU and memory (as long as it's less than the total amount of CPU and memory available on the host). Thanks to this design decision, secure enclaves can run arbitrary applications and consume as much memory and CPU as the host system is willing to allocate.
One crucial security feature is the VSOCK link between the host VM and the enclave VM. [VSOCK](https://man7.org/linux/man-pages/man7/vsock.7.html)
is a socket-like protocol to facilitate VM-to-VM communication. This VSOCK link is the only way in and out of the enclave. Nitro enclaves do not have any other data I/O, and in particular, they do not have any networking capabilities.
Finally, the Nitro Card, in orange, is a physical resource only accessible to the Nitro Enclave. The Nitro Secure Module (NSM) connects to it[4](https://whitepaper.turnkey.com/foundations/#4)
to provide a secure source of entropy and time, which the enclave VM can access.
Secure entropy is crucial not only for key generation, but also relevant to generate random nonces at signing time, or random Initialization Vectors (IVs) when encrypting data.
The secure source of time is used to protect against replay and downgrade attacks. It's also used to verify the validity of SSL certificates, for example. Time is a natural monotonically increasing counter so it can serve as a secure nonce or watermark when required.
Summarizing: TEEs are dedicated areas within a device or system to provide confidentiality and integrity guarantees when running secure workloads.
Turnkey uses AWS Nitro enclaves, a specific type of TEEs, which are isolated (virtual) machines provisioned with their own CPU and memory resources. They have the following important properties:
* A Nitro enclave is **stateless** and does not have the ability to write to a persistent disk or cache. Its only form of persistence is volatile memory (RAM), cleared on every restart[5](https://whitepaper.turnkey.com/foundations/#5)
.
* A Nitro enclave is **not connected to the network**. The only networking element attached to a Nitro enclave is a VSOCK interface to enable communication with the enclave host.
* A Nitro enclave has access to an independent **secure source of entropy and time** via the Nitro Security Module (“NSM”)
Finally, it's worth double clicking on one key feature of Nitro enclaves so far: **remote attestations** (AWS specific docs [here](https://docs.aws.amazon.com/enclaves/latest/user/set-up-attestation.html)
). Within AWS, the Nitro Card measures and signs Platform Configuration Registers (PCRs) with a trusted key to produce an attestation document. This document is what anyone in the world can **verify**, to ensure that a particular enclave is running exactly what it should be running without having physical access to the underlying hardware. Let's dive into remote attestations in more detail.
### [](https://whitepaper.turnkey.com/foundations/#remote-attestations "Copy link to this section")
Remote Attestations
An attestation document, at its core, is **a signed message from an enclave**. The message contains information, and the signature comes from a key pair generated inside the enclave when it boots. Let's dive into the following:
* What structure does this “message” have?
* Who generates the key pair signing attestation documents? Why should we trust it?
#### Attestation message
An attestation message is structured binary data[6](https://whitepaper.turnkey.com/foundations/#6)
, encoded using [CBOR](https://cbor.io/)
. The important fields contained in each attestation are:
* **`PCR0`**: Hash of the Enclave Image File (EIF). EIFs are what the Nitro enclaves boot with. Turnkey's enclaves all use [QuorumOS](https://github.com/tkhq/qos)
as an EIF. The recipe to build it is [here](https://github.com/tkhq/qos/blob/b960756a67b4b6f0dac16f5388dc9555bd8978a2/src/images/qos_enclave/Containerfile#L49-L55)
.
* **`PCR1`**: Linux kernel and initial RAM data hash (aka [initramfs](https://en.wikipedia.org/wiki/Initial_ramdisk)
)
* **`PCR2`**: Hash of user applications, without the boot ramfs. For Turnkey enclaves, `PCR2` and `PCR1` measurements are identical because we do not use [`pivot_root`](https://man7.org/linux/man-pages/man2/pivot_root.2.html)
, and simply use `initramfs` as our final filesystem.
* **`PCR3`**: Hash of the IAM role of the Nitro host. Concretely this is the hash of an [AWS IAM](https://aws.amazon.com/iam/)
identifier (for example `arn:aws:iam::123456789012:role/nitrohost`).
* **`certificate`**: Contains an X.509 certificate specific to the enclave. It contains an enclave-specific public key. This enclave-specific key pair is generated during boot, and signs the attestation document.
* **`cabundle`**: Certificate chain to certify the “certificate” above.
* **`user_data`**: User-supplied field. QuorumOS sets it to be the hash of the QOS manifest [here](https://github.com/tkhq/qos/blob/b960756a67b4b6f0dac16f5388dc9555bd8978a2/src/qos_core/src/protocol/services/attestation.rs#L23)
(more on this later).
* **`nonce`**: User-supplied field, unused by QuorumOS at the time of writing.
* **`public_key`**: User-supplied field, set to the QOS ephemeral public key ([here](https://github.com/tkhq/qos/blob/b960756a67b4b6f0dac16f5388dc9555bd8978a2/src/qos_core/src/protocol/services/attestation.rs#L23)
).
All of the fields above are part of the signed payload (our attestation “message”).
#### Signature and chain of trust
Attestation signatures use P-384, a standard signature scheme defined by NIST (in [FIPS 186-4](http://nvlpubs.nist.gov/nistpubs/FIPS/NIST.FIPS.186-4.pdf)
). This signature is produced by a **new random key pair created by the enclave when it boots.**
This brand new public key is referenced inside of the attestation's `certificate` field. This certificate is signed by another certificate, all the way to a top-level AWS certificate. This certificate “chain” or “bundle” is contained in the `cabundle` field. The same concept of certificate chain underpins TLS: your browser trusts a website's TLS certs because of its certificate chain, going up to top-level Certificate Authorities (CAs) trusted by your browser or trusted by your operating system. AWS is a CA for the Nitro enclaves it owns and operates.
To verify that an enclave certificate originates from Amazon, the certificate chain ascends to the root certificate for the commercial AWS partitions as described [here](https://docs.aws.amazon.com/enclaves/latest/user/verify-root.html#validation-process)
. This top-level certificate is stable, can be downloaded, pinned, and checked. Indeed it's valid until **October 28th, 2049**! Here it is in its parsed form (note the `Not After` field):
$ openssl x509 -in ~/Downloads/root.pem -text -noout
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
f9:31:75:68:1b:90:af:e1:1d:46:cc:b4:e4:e7:f8:56
Signature Algorithm: ecdsa-with-SHA384
Issuer: C = US, O = Amazon, OU = AWS, CN = aws.nitro-enclaves
Validity
Not Before: Oct 28 13:28:05 2019 GMT
Not After : Oct 28 14:28:05 2049 GMT
Subject: C = US, O = Amazon, OU = AWS, CN = aws.nitro-enclaves
(...etc)
We trust AWS attestations because we trust Amazon as infrastructure operators. An attestation's CA bundle (in the `cabundle` field) transfers trust from Amazon's top-level certificate to the enclave-specific key which signs the attestation document.
If “→” means “signs”, we can summarize the chain of trust for an AWS remote attestation with:
* Root AWS cert → Intermediate cert 1
* Intermediate cert 1 → Intermediate cert 2
* …
* Intermediate cert N-1 → Intermediate cert N
* Intermediate cert N → Enclave X.509 certificate attesting to the enclave public key
* Enclave key pair → Remote Attestation document “message”
[](https://whitepaper.turnkey.com/foundations/#running-secure-applications-with-quorumos "Copy link to this section")
Running secure applications with QuorumOS
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
In this section we introduce [QuorumOS](https://github.com/tkhq/qos)
(“QOS”), a new minimal, open-source operating system engineered for verifiability. QuorumOS is our base operating system: it is the Enclave Image File (“EIF”) used to boot Turnkey enclaves.
In the previous section we've established that remote attestations prove that the correct EIF runs inside of an enclave. As a result it's possible for anyone in possession of a remote attestation to verify that an enclave runs the expected version of QuorumOS, by hashing and comparing a local EIF with the `PRC0` measurement contained within the attestation.
QuorumOS was engineered to run any application verifiably, on top of QuorumOS. Applications come in the form of binary artifacts. We'll explain in the sections below how we've designed QOS Manifests to contain not only the expected artifact digest, but also critical configuration such as quorum settings and public keys.
QuorumOS itself provides QOS Manifests as user data when fetching remote attestations. This lets anyone verify that an enclave is running the correct EIF (QuorumOS itself), and that QuorumOS is running the correct application binary (as specified in the QOS Manifest).
We've been using QuorumOS in production for 2+ years and it has gone through multiple rigorous audits, internal and external. We're excited to share this with the larger security community.
### [](https://whitepaper.turnkey.com/foundations/#what-is-quorumos "Copy link to this section")
What is QuorumOS?
QuorumOS (QOS) is an operating system and set of libraries to boot and provision[7](https://whitepaper.turnkey.com/foundations/#7)
applications in AWS Nitro enclaves. The repository, available at [github.com/tkhq/qos](https://github.com/tkhq/qos)
, is organized as a set of Rust crates (software packages) which are compiled into various places to either **boot** an enclave, or **provision** it. We'll see these flows in detail later in this document. For the moment let's look at a simplified diagram showing where each Rust crate fits:
QuorumOS crates and how they relate to enclave boot and provisioning
* [`qos_enclave`](https://github.com/tkhq/qos/tree/main/src/qos_enclave)
contains utilities to boot a new Nitro enclave from a host machine. It calls into the Nitro CLI to boot an enclave with a given EIF. This crate is used by Turnkey's infrastructure (Kubernetes) to boot new enclaves programmatically.
* [`init`](https://github.com/tkhq/qos/tree/main/src/init)
defines the enclave's init binary. It is the program which gets executed as PID 1 when QuorumOS boots. This crate is compiled into the QuorumOS base OS. This crate imports [`qos_aws`](https://github.com/tkhq/qos/tree/main/src/qos_aws)
to interact with the NSM to signal readiness, and [`qos_system`](https://github.com/tkhq/qos/tree/main/src/qos_system)
for lower level functionality.
* [`qos_core`](https://github.com/tkhq/qos/tree/main/src/qos_core)
defines a protocol and state machine to boot secure applications. The protocol is very straightforward: it's composed of [messages](https://github.com/tkhq/qos/blob/0f6728bf8f70322cf7feae4cedf4f3aad228d0ec/src/qos_core/src/protocol/msg.rs#L15)
that can be received or sent over the VSOCK connection, from host to enclave. The state machine[8](https://whitepaper.turnkey.com/foundations/#8)
is built on top of the protocol. This crate is used within application binaries to define the requests they can receive and respond accordingly.
* [`qos_nsm`](https://github.com/tkhq/qos/tree/main/src/qos_nsm)
is a simple wrapper crate to let Rust application code invoke the AWS NSM APIs for secure time or entropy.
* [`qos_client`](https://github.com/tkhq/qos/tree/main/src/qos_client)
is a CLI used to parse and verify manifests, and post shares of Quorum Keys into booted enclaves. When a QuorumOS enclave is provisioned with an application binary, a valid QOS Manifest, and enough approvals and shares, the enclave is provisioned. We describe the provisioning process in greater detail in the next section.
* [`qos_host`](https://github.com/tkhq/qos/tree/main/src/qos_host)
, not depicted in the diagram, is a library and CLI to build host-side servers which receive requests from the outside and talk to an enclave application to serve these requests.
* [`qos_net`](https://github.com/tkhq/qos/tree/main/src/qos_net)
, not depicted in the diagram, defines a VSOCK-to-TCP proxy server. It runs on the host side to provide optional external connectivity to enclave apps which need it, as well as Rust utilities to use it from inside secure applications. This connectivity is one-sided. With `qos_net` secure apps can choose to reach out to the outside, but not the other way around.
* Finally, [`qos_crypto`](https://github.com/tkhq/qos/tree/main/src/qos_crypto)
, [`qos_p256`](https://github.com/tkhq/qos/tree/main/src/qos_p256)
and [`qos_hex`](https://github.com/tkhq/qos/tree/main/src/qos_hex)
are libraries and abstraction implementing utility functions imported by previously listed crates.
Now that we've looked at the layout of the code, let's walk through the two important responsibilities fulfilled by QuorumOS: enclave **boot** and enclave **provisioning**.
### [](https://whitepaper.turnkey.com/foundations/#enclave-boot "Copy link to this section")
Enclave boot
Booting an enclave can be done by humans as a one-off, through the Nitro CLI, but we've automated the process with [`qos_enclave`](https://github.com/tkhq/qos/tree/main/src/qos_enclave)
. The main function provided by this crate is [`boot()`](https://github.com/tkhq/qos/blob/39e8b4194bdc9b68b8ae962d196951d60708a461/src/qos_enclave/src/main.rs#L64)
, which uses [aws/aws-nitro-enclaves-cli](https://github.com/aws/aws-nitro-enclaves-cli/)
to start a new enclave from a Nitro host, using information passed in as env vars: `EIF_PATH`, `MEMORY_MIB`, and `CPU_COUNT`.
In production settings this task is performed by our codified Kubernetes infrastructure.
Note that enclave boot is application agnostic. The process is the same regardless of which application needs to run, because the EIF file is application agnostic and only contains QuorumOS itself. The base operating system is a barebone Debian kernel (built [here](https://codeberg.org/stagex/stagex/src/commit/8cd3391db1b2880d87e927250ce6472d827128ad/packages/linux-nitro/Containerfile#L1-L5)
). We have [configured](https://codeberg.org/stagex/stagex/src/commit/aba8a21200bfa87e00c3dac95c22f9c76689e517/packages/user/linux/nitro.config)
it to minimize its footprint and attack surface. It secures all applications running on top of it.
The enclave boot process is provider-specific because it needs to interact with the Nitro APIs to launch enclaves, which are AWS specific. The boot process will need to evolve as we expand our deployment targets to Google Shielded VMS, Azure's confidential compute platform, or on-prem TEEs. External contributions are welcome in this area, check out [the QuorumOS repository](https://github.com/tkhq/qos)
if you want to start contributing!
### [](https://whitepaper.turnkey.com/foundations/#enclave-provisioning "Copy link to this section")
Enclave provisioning
Enclave provisioning is application-specific. It is how we turn a generic QuorumOS-booted enclave into an enclave running a particular binary, provisioned with a particular Quorum Key.
This process is provider-agnostic and largely remains the same regardless of the underlying infrastructure provider. Let's introduce some important terms before diving into the provisioning flow in detail:
* **QOS Manifest**: a binary file which contains configuration to specify a secure application. It contains the digest of the application binary, the Quorum Key (public key), the Share Set, the Manifest Set, expected PCR measurements, and any command-line arguments necessary to launch the application binary. The full specification is available [here](https://github.com/tkhq/qos/blob/39e8b4194bdc9b68b8ae962d196951d60708a461/src/qos_core/src/protocol/services/boot.rs#L310-L323)
.
* **Quorum Key**: an asymmetric key pair used to authenticate and encrypt data. This key should only ever be reconstructed inside of an enclave. Outside of the enclave, the key is stored as redundant shares (key is split using [Shamir's secret sharing](https://en.wikipedia.org/wiki/Shamir%27s_secret_sharing)
at genesis[9](https://whitepaper.turnkey.com/foundations/#9)
). These shares are encrypted to hardware keys held by humans (Turnkey operators or external operators). Never in plaintext, never outside of secure hardware.
* **Manifest Set**: collection of public keys and a threshold. The threshold indicates how many members are needed to approve the QOS Manifest. During the provisioning flow, enough members of the Manifest Set must approve the QOS Manifest to provision an enclave.
* **Share Set**: collection of public keys and a threshold. The public keys represent the members holding a Quorum Key share, and the threshold indicates how many shares are needed to reconstruct the Quorum Key. During the provisioning flow, enough members of the Share Set must post their Quorum Key shares.
* **Namespace**: a string identifying a group of individual enclaves provisioned with the same application. Enclaves within the same namespace share the same Quorum Key.
#### Provisioning sequence
When an enclave boots with the QuorumOS EIF, it immediately waits for provisioning to happen[10](https://whitepaper.turnkey.com/foundations/#10)
. To provision an enclave, three inputs are needed:
* The secure application binary
* The Manifest Envelope, which is composed of a QOS Manifest and enough approvals by its Manifest Set members
* Enough Quorum Key shares to reconstruct the Quorum Key
This is summarized in the diagram below, where we have (as an example) a 2-out-of-3 setting in our Manifest Set, and a 3-out-of-5 setting in our Share Set:
Provisioning inputs: application binary, Manifest Envelope, and encrypted shares
Provisioning is done in 2 steps:
* The application binary and the Manifest Envelope (which is a [QOS manifest](https://github.com/tkhq/qos/blob/0f6728bf8f70322cf7feae4cedf4f3aad228d0ec/src/qos_core/src/protocol/services/boot.rs#L310)
and cryptographic approvals bundled together) are sent together in a single request. QuorumOS checks the application binary against the QOS manifest (the digest of the binary needs to match the digest in the manifest), and checks that enough approvals have been provided in the manifest envelope. If everything is copacetic, QuorumOS transitions to the `WaitingForQuorumShards`[11](https://whitepaper.turnkey.com/foundations/#11)
state.
* Share Set members post their shares of the Quorum Key. Once enough shares are posted, the Quorum Key is reconstructed, and the enclave is fully provisioned with its core secret. In order to scale Turnkey and run it in modern clouds where underlying hardware can be shut down without notice, we had to design an alternative to this manual, cumbersome process: see [Appendix: scaling provisioning with Key Forwarding](https://whitepaper.turnkey.com/foundations/#appendix-scaling-provisioning-with-key-forwarding)
.
When an enclave is successfully provisioned, QuorumOS starts the application binary. External callers can send requests to the enclave application by sending a QOS [ProxyRequest](https://github.com/tkhq/qos/blob/39e8b4194bdc9b68b8ae962d196951d60708a461/src/qos_core/src/protocol/msg.rs#L68-L72)
messages. These messages contain raw bytes, unwrapped by QOS and passed along to the underlying application binary.
#### Secure share posting with remote attestations and Ephemeral Keys
We've explained above that the last step in the provisioning process is Quorum Key share posting. Here we answer the following questions:
* How do Share Set members know they're posting to the right machine?
* How are Quorum Key shares secured in transit?
After QuorumOS receives an application binary and a valid Manifest Envelope, it generates a new Ephemeral Key (a new asymmetric key pair – see code [here](https://github.com/tkhq/qos/blob/39e8b4194bdc9b68b8ae962d196951d60708a461/src/qos_core/src/protocol/services/boot.rs#L446-L447)
). This Ephemeral Key (public key) is referenced inside of the `public_key` field of AWS attestations. The flow for posting shares can be detailed as follows:
* The operator establishes a connection to a candidate enclave (in `WaitingForQuorumShards` state) and requests a remote attestation.
* The remote attestation contains a hash of the QOS manifest in the `user_data` field ([code link](https://github.com/tkhq/qos/blob/b960756a67b4b6f0dac16f5388dc9555bd8978a2/src/qos_core/src/protocol/services/attestation.rs#L29-L31)
), which the operator can compare to a locally computed digest of a copy they've previously reviewed. This ensures the candidate enclave has received the correct application binary and Manifest Envelope.
* The remote attestation contains the enclave Ephemeral Key (public key) in the `public_key` field. The operator, using ECDH, encrypts their Quorum Key share to the Ephemeral Key. This guarantees that only this particular enclave (which is provisioned with the correct binary and manifest envelope) is able to decrypt the share. The operator can thus send this encrypted payload over the network safely.
* The enclave receives the encrypted Quorum Key share, and decrypts it using their Ephemeral Key.
In practice, Quorum Set members follow strict runbooks when posting shares or approving manifests. For security reasons we mandate that members of Quorum Sets (either Share or Manifest Set) never expose their key material to an online machine. See [Appendix: airgapped workflows for Quorum Set members](https://whitepaper.turnkey.com/foundations/#appendix-airgapped-workflows-for-quorum-set-members)
for more information.
#### Summary
* QuorumOS provisioning is a crucial part of Turnkey's security because it enforces that the right logic and cryptographic checks are executed. In particular it enforces that the manifest is signed by enough members of the Manifest Set, and that the shares are posted by members of the Share Set who have reviewed the manifest.
* All Nitro enclaves boot with a generic QuorumOS EIF. Through the provisioning process they acquire an application binary, a signed manifest (“Manifest Envelope”), and a Quorum Key (reconstructed from shares posted by Share Set members).
* Once provisioned with a binary and Manifest Envelope, an enclave's attestations contain the hash of the QOS manifest in its `user_data` field and the enclave ephemeral key in its `public_key` field. This is fundamental to secure share posting.
A point worth re-stating: QuorumOS is responsible for providing the hash of the QOS manifest in the attestation's `user_data` field. **The QOS Manifest is the link between our application-agnostic QuorumOS EIF file and the application binary**. An AWS Nitro attestation contains a digest of the QOS manifest, and the QOS manifest contains a digest of the application binary.
[](https://whitepaper.turnkey.com/foundations/#reproducible-builds-through-stagex "Copy link to this section")
Reproducible builds through StageX
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
We now explain why reproducible builds are the last, crucial problem to resolve to run software verifiably: they create an immutable link between application binaries and their associated human-readable source code. Because the binary can be reproduced by anyone, human consensus around the **source code** becomes possible. This is the difference between proving “this enclave runs this binary digest” and “this enclave runs this source code”. With verifiability extending all the way to the source code, anyone can independently examine the enclave’s functionality in the finest detail. With a mere binary hash, that would not be possible. This is, unfortunately, the current status quo: Most TEE-based systems are not truly verifiable because they lack reproducible builds.
StageX is our answer to the reproducible build problem. It is a crucial component of Turnkey's verifiable foundations. It is [open-source](https://codeberg.org/stagex/stagex)
for all to see and use.
### [](https://whitepaper.turnkey.com/foundations/#why-are-reproducible-builds-required "Copy link to this section")
Why are reproducible builds required?
Humans write code in text format. This is generally referred to as “source code”. For compiled languages, source code is processed by a compiler to produce binary artifacts. These artifacts are the executables run by computers, and our Nitro enclaves are no exceptions: they accept an EIF file which runs QuorumOS, and QuorumOS accepts in turn a manifest and a application binary.
EIF file digests are referenced in AWS attestations, and application binary digests are referenced in QOS Manifests. It is of vital importance that these digests can be verified and mapped to the source code that produced them. As humans we can make sense of the source code but we can't understand binaries directly.
Although one would assume that each run of a compiler produces the exact same binary artifact, this isn’t true. The final binary generally depends on the system architecture, system library versions, system username, hostname, name of the folder you are building in, filesystem, system time, timezone, language, number of cores, core speed, linux kernel version, container runtime version, CPU brand and model, and more. Interestingly, these differences in the binary artifact do not always mean that the program will behave differently. Unfortunately there is no way to tell with certainty whether a difference is a “no-op” or would result in a (potentially malicious!) behavior change. That's because humans can't read binary artifacts directly. Tools like [Ghidra](https://ghidra-sre.org/)
exist but they aren't suitable for day-to-day use.
Because of this ambiguity (differences in binary artifacts can't be judged “good” or “bad” easily), a non-reproducible build causes an inherent single-point-of-failure: the human or machine building the source code and producing the binary artifact has to be trusted to faithfully compile the code, without modifying it. Everyone else has to trust that it is indeed the case.
Unfortunately the process to go from source code to binary artifacts (“the build”) is generally not well-known or cared for. We've often seen this be run by a single machine or set of machines responsible for other non-critical or outright insecure “development” workloads. This could be in the form of Github CI runners which automatically publish docker images, or self-hosted clusters which populate an internal store with tagged binaries. This build process is a major weak point because **it is an inherent single point of failure**. If the build process is compromised, arbitrary code can be snuck into secure applications. This class of exploits is generally known as “supply-chain attacks” and they're so common it'd be easy to fill pages of examples with real-world examples (here are a few famous ones: [one](https://www.theregister.com/2022/05/10/security_npm_email/)
, [two](https://www.bleepingcomputer.com/news/security/researcher-hacks-over-35-tech-firms-in-novel-supply-chain-attack/)
, [three](https://www.techrepublic.com/article/xz-backdoor-linux/)
, and most recently [four](https://decrypt.co/294742/solana-web3-js-library-compromised-in-targeted-supply-chain-attack)
). This is not a risk we can tolerate at Turnkey.
Without a reproducible build, the expensive social consensus formed by multiple (human) parties around the security of the **source code** is all for naught: two parties building the same agreed upon code revision will arrive at two different artifacts, with two different digests:
Non-reproducible build, where 2 separate builds result in different digests.
A reproducible build allows linking artifacts back to their source code in an immutable way: a single source code revision always yields the same binary artifact, byte-for-byte. As a result the digest is the same and can be signed by multiple parties:
A reproducible build guarantees separate builds result in the same final digest.
If multiple machines or humans can independently attest to the fact that a set of source files yields the same artifact, and thus the same digest, the single point of failure is gone. As a bonus, this scales well: it's possible to achieve arbitrarily strong consensus about digests: we simply need to ask arbitrarily many parties to reproduce these binary artifacts and cryptographically sign the resulting digest. We use this to our advantage to gain confidence about secure app binaries or EIF files for example.
### [](https://whitepaper.turnkey.com/foundations/#reproducible-builds-in-practice "Copy link to this section")
Reproducible builds in practice
The first version of reproducible builds at Turnkey used Debian containers as a base and [Toolchain](https://codeberg.org/distrust/toolchain)
to build them in a reproducible way. The main idea was to abstract away differences between build environments (such as [user and group IDs](https://codeberg.org/distrust/toolchain/src/branch/master/scripts/host-env)
, [number of CPUs](https://codeberg.org/distrust/toolchain/src/commit/2560e543a1871b86cb1c29b31ce9141eaec94e8c/Makefile#L324)
, [timestamp](https://codeberg.org/distrust/toolchain/src/commit/2560e543a1871b86cb1c29b31ce9141eaec94e8c/scripts/environment#L24-L25)
and [many others](https://codeberg.org/distrust/toolchain/src/commit/2560e543a1871b86cb1c29b31ce9141eaec94e8c/scripts/environment#L8-L39)
) with custom environment variables and system configuration baked into build processes via Makefile macros. This came with major downsides that slowed down developer productivity:
* Repositories needed to keep costly snapshots of all dependencies in Git LFS or similar to be able to reproduce the exact build container. Otherwise the “latest” packages that would otherwise be downloaded would shift over time. This created a lot of friction for our team having to regularly archive, hash-lock, and sign hundreds of `.deb` files for every project.
* Debian has very old versions of Rust, which we rely on heavily. This very frequently caused frustration when trying to upgrade external crates.
* The builds themselves relied heavily on Makefile and macros. Most engineers are not familiar with this syntax; as a result debugging builds was really hard.
After a few months with this setup, we concluded that something had to change. Today our secure builds are powered by [StageX](https://codeberg.org/stagex/stagex)
, a new Linux distro focused on immutable, reproducible packages distributed in the form of Docker [OCI](https://opencontainers.org/)
images. It builds on classical Stage 0-3 compiler [bootstrapping](https://en.wikipedia.org/wiki/Bootstrapping_(compilers))
to produce a container-native, minimal, and reproducible toolchain. Curious readers are encouraged to look at [Appendix: Why We Created StageX instead of using X](https://whitepaper.turnkey.com/foundations/#appendix-why-we-created-stagex-instead-of-using-x)
for details about available Linux distributions and why they ultimately didn't meet our bar.
### [](https://whitepaper.turnkey.com/foundations/#how-stagex-works "Copy link to this section")
How StageX works
StageX distributes packages as [OCI](https://opencontainers.org/)
containers. This allows hosting them just like any other images, on DockerHub[12](https://whitepaper.turnkey.com/foundations/#12)
, and allows for hash-locked pulls out of the gate. OCI is the only well-documented packaging standard with multiple competing toolchain implementations and multiple-signature support. It's the most widely used and understood way to deploy software today.
Because StageX packages are OCI images, using StageX's reproducible Rust is a simple `FROM` away:
FROM stagex/rust@sha256:b7c834268a81bfcc473246995c55b47fe18414cc553e3293b6294fde4e579163
This forces a download of an exact image, pinned to a specific digest (`b7c83426…`). You can see existing signatures for this image at [stagex:signatures/stagex/rust@sha256=b7c83426…](https://codeberg.org/stagex/stagex/src/branch/main/signatures/stagex/rust@sha256=b7c834268a81bfcc473246995c55b47fe18414cc553e3293b6294fde4e579163)
, or reproduce it yourself from source with `make rust`. As a result you can trust that the Rust image you're pulling comes from [this `Containerfile`](https://codeberg.org/stagex/stagex/src/branch/main/packages/core/rust/Containerfile)
and contains nothing malicious, even if you pull it from an untrusted source. If the downloaded image is corrupted, its sha256 digest won't match the pinned digest, and the build will error out.
### [](https://whitepaper.turnkey.com/foundations/#how-turnkey-uses-stagex "Copy link to this section")
How Turnkey uses StageX
Secure applications at Turnkey are all built this way. As a result anyone can reproduce builds independently, and validate remote attestations meaningfully when we deploy critical software into Nitro enclaves.
As a concrete example, take a look at [`linux-nitro`](https://codeberg.org/stagex/stagex/src/branch/main/packages/linux-nitro/Containerfile)
: this is the kernel we use inside of QuorumOS. It is maintained in StageX because Amazon itself does not provide a properly signed, reproducible, source-bootstrapped or even recent kernel for Nitro enclaves. Thanks to StageX, the kernel we use in Turnkey enclaves is built reproducibly and signed by multiple parties, all the way down to a few hundred bytes of assembly (see [`stage0`](https://codeberg.org/stagex/stagex/src/branch/main/packages/bootstrap/stage0/Containerfile)
).
We use StageX to power EIF builds and all other Turnkey builds. As a result EIF files are byte-for-byte identical, every time, regardless of who builds them, regardless of which computer in the world kicks off “the build”. And that's no small feat: the AWS developer SDK can't do this for you!
Because we can verify EIF builds, we're able to use remote attestations meaningfully: `PCR0` values can be verified and mapped to source code, which can be independently reviewed and audited by multiple (human) parties.
### [](https://whitepaper.turnkey.com/foundations/#the-invisible-hard-problems-resolved-by-stagex "Copy link to this section")
The invisible hard problems resolved by StageX
The fact that StageX works is a miracle that could not have been possible without relying on other people's work. Here we highlight a few of the big hurdles.
**Bootstrapping GCC**
This was by far the thorniest issue to resolve. Many individuals and projects have contributed to solving it over the years. [Carl Dong](https://github.com/dongcarl)
gave a [talk about bootstrapping](https://diyhpl.us/wiki/transcripts/breaking-bitcoin/2019/bitcoin-build-system/)
which rallied people to the effort started by the Bitcoin community, Guix recently [proved it could bootstrap a modern Linux distribution](https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/)
for which the [Stage0](https://github.com/oriansj/stage0)
and the [Gnu Mes](https://www.gnu.org/software/mes/)
teams provided key ingredients, and the [bootstrappable builds](https://bootstrappable.org/)
and [live-bootstrap](https://github.com/fosslinux/live-bootstrap)
projects glued it all together.
StageX follows the footsteps of Guix and uses the same full-source bootstrap process, starting from [`hex0`](https://github.com/oriansj/bootstrap-seeds/blob/master/POSIX/x86/hex0_x86.hex0)
, a 190 bytes seed of well-understood assembly code. This seed is used to compile [`kaem`](https://github.com/oriansj/kaem/tree/master)
, “the world's worst build tool”, in [`stage0`](https://codeberg.org/stagex/stagex/src/branch/main/packages/bootstrap/stage0/Containerfile)
. Stage [1](https://codeberg.org/stagex/stagex/src/branch/main/packages/bootstrap/stage1/Containerfile)
, [2](https://codeberg.org/stagex/stagex/src/branch/main/packages/bootstrap/stage2/Containerfile)
, and [3](https://codeberg.org/stagex/stagex/src/branch/main/packages/bootstrap/stage3/Containerfile)
build on this just enough to build [`gcc`](https://codeberg.org/stagex/stagex/src/branch/main/packages/core/gcc/Containerfile)
, which is used to build many other compilers and tools.
**GCC to Golang**
It is worth acknowledging the excellent work done by Google. They have [documented this path well](https://go.dev/blog/rebuild)
and provide all the tooling to do it. You only need 3 versions of Golang to get all the way back to GCC. See [stagex:packages/core/go](https://codeberg.org/stagex/stagex/src/branch/main/packages/core/go/Containerfile)
.
**Bootstrapping Rust**
A given version of Rust can only ever be built with the immediately previous version. If you go down this chicken-and-egg problem far enough and you realize that in most distros the chicken comes first: most include a non-reproducible “seed” Rust binary presumably compiled by some member of the Rust team, use that to build the next version, and carry on from there. Even some of the distros that say their Rust builds are reproducible have a pretty major asterisk.
Thankfully [John Hodges](https://github.com/thepowersgang)
created [mrustc](https://github.com/thepowersgang/mrustc)
, which implements a minimal semi-modern Rust 1.54 compiler in C++. It is missing a lot of critical features but it **does** support enough features to compile the official Rust 1.54 sources, which can compile Rust 1.55 and so on. This is the path Guix and Nix both went down, and StageX is following their lead, except using `musl`. A [quick patch](https://github.com/thepowersgang/mrustc/pull/325)
did the trick to make `mrustc` work with `musl`. See this in action for yourself at [stagex:packages/core/rust](https://codeberg.org/stagex/stagex/src/branch/main/packages/core/rust)
[13](https://whitepaper.turnkey.com/foundations/#13)
.
**Reproducible NodeJS (!)**
NodeJS was never designed with reproducible builds in mind. Through extensive [discussion with the maintainers](https://github.com/nodejs/build/issues/3043)
and a lot of [effort](https://codeberg.org/stagex/stagex/pulls/95/files)
, NodeJS is now packaged in StageX: [packages/core/nodejs](https://codeberg.org/stagex/stagex/src/branch/main/packages/core/nodejs)
. This is (to our knowledge) an industry first.
[](https://whitepaper.turnkey.com/foundations/#boot-proofs-and-app-proofs "Copy link to this section")
Boot Proofs and App Proofs
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
So far we have introduced Turnkey's foundations in pieces: the base layer is composed of AWS Nitro enclaves, then comes QuorumOS which is the glue between TEEs and applications running within them, and finally, StageX is a foundational build system to solve the reproducible build problem. In this section we explain more concretely why and how anyone can verify Turnkey enclaves do what they claim to be doing.
### [](https://whitepaper.turnkey.com/foundations/#introduction "Copy link to this section")
Introduction
We have designed Turnkey's foundations to provide two main types of proofs:
* **Boot Proofs** are bundles composed of an AWS Nitro Attestations and a QOS Manifests. Together, an AWS Nitro Attestation and a QOS Manifest prove that a given application binary has been provisioned inside of an enclave.
* **App Proofs** are arbitrary messages signed by an enclave's ephemeral key pair. Recall that a new ephemeral key pair is generated when enclaves boot. As a result, ephemeral keys are globally unique and can be used as enclave identifiers (unlike Quorum Keys, which are stable across enclaves running the same application). The messages in App Proofs are application-specific and can be used to prove arbitrary parts of an enclave's functionality.
These proofs are linked together by the enclave ephemeral key. Boot Proofs attest to boot configuration of an enclave, which includes the QOS Manifest digest and the enclave ephemeral public key. This is where App Proofs come from: application-specific messages are signed by this same ephemeral key. To walk the chain back up, one must:
* Obtain an App Proof and verify its validity against the ephemeral public key
* Obtain a Boot Proof for this particular ephemeral public key. Because the ephemeral public key is globally unique, there is a single AWS Attestation document and QOS Manifest associated with an ephemeral public key.
The relationship between Boot Proof and App Proof is one-to-many: there are many App Proofs for a single Boot Proof (an enclave may sign many application-specific messages during its lifespan, once booted), but there is a single Boot Proof for every App Proof (an enclave ephemeral public key is globally unique, and specific to an enclave which is provisioned with a particular application binary)
### [](https://whitepaper.turnkey.com/foundations/#proven-by-boot-proofs "Copy link to this section")
Proven by Boot Proofs
Recall that a Boot Proof is an AWS attestation document bundled with a particular QOS Manifest. A Boot Proof proves the following facts:
* **The enclave is a legitimate Nitro enclave and the attestation document is valid**. This can be verified through the certificate bundle contained in the AWS attestation document and verifying the signature against Amazon's root certificate, documented [here](https://docs.aws.amazon.com/enclaves/latest/user/verify-root.html#validation-process)
.
* **The enclave runs a QuorumOS EIF**. This can be verified through the PCR measurements in the AWS attestation document. More precisely:
* Anyone can take the Boot Proof's Attestation document and look at its `PCR0` measurement.
* Anyone can download QuorumOS's [source code](https://github.com/tkhq/qos)
and build the EIF locally. Because this build process uses [StageX](https://codeberg.org/stagex/stagex)
it is reproducible. It will produce the same EIF, and yield the same digest when hashed.
* This is of crucial importance: now anyone can inspect QuorumOS's source code and know that what they read is what runs inside of the enclave.
* As a consequence of this **verifiability all the way to the source code**, anyone can verify that QuorumOS does its job, the full job, and _nothing but its job_, by reading its source code independently. The crucial parts include:
* Generating a brand new key pair when enclaves boot (the ephemeral key pair), done [here](https://github.com/tkhq/qos/blob/d12677c26d1bcb706691561a2472361588623197/src/qos_core/src/protocol/services/boot.rs#L446-L450)
.
* Passing the ephemeral public key in the `public_key` field of the attestation document, done [here](https://github.com/tkhq/qos/blob/b960756a67b4b6f0dac16f5388dc9555bd8978a2/src/qos_core/src/protocol/services/attestation.rs#L31)
.
* Passing the QOS Manifest digest in the `user_data` field of the attestation document, done [here](https://github.com/tkhq/qos/blob/b960756a67b4b6f0dac16f5388dc9555bd8978a2/src/qos_core/src/protocol/services/attestation.rs#L29)
.
* **The enclave runs a particular application**. This can be verified in two steps:
* First, by hashing the QOS Manifest (in the Boot Proof), anyone can verify that it matches the digest in the `user_data` field of the AWS attestation document. This proves that the Boot Proof's QOS Manifest is indeed the correct manifest, and by extension, proves that the data within it describe the application running inside of QuorumOS. The QOS Manifest includes the application binary hash, the Quorum public key, the Quorum Set member public keys, [and more](https://github.com/tkhq/qos/blob/39e8b4194bdc9b68b8ae962d196951d60708a461/src/qos_core/src/protocol/services/boot.rs#L310-L323)
.
* Second, anyone can verify that the source code for a particular application produces the correct digest (inside of the QOS manifest) by reproducing it. This is made possible (once again) by [StageX](https://codeberg.org/stagex/stagex)
, which Turnkey uses for all enclave application builds.
Because ephemeral key pairs are unique to each enclave we know that App Proofs come from a particular enclave, running a particular application, itself running within a particular version of QuorumOS. Verifiability down to the source code, all the way down.
### [](https://whitepaper.turnkey.com/foundations/#proven-by-app-proofs "Copy link to this section")
Proven by App Proofs
App Proofs are application-specific and prove different classes of facts depending on which enclave application produces them. Recall that an App Proof is always associated with a Boot Proof, which contains the digest of the application binary (in the QOS Manifest) as well as the digest of the QuorumOS EIF (in the AWS Attestation document).
Because applications and QuorumOS use StageX and are built reproducibly, anyone can thus verify the source code for themselves and see where App Proofs originate from exactly. App Proofs are messages signed by an enclave's ephemeral key pair, thus extremely simple and lightweight to verify. We'll see in [Architecture](https://whitepaper.turnkey.com/architecture/#utility-of-app-proofs)
the different types of enclave applications Turnkey has built so far, and where App Proofs can provide the most value.
[](https://whitepaper.turnkey.com/foundations/#conclusion "Copy link to this section")
Conclusion
---------------------------------------------------------------------------------------------------------------------------------------------------
We've explained how Turnkey's foundations enable running applications securely and verifiably inside of TEEs. They're composed of three main components: TEEs, QuorumOS, and StageX.
* Turnkey uses an application-agnostic QuorumOS EIF to boot enclaves.
* Remote attestations provide proof that an enclave is provisioned with the correct EIF, by signing PCR measurements.
* QuorumOS is the base operating system running in TEEs and the glue between provider-specific hardware and provider-agnostic applications, written in Rust. We've designed QuorumOS to run secure applications at scale, in modern cloud environments, without any single points of failure (“Quorum” based approach)
* QuorumOS works with TEE attestations to prove that a given instance of QuorumOS is provisioned with a specific application. This is done with QOS Manifests, which references the application digest and configuration. This lets anyone verify an enclave booted in the correct datacenter, with the right hardware, provisioned with the correct QuorumOS base image, and running the correct application binary.
* StageX provides reproducible builds to ensure binary artifacts and their digests can be reproduced, agreed upon, and certified by multiple parties meaningfully. StageX guarantees a 1-to-1, immutable relationship between human-readable source code and the resulting machine-executable artifacts.
We've also introduced the concepts of Boot Proofs and App Proofs, and explained in detail how anyone can verify what runs in enclaves, all the way down to the exact source code which produced the QuorumOS EIF and the application binary running within it.
Now that we've seen how applications can be run securely and verifiably, it's time to move up a layer and explain how we've architected and built best-in-class key management APIs using our verifiable foundations. See you in [Turnkey's Architecture](https://whitepaper.turnkey.com/architecture/)
.
[](https://whitepaper.turnkey.com/foundations/#appendix-why-we-created-stagex-instead-of-using-x "Copy link to this section")
Appendix: Why We Created StageX instead of using X
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
To achieve reliable reproducible builds we took a hard look at the available options around us to avoid building anything from scratch ourselves if we did not have to. This is a list of what we evaluated and why we ultimately rejected those options:
* [Alpine](https://alpinelinux.org/)
is the most popular distro in container-land and has made great strides in proving a minimal [musl](https://www.musl-libc.org/how.html)
\-based distro with reasonable security defaults. It is suitable for most use cases, however in the interest of developer productivity and low friction for contributors, none of it is signed[14](https://whitepaper.turnkey.com/foundations/#14)
.
* [Chainguard](https://www.chainguard.dev/)
sounds great on paper (container-native!), but on closer inspection they built their framework on top of Alpine which is neither signed nor reproducible and Chainguard image authors do not sign commits or packages with their own keys. They double down on centralized signing with [cosign](https://github.com/sigstore/cosign)
and the [SLSA framework](https://slsa.dev/)
to prove their centrally built images were built by a known trusted CI system. This is however only as good as those central signing keys and the people who manage them which we have no way to trust independently.
* [Fedora](https://fedoraproject.org/)
(and [RedHat](https://www.redhat.com/)
\-based distros) sign packages with a global signing key, similar to Chainguard, which is not great. They otherwise suffer from similar one-size-fits-all bloat problems as Debian with a different coat of paint. Their reliance on centralized builds has been used as justification for them to not pursue reproducibility, which makes them a non-starter for security-focused use cases.
* [Arch Linux](https://archlinux.org/)
has very fast updates as a rolling release distro. Package definitions are signed, and often reproducible, but they change from one minute to the next. Reproducible builds require pinning and archiving sets of dependencies that work well together for your own projects.
* [Debian](https://www.debian.org/)
(and derivatives like [Ubuntu](https://ubuntu.com/)
) is one of most popular options for servers, and also sign most packages. However, these distros are [glibc](https://www.gnu.org/software/libc/)
\-based with a focus on compatibility and desktop use-cases. As a result they have a huge number of dependencies, partial code freezes for long periods of time between releases, and stale packages as various compatibility goals block updates.
* [Nix](https://nixos.org/)
is almost entirely reproducible by design and allows for lean and minimal output artifacts. It is also a big leap forward in having good separation of concerns between privileged immutable and unprivileged mutable spaces, however they don’t do any maintainer-level signing in order to ensure any hobbyist can contribute with low friction.
* [Guix](https://guix.gnu.org/)
is reproducible by design, borrowing a lot from Nix[15](https://whitepaper.turnkey.com/foundations/#15)
. It also does maintainer-level signing like Debian. It comes the closest to what we need overall (and this is what [Bitcoin settled on](https://github.com/bitcoin/bitcoin/tree/master/contrib/guix)
!), but lacks multi-sig package contributions as well as minimalism. The dependency tree is large because of glibc.
Summarizing the above in a table:
| | | | | | |
| --- | --- | --- | --- | --- | --- |
| Distro | OCI support | Signatures | Libc | Reproducible | Bootstrapped |
| Alpine | Published | None | musl | No | No |
| Chainguard | Native | 1 Bot | musl | No | No |
| Fedora | Published | 1 Bot | glibc | No | No |
| Arch | Published | 1+ Human | glibc | Partial ([90%](https://reproducible.archlinux.org/)
) | No |
| Debian | Published | 1 Human | glibc | Partial (96%) | No |
| Nix | Exported | 1 Bot | glibc | Partial (95%) | Partially |
| Guix | Exported | 1+ Human | glibc | Partial (90%) | yes |
| **StageX** | **Native** | **2+ Humans** | **musl** | **Yes (100%)** | **yes** |
This should speak for itself: these candidates didn't quite meet our bar. We wanted the musl-based container-ideal minimalism of Alpine, the obsessive reproducibility and full-source supply chain goals of Guix, and a step beyond the single-party signed packages of Debian or Arch.
[](https://whitepaper.turnkey.com/foundations/#appendix-airgapped-workflows-for-quorum-set-members "Copy link to this section")
Appendix: airgapped workflows for Quorum Set members
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
This appendix discusses the airgap process we've designed to provision enclaves with a particular application. Recall from the [Enclave provisioning](https://whitepaper.turnkey.com/foundations/#enclave-provisioning)
section that we've split the provisioning process into two parts, involving two different Quorum Sets.
* Manifest approval: in this phase, enough Manifest Set members must approve the QOS Manifest with their private key.
* Share posting: in this phase, enough Share Set members must post their share of the Quorum Key to the enclave. We have seen in [Secure share posting with remote attestations and Ephemeral Keys](https://whitepaper.turnkey.com/foundations/#secure-share-posting-with-remote-attestations-and-ephemeral-keys)
that Share Set members use their private keys to decrypt and re-encrypt Quorum Key shares.
In both cases we are performing critical operations:
* Manifest approval requires the use of a private key to sign
* Share posting requires the use of a private key to decrypt and re-encrypt Quorum Key shares. A decrypted share lives in memory for a brief period of time.
For this reason we've designed an airgapped workflow based on [AirgapOS](https://git.distrust.co/public/airgap)
, an open-source OS built for these critical use-cases. Operator key material is always held on secure hardware and connected to an offline machine. This drastically reduces the odds of compromise.
Both workflows have the same goal: ensure that no private key material is ever exposed to an online environment. We accomplish this by dedicating a device (the “offline” device) to the critical operations (signing, decryption, encryption). Inputs to these critical operations are transported from a standard machine (the “online” device) to the offline device in a one-way fashion (see [data diode](https://en.wikipedia.org/wiki/Unidirectional_network)
).
Offline and online devices can be separate physical laptops or machines, but can also be implemented as separate Qubes (in [QubesOS](https://www.qubes-os.org/)
). For data diodes, we use SD cards physically written to at the source, and transported to the destination device. If using Qubes, [qvm-copy](https://www.qubes-os.org/doc/how-to-copy-and-move-files/)
can be used to safely transport files from one Qube to another.
### [](https://whitepaper.turnkey.com/foundations/#manifest-approval "Copy link to this section")
Manifest approval
Airgapped manifest approval workflow
This workflow is performed by Manifest Set members.
* \[online device\] Each member downloads the to-be-approved Manifest and software to parse and approve it. This software is built with StageX, reproducibly.
* \[online device\] Each member reproduces the application binary to ensure its digest matches the application digest in the QOS Manifest.
* \[one-way transport\] QOS Manifest and software moves to offline device
* \[offline device\] Operator connects their hardware key
* \[offline device\] The verification software parses the QOS manifest and prompts the operator to manually verify its attributes such as: command line argument to run the application, AWS region, and so on.
* \[offline device\] Once the operator finishes manual verification, they sign the manifest using their connected hardware key. This produces the manifest approval.
* \[one-way transport\] Manifest approval moves to the online device
* \[online device\] Manifest approval is persisted in a git repository. Once enough manifest approvals are persisted, the manifest is considered approved.
### [](https://whitepaper.turnkey.com/foundations/#quorum-key-share-posting "Copy link to this section")
Quorum Key share posting
Airgapped share posting workflow
This workflow is performed by Share Set members.
* \[online device\] Each member downloads software, the to-be-posted encrypted share which belongs to them, as well as a copy of the QOS manifest and its approvals. This ensures share posting can't take place unless the QOS manifest is approved.
* \[online device\] Each member also obtains a live attestation document coming from the to-be-provisioned enclave. This attestation contains a reference to the QOS manifest as well as an ephemeral public key specific to the enclave.
* \[one-way transport\] QOS Manifest, manifest approvals, encrypted share, and software moves to the offline device.
* \[offline device\] Operator connects their hardware key
* \[offline device\] The verification software parses the QOS manifest, approvals, and attestations and prompts the operator to manually agree to the configuration. This ensures the operator has full context about the application being provisioned.
* \[offline device\] Each member decrypts the encrypted share and re-encrypt it to the enclave's ephemeral public key. This produces an encrypted share (encrypted to a particular enclave).
* \[one-way transport\] The encrypted share moves to the online device
* \[online device\] The encrypted share is then posted to the target enclave. Once enough shares are posted, the enclave is fully provisioned.
[](https://whitepaper.turnkey.com/foundations/#appendix-scaling-provisioning-with-key-forwarding "Copy link to this section")
Appendix: scaling provisioning with Key Forwarding
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Provisioning enclaves as explained above might seem “good enough” but in practice we've found a few problems when running at scale in cloud environments:
* When running on cloud machines there is no guarantee that the underlying hardware won't be taken away or rebooted. Amazon provides a generous [2 minutes warning](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-instance-termination-notices.html)
. A replacement machine cannot be automatically provisioned without human involvement given Quorum Key shares have to be posted by Share Set members. This is a massive problem during off-hours periods (nights and weekends)
* When an application is under heavy load, provisioning extra capacity requires human involvement for the same reason (Share Set members need to post Quorum Key shares).
* If there is a bug in an application and a new version needs to be rolled out, a completely new fleet of enclaves needs to be manually provisioned with the updated code. This requires Manifest Set members and Share Set members involvement.
To solve the provisioning problem at scale we've designed Key Forwarding\[^19\] as a solution. Key forwarding allows booting new QOS nodes without manually submitting Quorum Key shares. Instead, a new QOS enclave receives a Quorum Key from an already-provisioned enclave after proving it runs the same application (new QOS nodes must be in the same namespace as the one they are requesting the Quorum Key from).
Sequence diagram for Key Forwarding
Key forwarding involves 3 parties: an already-provisioned enclave (“Old”) with access to a live Quorum Key, a freshly booted enclave (“New”), and a Client. The Client's only role is to facilitate communication between old and new enclaves: indeed enclaves do not know about each other and have no way to issue outbound requests. Here's how a Key Forward provisioning flow works:
* First, the client makes a Key Forward provisioning request to the New Enclave. This is similar to a normal provisioning request: it contains the Manifest Envelope and the associated application binary. Instead of being in `WaitingForQuorumShards` state, our New enclave transitions to `WaitingForForwardedKey`. It creates a brand new Ephemeral Key, similar to the standard provisioning flow.
* An attestation document containing the New enclave's Ephemeral Key (public key) is returned to the Client.
* The Client connects to the Old enclave and makes an Export Key request which contains the New enclave's attestation. Many cryptographic and consistency checks happen, among which: validity of the attestation, old and new Quorum Set and Share Set must be identical, and PCR values must be identical. For the full specification see [`KEY_FORWARDING.MD`](https://github.com/tkhq/qos/blob/main/src/qos_core/KEY_FORWARDING.MD#routine)
. Once the verification is complete the Old enclave is “convinced” that the New enclave is a peer, running the same secure application, maintained by the same groups of operators. It encrypts its Quorum Key to the New Enclave's Ephemeral Key.
* The encrypted Quorum Key is returned to the Client
* The Client makes an Inject Key request with the encrypted Quorum Key. The New enclave decrypts it, verifies the public key matches with the expected value in the manifest, and signals success. The New enclave has been provisioned successfully.
As a result of Key Forwarding, enclave fleets can be scaled up or down without human involvement. They can also recover from cloud terminations: as long as at least one healthy enclave remains in the fleet, more enclaves can be spun up without manual share posting. Key forwarding has been an invaluable solution to scaling Turnkey.
* * *
1
For a complete overview, check out [this video](https://youtu.be/jAaqfeyvvSE?t=1792)
.[↩](https://whitepaper.turnkey.com/foundations/#ref:1)
2
A single Nitro Host can start multiple enclaves if the underlying hardware has enough CPU and RAM to accommodate them.[↩](https://whitepaper.turnkey.com/foundations/#ref:2)
3
See [https://github.com/aws/aws-nitro-enclaves-image-format](https://github.com/aws/aws-nitro-enclaves-image-format)
for the format definition.[↩](https://whitepaper.turnkey.com/foundations/#ref:3)
4
The Nitro Secure Module is a driver loaded inside of the base OS, connecting to the physically separate Nitro Card via a PCIe interface. Loading this driver is done in the StageX Containerfile, [here](https://codeberg.org/stagex/stagex/src/commit/17ca3201166be1cd9c0853db20ee2b50bdd2207b/packages/user/linux-nitro/Containerfile#L51)
. For more information about the AWS Nitro system, check out their [whitepaper](https://docs.aws.amazon.com/pdfs/whitepapers/latest/security-design-of-aws-nitro-system/security-design-of-aws-nitro-system.pdf)
.[↩](https://whitepaper.turnkey.com/foundations/#ref:4)
5
In case you are wondering “is using RAM safe?”: RAM is never shared between enclave requests, and we write applications in Rust to guarantee memory safety within a single request.[↩](https://whitepaper.turnkey.com/foundations/#ref:5)
6
The set of fields is documented by AWS [here](https://docs.aws.amazon.com/enclaves/latest/user/verify-root.html#doc-def)
for the general structure and [here](https://docs.aws.amazon.com/enclaves/latest/user/set-up-attestation.html)
for PCRs specifically[↩](https://whitepaper.turnkey.com/foundations/#ref:6)
7
By “provision” here, we mean provisioning an enclave with its final application binary and Quorum Key (stable secret). See later sections for details.[↩](https://whitepaper.turnkey.com/foundations/#ref:7)
8
The set of possible states for an enclave is listed [here](https://github.com/tkhq/qos/blob/0f6728bf8f70322cf7feae4cedf4f3aad228d0ec/src/qos_core/src/protocol/state.rs#L25)
.[↩](https://whitepaper.turnkey.com/foundations/#ref:8)
9
In case you're wondering: the genesis ceremony also happens inside of an enclave. See [genesis.rs](https://github.com/tkhq/qos/blob/main/src/qos_core/src/protocol/services/genesis.rs)
.[↩](https://whitepaper.turnkey.com/foundations/#ref:9)
10
Confusingly, this initial state is called WaitingForBootInstruction in the code. The enclave is booted, waiting for _provisioning_ instructions. We might rename this at some point in the future.[↩](https://whitepaper.turnkey.com/foundations/#ref:10)
11
The place where this is declared is [here](https://github.com/tkhq/qos/blob/0f6728bf8f70322cf7feae4cedf4f3aad228d0ec/src/qos_core/src/protocol/state.rs#L125-L131)
. Each ProtocolRoute takes “Ok phase” and “Err phase” as the 2nd and 3rd argument.[↩](https://whitepaper.turnkey.com/foundations/#ref:11)
12
We're thankful for the Docker team's help: they advised on finding a path to fully reproducible OCI images and offered unlimited free bandwidth to upload and host StageX images.[↩](https://whitepaper.turnkey.com/foundations/#ref:12)
13
Because of this 1 step version-to-version process in rust, when reproducing StageX from stage0 to stage3, building up all the rust versions up to 1.81 is often the longest part of the build process.[↩](https://whitepaper.turnkey.com/foundations/#ref:13)
14
By “none of it is signed” we mean that published artifacts aren't signed by the maintainer or contributor who produced them. In the absence of code signing the publisher is potentially misrepresented because it is easy to impersonate.[↩](https://whitepaper.turnkey.com/foundations/#ref:14)
15
Turns out Guix is not 100% reproducible either and is in a similar position to Nix. Packages that include binary blobs like the firmware blobs are just copied directly. Hitting 100% reproducibility may take ages, particularly with no forcing function.[↩](https://whitepaper.turnkey.com/foundations/#ref:15)
16
See [this documentation](https://github.com/tkhq/qos/blob/main/src/qos_core/KEY_FORWARDING.MD)
for more information.[↩](https://whitepaper.turnkey.com/foundations/#ref:16)
---