# Table of Contents - [What is Boost? | Boost](#what-is-boost-boost) - [boostd | Boost](#boostd-boost) - [Features | Boost](#features-boost) - [Components | Boost](#components-boost) - [Repository | Boost](#repository-boost) - [SQLite metadata database | Boost](#sqlite-metadata-database-boost) - [GraphQL API | Boost](#graphql-api-boost) - [Local Index Directory | Boost](#local-index-directory-boost) - [Hardware requirements | Boost](#hardware-requirements-boost) - [YugabyteDB | Boost](#yugabytedb-boost) - [booster-http | Boost](#booster-http-boost) - [Monitoring | Boost](#monitoring-boost) - [Installation | Boost](#installation-boost) - [Dependencies | Boost](#dependencies-boost) - [booster-bitswap | Boost](#booster-bitswap-boost) - [Retrieving data from Filecoin | Boost](#retrieving-data-from-filecoin-boost) - [Initialisation | Boost](#initialisation-boost) - [Deal Filters | Boost](#deal-filters-boost) - [Manual Publish Storage Deal Message | Boost](#manual-publish-storage-deal-message-boost) - [Index types | Boost](#index-types-boost) - [HTTP indexer announcement | Boost](#http-indexer-announcement-boost) - [Remote CommP | Boost](#remote-commp-boost) - [YugabyteDB | Boost](#yugabytedb-boost) - [Storing data on Filecoin | Boost](#storing-data-on-filecoin-boost) - [boostd-data | Boost](#boostd-data-boost) - [New Boost Setup | Boost](#new-boost-setup-boost) - [HTTP Transfer limit | Boost](#http-transfer-limit-boost) - [Setting up a monitoring stack for Boost | Boost](#setting-up-a-monitoring-stack-for-boost-boost) - [Backup and Restore | Boost](#backup-and-restore-boost) - [Need help? | Boost](#need-help-boost) - [Experimental Features | Boost](#experimental-features-boost) - [What is Boost? | v1.x (Deprecated) | Boost](#what-is-boost-v1-x-deprecated-boost) - [HTTP retrieval | Boost](#http-retrieval-boost) - [FAQ | Boost](#faq-boost) - [Legacy Deal configuration | Boost](#legacy-deal-configuration-boost) - [FVM Contract Deals | Boost](#fvm-contract-deals-boost) - [Tutorials | Boost](#tutorials-boost) - [Features | v1.x (Deprecated) | Boost](#features-v1-x-deprecated-boost) - [Data Segment Indexing | Boost](#data-segment-indexing-boost) - [Bitswap retrieval | Boost](#bitswap-retrieval-boost) - [Hardware requirements | v1.x (Deprecated) | Boost](#hardware-requirements-v1-x-deprecated-boost) - [Troubleshooting | Boost](#troubleshooting-boost) - [How to store files with Boost on Filecoin | Boost](#how-to-store-files-with-boost-on-filecoin-boost) - [UI Settings | Boost](#ui-settings-boost) - [How to re-index unsealed pieces that are flagged by LID in Boost v2 | Boost](#how-to-re-index-unsealed-pieces-that-are-flagged-by-lid-in-boost-v2-boost) - [Database | v1.x (Deprecated) | Boost](#database-v1-x-deprecated-boost) - [How to upgrade from v2.0.0 to v2.1.0 | Boost](#how-to-upgrade-from-v2-0-0-to-v2-1-0-boost) - [libp2p Protocols | Boost](#libp2p-protocols-boost) - [Direct Deals | Boost](#direct-deals-boost) - [How to upgrade from v2.1.x to v2.2.0 | Boost](#how-to-upgrade-from-v2-1-x-to-v2-2-0-boost) - [Tutorials | v1.x (Deprecated) | Boost](#tutorials-v1-x-deprecated-boost) - [Roll back to Lotus markets service process | v1.x (Deprecated) | Boost](#roll-back-to-lotus-markets-service-process-v1-x-deprecated-boost) - [Migrate from Lotus to Boost | v1.x (Deprecated) | Boost](#migrate-from-lotus-to-boost-v1-x-deprecated-boost) - [Backup and Restore | v1.x (Deprecated) | Boost](#backup-and-restore-v1-x-deprecated-boost) - [Architecture | v1.x (Deprecated) | Boost](#architecture-v1-x-deprecated-boost) - [How to onboard data using DDO deals | Boost](#how-to-onboard-data-using-ddo-deals-boost) - [As a client | v1.x (Deprecated) | Boost](#as-a-client-v1-x-deprecated-boost) - [Start and stop Boost processes | Boost](#start-and-stop-boost-processes-boost) - [Need help? | v1.x (Deprecated) | Boost](#need-help-v1-x-deprecated-boost) - [Configuration | Boost](#configuration-boost) - [FVM Contract Deals | v1.x (Deprecated) | Boost](#fvm-contract-deals-v1-x-deprecated-boost) - [How to store files with Boost on Filecoin | v1.x (Deprecated) | Boost](#how-to-store-files-with-boost-on-filecoin-v1-x-deprecated-boost) - [UI Settings | v1.x (Deprecated) | Boost](#ui-settings-v1-x-deprecated-boost) - [Repository | v1.x (Deprecated) | Boost](#repository-v1-x-deprecated-boost) - [libp2p Protocols | v1.x (Deprecated) | Boost](#libp2p-protocols-v1-x-deprecated-boost) - [Migrate a Lotus markets service process to Boost | v1.x (Deprecated) | Boost](#migrate-a-lotus-markets-service-process-to-boost-v1-x-deprecated-boost) - [DAG store | v1.x (Deprecated) | Boost](#dag-store-v1-x-deprecated-boost) - [Getting started | v1.x (Deprecated) | Boost](#getting-started-v1-x-deprecated-boost) - [GraphQL API | v1.x (Deprecated) | Boost](#graphql-api-v1-x-deprecated-boost) - [Advanced Configuration of booster-http | Boost](#advanced-configuration-of-booster-http-boost) - [HTTP indexer announcement | v1.x (Deprecated) | Boost](#http-indexer-announcement-v1-x-deprecated-boost) - [Setting up a monitoring stack for Boost | v1.x (Deprecated) | Boost](#setting-up-a-monitoring-stack-for-boost-v1-x-deprecated-boost) - [Troubleshooting | v1.x (Deprecated) | Boost](#troubleshooting-v1-x-deprecated-boost) - [JSON-RPC API | Boost](#json-rpc-api-boost) - [Migrate from Lotus to Boost | Boost](#migrate-from-lotus-to-boost-boost) - [Using filters for storage and retrieval deals | Boost](#using-filters-for-storage-and-retrieval-deals-boost) - [Architecture | v1.x (Deprecated) | Boost](#architecture-v1-x-deprecated-boost) - [Experimental Features | v1.x (Deprecated) | Boost](#experimental-features-v1-x-deprecated-boost) - [Remote CommP | v1.x (Deprecated) | Boost](#remote-commp-v1-x-deprecated-boost) - [FAQ | v1.x (Deprecated) | Boost](#faq-v1-x-deprecated-boost) - [As a storage provider | v1.x (Deprecated) | Boost](#as-a-storage-provider-v1-x-deprecated-boost) - [Serving files with booster-http | v1.x (Deprecated) | Boost](#serving-files-with-booster-http-v1-x-deprecated-boost) - [HTTP Transfer limit | v1.x (Deprecated) | Boost](#http-transfer-limit-v1-x-deprecated-boost) - [Requirements | v1.x (Deprecated) | Boost](#requirements-v1-x-deprecated-boost) - [Local Index Directory | v1.x (Deprecated) | Boost](#local-index-directory-v1-x-deprecated-boost) - [How to upgrade from Boost v1 to Boost v2 | Boost](#how-to-upgrade-from-boost-v1-to-boost-v2-boost) - [Initialisation | v1.x (Deprecated) | Boost](#initialisation-v1-x-deprecated-boost) - [Deal Filters | v1.x (Deprecated) | Boost](#deal-filters-v1-x-deprecated-boost) - [Bitswap Retrieval | v1.x (Deprecated) | Boost](#bitswap-retrieval-v1-x-deprecated-boost) - [HTTP Retrieval | v1.x (Deprecated) | Boost](#http-retrieval-v1-x-deprecated-boost) - [Using filters for storage and retrieval deals | v1.x (Deprecated) | Boost](#using-filters-for-storage-and-retrieval-deals-v1-x-deprecated-boost) - [Migrate a monolith lotus-miner to Boost | v1.x (Deprecated) | Boost](#migrate-a-monolith-lotus-miner-to-boost-v1-x-deprecated-boost) - [Legacy Deal configuration | v1.x (Deprecated) | Boost](#legacy-deal-configuration-v1-x-deprecated-boost) - [Configuration | v1.x (Deprecated) | Boost](#configuration-v1-x-deprecated-boost) - [JSON-RPC API | v1.x (Deprecated) | Boost](#json-rpc-api-v1-x-deprecated-boost) --- # What is Boost? | Boost Boost is a tool for Storage Providers to manage data onboarding and retrieval on the Filecoin network. Boost exposes libp2p interfaces for making storage and retrieval deals, a web interface for managing storage deals, and a GraphQL interface for accessing and updating real-time deal information. ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252Fx5BzRXgnUzh7kuYrIuix%252FBoost%2520-%2520storage%2520deals.png%3Falt%3Dmedia%26token%3Dcf8061b4-e252-4d7d-b2cc-d0fe7c893ee7&width=768&dpr=3&quality=100&sign=b1f40425&sv=2) Web UI - Storage Deals screen ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252F8ek9kD4J9eRBbeM1y9bw%252FBoost%2520-%2520storage%2520space.png%3Falt%3Dmedia%26token%3D3f617a69-c959-45a5-858e-906cf8430fe2&width=768&dpr=3&quality=100&sign=c84caf03&sv=2) Web UI - Storage Space screen ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252FiypFxTx2JnhDchegOGc4%252FBoost%2520-%2520sealing%2520pipeline.png%3Falt%3Dmedia%26token%3Dbed249c9-dd03-4ec8-a7f5-21e3cb80af0d&width=768&dpr=3&quality=100&sign=93e65a46&sv=2) Web UI - Sealing Pipeline screen [NextFeatureschevron-right](https://boost.filecoin.io/features) Last updated 4 months ago --- # boostd | Boost The `boostd` executable runs as a daemon alongside a Lotus full node and Lotus miner. The daemon exposes a libp2p interface for storage and retrieval deals. It performs on-chain operations by making API calls to the Lotus node. The daemon hands off downloaded data to the Lotus miner for sealing via API calls to the Lotus miner. `boostd` has a web interface for fund management and deal monitoring. The web interface is a React application that consumes a GraphQL interface exposed by the daemon. ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252FNsF3A10TYM4iZLtvj5DB%252FBoost%2520Interfaces.png%3Falt%3Dmedia%26token%3Daff4d843-2ec5-4c06-b5fb-69e42a97fad3&width=768&dpr=3&quality=100&sign=e114309b&sv=2) ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture#storage-deal-flow) Storage Deal Flow The typical flow for a Storage Deal is: 1. The Client puts funds in escrow with the Storage Market Actor on chain. 2. The Client uploads a CAR file to a web server. 3. The Client sends a storage deal proposal to Boost with the URL of the CAR file. 4. Boost checks that the client has enough funds in escrow to pay for storing the file. 5. Boost accepts the storage deal proposal. 6. Boost downloads the CAR file from the web server. 7. Boost publishes the deal on chain. 8. The client checks that the deal was successfully published on chain. ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252FpVoiLESDjm8J4b7ueSqK%252FBoost%2520Flow.png%3Falt%3Dmedia%26token%3Dd2cf5ade-8e33-4336-ba95-47c3fbf40894&width=768&dpr=3&quality=100&sign=329a4748&sv=2) Boost exposes a libp2p interface to listen for storage deal proposals from clients. Boost communicates with the Lotus node over its JSON-RPC API for on-chain operations like checking client funds and publishing the deal. Once the deal has been published, Boost hands off the downloaded file to `lotus-miner` for sealing. [PreviousComponentschevron-left](https://boost.filecoin.io/deployment) [NextRepositorychevron-right](https://boost.filecoin.io/deployment/boost-architecture/architecture) Last updated 1 month ago --- # Features | Boost [hashtag](https://boost.filecoin.io/features#make-storage-deals-with-http-data-transfer) Make storage deals with HTTP data transfer ---------------------------------------------------------------------------------------------------------------------------------------- Boost supports multiple options for data transfer when making storage deals, including HTTP. Clients can host their CAR file on an HTTP server, such as S3, and provide that URL when proposing the storage deal. Once accepted, Boost will automatically fetch the CAR file from the specified URL. See [Storing data on Filecoin](https://boost.filecoin.io/storing-data-on-filecoin) for more details. [hashtag](https://boost.filecoin.io/features#web-ui) Web UI ---------------------------------------------------------------- Boost comes with a web interface that can be used to manage deals, watch disk usage, monitor funds, adjust settings and more. ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252FE52hy623yD9LdsJjvLVP%252FScreen%2520Shot%25202022-03-23%2520at%252010.12.26%2520AM.png%3Falt%3Dmedia%26token%3D43e6fc9d-a9be-480b-9cc0-2c9b41ffc361&width=768&dpr=3&quality=100&sign=6dc5532&sv=2) Boost Web UI [hashtag](https://boost.filecoin.io/features#a-client-for-proposing-deals-that-doesnt-require-a-lotus-node) A client for proposing deals that doesn't require a Lotus node ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Boost comes with a client that can be used to make storage deals, and can be configured to point at a public Filecoin API endpoint. That means clients don't need to run a Filecoin node or sync from chain. See [Storing data on Filecoin](https://boost.filecoin.io/storing-data-on-filecoin) for more details. [PreviousWhat is Boost?chevron-left](https://boost.filecoin.io/) [NextComponentschevron-right](https://boost.filecoin.io/deployment) Last updated 1 month ago * [Make storage deals with HTTP data transfer](https://boost.filecoin.io/features#make-storage-deals-with-http-data-transfer) * [Web UI](https://boost.filecoin.io/features#web-ui) * [A client for proposing deals that doesn't require a Lotus node](https://boost.filecoin.io/features#a-client-for-proposing-deals-that-doesnt-require-a-lotus-node) sun-brightdesktopmoon sun-brightdesktopmoon --- # Components | Boost The **Lotus** stack is responsible for running a Filecoin full chain node, handling sector lifecycle, storing sectors, sector sealing, sector proving, handling wallets, and more. You can read about the Lotus components at [https://docs.filecoin.io/storage-provider/architecture/lotus-components/arrow-up-right](https://docs.filecoin.io/storage-provider/architecture/lotus-components/) Component Description **lotus** Lotus Daemon following the Filecoin chain **lotus-miner** Lotus Miner handling sector storage, jobs scheduling, etc. **lotus-worker** Lotus Worker handling various types of jobs (PC1, UNS, PC2, C1, DC, etc.) The **Boost** stack is responsible for handling storage and retrieval requests, transferring data, indexing data, announcing data to network indexers, and ensuring that clients can retrieve their data once it is stored on the SP's infrastructure. Component Description **boost** Boost client-side executable that can be used to send a deal proposal to a Boost Storage Provider. **boostd** Boost SP daemon handling storage deals processing, Boost UI server, GraphQL server, etc. **booster-http** Serve blocks and files over HTTP **booster-bitswap** Serve blocks and files over Bitswap **boostd-data** Data service handling Boost SP daemon state, and providing interface to the Local Index Directory (hosted on YugabyteDB, or LevelDB) **yugabyted** YugabyteDB backend for Local Index Directory [PreviousFeatureschevron-left](https://boost.filecoin.io/features) [Nextboostdchevron-right](https://boost.filecoin.io/deployment/boost-architecture) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Repository | Boost By default the Boost daemon repository is located at `~/.boost` It contains the following files: * `api` The local multi-address of Boost's libp2p API * `boost.db` The sqlite database with all deal metadata * `boost.logs.db` The sqlite database with the logs for deals * `config.toml` The config file with all Boost's settings * `repo.lock` A lock file created when Boost is running * `storage.json` Deprecated (needed by legacy markets) * `token` The token used when calling Boost's JSON RPC endpoints It has the following directories: * `datastore` Contains metadata about deals for legacy markets * `deal-staging` The directory used by legacy markets for incoming data transfers * `incoming` The directory used by Boost for incoming data transfers * `journal` Contains journal events (used by legacy markets) * `keystore` Contains the secret keys used by libp2p (eg the peer ID) * `kvlog` Used by legacy markets datastore [Previousboostdchevron-left](https://boost.filecoin.io/deployment/boost-architecture) [NextGraphQL APIchevron-right](https://boost.filecoin.io/deployment/boost-architecture/graphql-api) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # SQLite metadata database | Boost Boost stores metadata about deals in a SQLite database in the root directory of the Boost repository. To open the database use a SQLite client: `sqlite3 boost.db` The database tables are * `Deals` metadata about Boost storage deals (eg deal proposal) and their current state (e.g. checkpoint) * `FundsLogs` log of each change in funds reserved for a deal * `FundsTagged` how much FIL is tagged for deal collateral and publish message for a deal * `StorageLogs` log of each change in storage reserved for a deal * `StorageTagged` how much storage is tagged for a deal Boost keeps a separate database just for deal logs, so as to make it easier to manage log data separately from deal metadata. The logs database is named `boost.logs.db` and it has a single table `DealLogs` that stores logs for each deal, indexed by UUID. ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/getting-started-1#migrations) Migrations Boost uses `goose` ([https://pressly.github.io/goose/arrow-up-right](https://pressly.github.io/goose/) ) tool and library for handling sqlite3 migrations. `goose` can be installed following the instructions at [https://pressly.github.io/goose/installation/arrow-up-right](https://pressly.github.io/goose/installation/) Migrations in Boost are stored in the `/db/migrations` directory. Boost handles database migrations on start-up. If a user is running an older version of Boost, migrations up to the latest version are automatically applied on start-up. Developers can use `goose` to inspect and apply migrations using the CLI: Copy ➜ ~ goose Usage: goose [OPTIONS] DRIVER DBSTRING COMMAND ... Commands: up Migrate the DB to the most recent version available up-by-one Migrate the DB up by 1 up-to VERSION Migrate the DB to a specific VERSION down Roll back the version by 1 down-to VERSION Roll back to a specific VERSION redo Re-run the latest migration reset Roll back all migrations status Dump the migration status for the current DB version Print the current version of the database create NAME [sql|go] Creates new migration file with the current timestamp fix Apply sequential ordering to migrations [PreviousJSON-RPC APIchevron-left](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api) [NextLocal Index Directorychevron-right](https://boost.filecoin.io/deployment/local-index-directory) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # GraphQL API | Boost Boost daemon exposes a GraphQL API that is used by the Web UI to query and update information about storage and retrieval deals. The GraphQL API query endpoint is at [http://localhost:8080/graphql/queryarrow-up-right](http://localhost:8080/graphql/query) You can also run your own queries against the GraphQL API using `curl` or a programming language that has a [GraphQL clientarrow-up-right](https://graphql.org/code/) . Boost has a built-in GraphQL explorer at [http://localhost:8080/graphiqlarrow-up-right](http://localhost:8080/graphiql) You can test out queries, or explore the GraphQL API by clicking on the `< Docs` link at the top right of the page: ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252F0TxmNcPzTZyOMjiNzCQg%252FScreen%2520Shot%25202022-04-21%2520at%252010.06.16%2520AM.png%3Falt%3Dmedia%26token%3D861edbcf-884a-4f55-8873-615983aadfe4&width=768&dpr=3&quality=100&sign=61c0f808&sv=2) To run a GraphQL query with `curl`: This 1m video shows how to use these tools to build an run a GraphQL query against Boost: Use the GraphQL explorer to create a query against Boost ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/graphql-api#example-queries) Example Queries 1\. Query failed deals 2\. Cancel a deal where `ab12345c-5678-90de-12f3-45a6b78cd9ef` is the deal ID [PreviousRepositorychevron-left](https://boost.filecoin.io/deployment/boost-architecture/architecture) [NextJSON-RPC APIchevron-right](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api) Last updated 2 years ago sun-brightdesktopmoon Copy curl -X POST -H "Content-Type: application/json" -d '{"query":"query { deals(offset: 5, limit: 10) { deals { ID CreatedAt PieceCid } } }"}' http://localhost:8080/graphql/query | jq Copy curl -X POST \ -H "Content-Type: application/json" \ -d '{"query":"query { deals(limit: 10, query: \"failed to get size of imported\") { deals { ID CreatedAt Message } } }"}' \ http://localhost:8080/graphql/query | jq Copy curl -X POST \ -H "Content-Type: application/json" \ -d '{"query":"mutation { dealCancel(id: \"ab12345c-5678-90de-12f3-45a6b78cd9ef\") }"}' \ http://localhost:8080/graphql/query | jq sun-brightdesktopmoon --- # Local Index Directory | Boost [hashtag](https://boost.filecoin.io/deployment/local-index-directory#background) Background ------------------------------------------------------------------------------------------------ The Local Index Directory (_LID_) manages and stores indices of deal data so that it can be retrieved by a content identifier (_cid_). Currently this task is performed by the _DAG store_ component. The DAG store keeps its indices on disk on a single machine. LID replaces the DAG store and introduces a horizontally scalable backend database for storing the data - YugabyteDB. LID is designed to provide a more intuitive experience for the user, by surfacing problems and providing various repair tools. To summarize, LID is the component which keeps fine-grained metadata about all the deals on Filecoin that a given Storage Provider stores, and without it, the client would only be able to retrieve full pieces, which generally are between 8GiB and 32GiB in size. At the moment there are two implementations of LID: - a simple **LevelDB** implementation, for small SPs who want to keep all information in a single process database. - a scalable **YugabyteDB** implementation, for medium and large size SPs with tens of thousands of deals. When designing LID we considered the needs of various Storage Providers (SPs) and the operational overhead LID would have on their systems. We built a solution for: - small- SPs - holding up to 1PiB, and - mid- and large- size SPs - holding anywhere from 1PiB, up to 100PiB data Depending on underlying block size and data format, index size can vary in size. Typically block sizes are between 16KiB and 1MiB. [hashtag](https://boost.filecoin.io/deployment/local-index-directory#storing-data-on-filecoin) Storing data on Filecoin ---------------------------------------------------------------------------------------------------------------------------- When a client uploads deal data to Boost, LID records the sector that the deal data is stored in and scans the deal data to create an index of all its blocks indexed by block cid. This way clients can later retrieve subsets of the original deal data, without retrieving the full deal data. ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252F7N89CFARLvlm7eYcknM9%252FScreenshot%25202023-05-18%2520at%252013.39.53.png%3Falt%3Dmedia%26token%3D512c15e6-b35d-405d-b83b-d29805cff772&width=768&dpr=3&quality=100&sign=698cd8c5&sv=2) How Boost stores deal data from clients [hashtag](https://boost.filecoin.io/deployment/local-index-directory#retrieving-data) Retrieving data ---------------------------------------------------------------------------------------------------------- When a client makes a request for data by cid, LID: - checks which piece the cid is in, and where in the piece the data is - checks which sector the piece is in, and where in the sector the piece is - reads the data from the sector ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252F5I7KTG0Y0YLeVO1w2vmi%252FScreenshot%25202023-05-18%2520at%252013.45.14.png%3Falt%3Dmedia%26token%3Dc4dfeb4f-f81f-4d89-a148-2055b132a663&width=768&dpr=3&quality=100&sign=4028f2cd&sv=2) How clients retrieve their data from Boost [hashtag](https://boost.filecoin.io/deployment/local-index-directory#use-cases) Use cases ---------------------------------------------------------------------------------------------- The retrieval use cases that the Local Index Directory supports are: #### [hashtag](https://boost.filecoin.io/deployment/local-index-directory#graphsync-retrieval) Graphsync retrieval _Request one root cid with a selector, receive many blocks_ LID is able to: - look up which piece contains the root cid - look up which sector contains the piece - for each block, get the offset into the piece for the block #### [hashtag](https://boost.filecoin.io/deployment/local-index-directory#bitswap-retrieval) Bitswap retrieval _Request one block at a time_ LID is able to: - look up which piece contains the block - get the size of the block (Bitswap asks for the size before getting the block data) - look up which sector contains the piece - get the offset into the piece for the block #### [hashtag](https://boost.filecoin.io/deployment/local-index-directory#http-retrieval) HTTP retrieval _Request a whole piece_ LID is able to look up which sector contains the piece. _Request an individual block_ LID is able to: - look up which piece contains the block - look up which sector contains the piece - get the offset into the piece for the block _Request a file by root cid_ LID is able to: - look up which piece contains the block - look up which sector contains the piece - for each block, get the offset into the piece for the block [PreviousSQLite metadata databasechevron-left](https://boost.filecoin.io/deployment/boost-architecture/getting-started-1) [NextIndex typeschevron-right](https://boost.filecoin.io/deployment/local-index-directory/architecture) Last updated 1 year ago * [Background](https://boost.filecoin.io/deployment/local-index-directory#background) * [Storing data on Filecoin](https://boost.filecoin.io/deployment/local-index-directory#storing-data-on-filecoin) * [Retrieving data](https://boost.filecoin.io/deployment/local-index-directory#retrieving-data) * [Use cases](https://boost.filecoin.io/deployment/local-index-directory#use-cases) sun-brightdesktopmoon sun-brightdesktopmoon --- # Hardware requirements | Boost The hardware requirements for Boost are tied to the Lotus stack in a Filecoin SP deployment. Depending on how much data you need to onboard, and how many deals you need to make with clients, hardware requirements in terms of _CPU_ and _Disk_ will vary. [hashtag](https://boost.filecoin.io/hardware-requirements#general-hardware-requirements) General hardware requirements --------------------------------------------------------------------------------------------------------------------------- ### [hashtag](https://boost.filecoin.io/hardware-requirements#cpu) CPU A miner will need an **8+ core CPU**. We strongly recommend a CPU model with support for _Intel SHA Extensions_: AMD since Zen microarchitecture, or Intel since Ice Lake. Lack of SHA Extensions results in a very significant slow down. The most significant computation that Boost has to do is the _Piece CID calculation (also known as Piece Commitment or CommP)_. When Boost receives data from a client, it calculates the _Merkle root_ out of the hashes of the _Piece_ (padded .car file). The resulting root of the clean binary Merkle tree is the _Piece CID._ ### [hashtag](https://boost.filecoin.io/hardware-requirements#ram) RAM **128 GiB** of RAM is needed at the very least. ### [hashtag](https://boost.filecoin.io/hardware-requirements#disk) Disk Boost stores all data received from clients before Piece CID is calculated and compared against deal parameters received from clients. Next, deals are published on-chain, and Boost waits for a number of epoch confirmations before proceeding to pass data to the Lotus sealing subsystem. This means that depending on the throughput of your operation, you must have disk space for at least a few staged sectors. For small deployments **100 GiB of disk are needed at the very least** if we assume that Boost is to keep three 32 GiB sectors before passing them to the sealing subsystem. We recommend using **NVME disk** for Boost. As Dagstore grows in size, the overall performance might slow down due to slow disk. [Previouslibp2p Protocolschevron-left](https://boost.filecoin.io/deployment/libp2p-protocols) [NextYugabyteDBchevron-right](https://boost.filecoin.io/hardware-requirements/yugabytedb) Last updated 1 month ago * [General hardware requirements](https://boost.filecoin.io/hardware-requirements#general-hardware-requirements) * [CPU](https://boost.filecoin.io/hardware-requirements#cpu) * [RAM](https://boost.filecoin.io/hardware-requirements#ram) * [Disk](https://boost.filecoin.io/hardware-requirements#disk) sun-brightdesktopmoon sun-brightdesktopmoon --- # YugabyteDB | Boost circle-info For detailed instructions, playbooks and hardware recommendations, see the YugabyteDB website - [https://docs.yugabyte.comarrow-up-right](https://docs.yugabyte.com/) YugabyteDB is designed to run on bare-metal machines, virtual machines (VMs), and containers. CPU and RAM You should allocate adequate CPU and RAM. YugabyteDB has adequate defaults for running on a wide range of machines, and has been tested from 2 core to 64 core machines, and up to 200GB RAM. #### [hashtag](https://boost.filecoin.io/hardware-requirements/yugabytedb#minimum-requirement-testing) Minimum requirement (Testing) Copy 2 cores 2GB RAM #### [hashtag](https://boost.filecoin.io/hardware-requirements/yugabytedb#production-requirement) Production requirement Copy 16+ cores 32GB+ RAM Add more CPU (compared to adding more RAM) to improve performance. #### [hashtag](https://boost.filecoin.io/hardware-requirements/yugabytedb#verify-support-for-sse2-and-sse4.2) Verify support for SSE2 and SSE4.2 YugabyteDB requires the SSE2 instruction set support, which was introduced into Intel chips with the Pentium 4 in 2001 and AMD processors in 2003. Most systems produced in the last several years are equipped with SSE2. In addition, YugabyteDB requires SSE4.2. To verify that your system supports SSE2, run the following command: `cat /proc/cpuinfo | grep sse2` To verify that your system supports SSE4.2, run the following command: `cat /proc/cpuinfo | grep sse4.2` #### [hashtag](https://boost.filecoin.io/hardware-requirements/yugabytedb#disks) Disks It is recommend that a minimum of 1TiB or more is allocated for YugabyteDB, depending on the amount of deal data you store and its average block size. circle-exclamation Assuming you've kept unsealed copies of all your data and have consistently indexed deal data, the size of your DAG store directory should be comparable with the requirements for YugabyteDB. [PreviousHardware requirementschevron-left](https://boost.filecoin.io/hardware-requirements) [NextInstallationchevron-right](https://boost.filecoin.io/getting-started) Last updated 2 years ago sun-brightdesktopmoon Copy SSDs (solid state disks) are required. sun-brightdesktopmoon --- # booster-http | Boost Go to the following page for more information on `booster-http`: [HTTP retrievalchevron-right](https://boost.filecoin.io/retrieving-data-from-filecoin/http-retrieval) [PreviousYugabyteDBchevron-left](https://boost.filecoin.io/deployment/yugabytedb) [Nextbooster-bitswapchevron-right](https://boost.filecoin.io/deployment/booster-bitswap) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Monitoring | Boost Boost provides multiple metrics for monitoring the services and APIs. All the metrics are emitted in Prometheus format and can be used to monitor and identify bottlenecks. The metrics endpoint for all Boost services is `/metrics`. The full list of metrics emitted by `boostd`, `booster-http` and `booster-bitswap` are [herearrow-up-right](https://github.com/filecoin-project/boost/blob/main/metrics/metrics.go#L30-L90) . The `boostd-data` (LID) metrics are separate from the other metrics and their list can be found [herearrow-up-right](https://github.com/filecoin-project/boost/blob/main/extern/boostd-data/metrics/metrics.go#L28-L75) . Apart from the prometheus metrics, all Boost services also provides tracing spans. These tracing spans can be useful to debug the bottleneck and low performance sections of the execution. You can enable tracing using `--tracing` flag. circle-info The default URL to export tracing is [http://tempo:14268/api/tracesarrow-up-right](http://tempo:14268/api/traces) , which is not correct outside of a Kubernetes or docker environment. Users must set `--tracing-endpoint` flag to correct IP/Hostname pointing to their `tempo` instance. By default, Boost ships a fully configured monitoring stack. This monitoring stack can be deployed on docker and allows storage provider to get started with monitoring in less than 10 minutes. We highly recommend using this stack to monitor your Boost services unless you are familiar with how to setup monitoring manually and create dashboards. [PreviousManual Publish Storage Deal Messagechevron-left](https://boost.filecoin.io/configuration/manual-publish-storage-deal-message) [NextSetting up a monitoring stack for Boostchevron-right](https://boost.filecoin.io/monitoring/setting-up-a-monitoring-stack-for-boost) Last updated 1 month ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Installation | Boost The Boost source code repository is hosted at [github.com/filecoin-project/boostarrow-up-right](https://github.com/filecoin-project/boost) ### [hashtag](https://boost.filecoin.io/getting-started#boost-compatibility-matrix) Boost compatibility matrix Boost Version Lotus Version Golang Version Curio Version < v2.0.0 [https://boost.filecoin.io/v/v1/getting-started#boost-and-lotus-compatibility-matrixarrow-up-right](https://boost.filecoin.io/v/v1/getting-started#boost-and-lotus-compatibility-matrix) [https://boost.filecoin.io/v/v1/getting-started#boost-and-lotus-compatibility-matrixarrow-up-right](https://boost.filecoin.io/v/v1/getting-started#boost-and-lotus-compatibility-matrix) NA v2.0.0 v1.23.x 1.20.x NA v2.1.0, v2.1.1, v2.1.2 v1.24.x, v1.25.x 1.20.x NA v2.2.0 v1.26.0, v1.26.1 1.21.7 NA v2.3.0 v1.27.x, v1.28.x 1.22.3 1.22.x, 1.23.x v2.4.0 v1.30.0 1.22.3 1.24.2 v2.4.1 v1.32.0-rc1 1.22.8 1.24.3, 1.24.4 v2.4.2 v1.32.1 1.23.7 NA v2.4.3 v1.32.2 1.23.7 NA v2.4.4 v1.34.0 1.24.7 NA v2.4.6 v1.34.1 1.24.7 NA v2.4.7 v1.34.1 1.24.7 NA v2.4.8 v1.35.0 1.24.7 NA v2.5.0 v1.35.0 1.25.0 NA v2.5.1 v1.35.1 1.26.2 NA ### [hashtag](https://boost.filecoin.io/getting-started#building-and-installing) Building and Installing #### [hashtag](https://boost.filecoin.io/getting-started#prerequisites) Prerequisites circle-exclamation Please make sure you have installed: **Go** - following [https://go.dev/learn/arrow-up-right](https://go.dev/learn/) **Rust** - following [https://www.rust-lang.org/tools/installarrow-up-right](https://www.rust-lang.org/tools/install) **Node 24.x** **Linux / Ubuntu** **macOS** #### [hashtag](https://boost.filecoin.io/getting-started#linux) Linux circle-exclamation Depending on your architecture, you will want to export additional environment variables: Please ignore any output or onscreen instruction during the `npm build` unless there is an error. #### [hashtag](https://boost.filecoin.io/getting-started#macos) macOS Please ignore any output or onscreen instruction during the `npm build` unless there is an error. #### [hashtag](https://boost.filecoin.io/getting-started#calibration-network) **Calibration Network** To build boost for _calibnet_, please complete the above pre-requisites and build using the following commands. [PreviousYugabyteDBchevron-left](https://boost.filecoin.io/hardware-requirements/yugabytedb) [NextNew Boost Setupchevron-right](https://boost.filecoin.io/new-boost-setup) Last updated 3 days ago * [Boost compatibility matrix](https://boost.filecoin.io/getting-started#boost-compatibility-matrix) * [Building and Installing](https://boost.filecoin.io/getting-started#building-and-installing) sun-brightdesktopmoon Copy curl -sL https://deb.nodesource.com/setup_24.x | sudo -E bash - sudo apt-get install -y nodejs sudo apt install mesa-opencl-icd ocl-icd-opencl-dev gcc git bzr jq pkg-config curl clang build-essential hwloc libhwloc-dev wget -y Copy brew install node@24 brew install jq pkg-config hwloc coreutils Copy export RUSTFLAGS="-C target-cpu=native -g" export FFI_BUILD_FROM_SOURCE=1 Copy git clone https://github.com/filecoin-project/boost cd boost git checkout make clean build sudo make install Copy export LIBRARY_PATH=$LIBRARY_PATH:/opt/homebrew/lib git clone https://github.com/filecoin-project/boost cd boost git checkout make clean build sudo make install Copy git clone https://github.com/filecoin-project/boost cd boost git checkout make clean calibnet sun-brightdesktopmoon --- # Dependencies | Boost Local Index Directory depends on a backend database to store various indices. Currently we support two implementations - YugabyteDB or LevelDB - depending on the size of deal data and indices a storage provider holds. **LevelDB** is an open source on-disk key-value store, and can be used when indices fit on a single host. **YugabyteDB** is an open source modern distributed database designed to run in any public, private, hybrid or multi-cloud environment. circle-info Storage providers who hold more than 1PiB data are encouraged to use YugabyteDB as it is horizontally scalable, provides better monitoring and management utilities and could support future growth. [PreviousIndex typeschevron-left](https://boost.filecoin.io/deployment/local-index-directory/architecture) [NextInitialisationchevron-right](https://boost.filecoin.io/deployment/local-index-directory/initialisation) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # booster-bitswap | Boost Go to the following page for more information on `booster-bitswap`: [Bitswap retrievalchevron-right](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval) [Previousbooster-httpchevron-left](https://boost.filecoin.io/deployment/booster-http) [Nextlibp2p Protocolschevron-right](https://boost.filecoin.io/deployment/libp2p-protocols) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Retrieving data from Filecoin | Boost Storage Providers on Filecoin support various modes of retrievals for deal data that they have accepted from clients. Reputation systems periodically crawl the network and record success rate for Storage Providers and determine whether they should make future FIL+ deals with them depending on how well they serve retrievals. [PreviousStoring data on Filecoinchevron-left](https://boost.filecoin.io/storing-data-on-filecoin) [NextHTTP retrievalchevron-right](https://boost.filecoin.io/retrieving-data-from-filecoin/http-retrieval) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Initialisation | Boost Considering that the Local Index Directory is a new feature, Storage Providers should initialise it after upgrading their Boost deployments. There are two ways a Storage Provider can do that: 1. **Migrate existing indices from the DAG store into LID**: this solution assumes that the Storage Provider has been keeping an _unsealed_ copy for every sector they prove on-chain, and has already indexed all their deal data into the DAG store. Typically index sizes for a given sector range between 100KiB up to 1GiB, depending on deal data and its blocks sizes. The DAG store keeps these indices in the repository directory of Boost under the `./dagstore/index` and `./dagstore/datastore` directories. This data should be migrated to LID with the `migrate-lid` utility. 2. **Recreate indices for deal data based on unsealed copies of sectors**: this solution assumes that the Storage Provider has _unsealed copies_ for every sector they prove on-chain. If this is not the case, then the SP should first trigger an _unseal (UNS)_ job on their system for every sector that contains user data and produce an unseal copy. SPs can use the `boostd recover lid` utility to produce an index for all deal data within an unsealed sector and store it in LID so that they enable retrievals for the data. Depending on SPs deployment and where unsealed copies are hosted (NFS, Ceph, external disks, etc.) and the performance of the hosting system, producing an index for a 32GiB sector can take anywhere from a few seconds up to a few minutes, as the unsealed copy needs to be processed by the utility. If you are migrating from Boost v1, make sure to read the tutorial: [How to upgrade from Boost v1 to Boost v2chevron-right](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2) [PreviousDependencieschevron-left](https://boost.filecoin.io/deployment/local-index-directory/requirements) [Nextboostd-datachevron-right](https://boost.filecoin.io/deployment/boostd-data) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Deal Filters | Boost ### [hashtag](https://boost.filecoin.io/configuration/deal-filters#using-filters-for-fine-grained-storage-and-retrieval-deal-acceptance) Using filters for fine-grained storage and retrieval deal acceptance Your use case might demand very precise and dynamic control over a combination of deal parameters. Lotus provides two IPC hooks allowing you to name a command to execute for every deal before the miner accepts it: * `Filter` for storage deals. * `RetrievalFilter` for retrieval deals. The executed command receives a JSON representation of the deal parameters on standard input, and upon completion, its exit code is interpreted as: * `0`: success, proceed with the deal. * `non-0`: failure, reject the deal. The most trivial filter rejecting any retrieval deal would be something like: `RetrievalFilter = "/bin/false"`. `/bin/false` is binary that immediately exits with a code of `1`. [This Perl scriptarrow-up-right](https://gist.github.com/ribasushi/53b7383aeb6e6f9b030210f4d64351d5/9bd6e898f94d20b50e7c7586dc8b8f3a45dab07c#file-dealfilter-pl) lets the miner deny specific clients and only accept deals that are set to start relatively soon. You can also use a third party content policy framework like [CIDgravityarrow-up-right](https://www.cidgravity.com/) or `bitscreen` by Murmuration Labs: Copy # grab filter program go get -u -v github.com/Murmuration-Labs/bitscreen # add it to both filters Filter = "/path/to/go/bin/bitscreen" RetrievalFilter = "/path/to/go/bin/bitscreen" [PreviousHTTP Transfer limitchevron-left](https://boost.filecoin.io/configuration/http-transfer-limit) [NextRemote CommPchevron-right](https://boost.filecoin.io/configuration/remote-commp) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Manual Publish Storage Deal Message | Boost By default boost publishes storage deals automatically once 8 deals are in the publish queue, or after 24 hours. However some SPs need to be able to control exactly which deals to publish and when. This new feature allows SPs to turn on manual PSD. Once it is turned on, Boost will no longer send any PSD messages unless explicitly prompted by the user. The feature can be turned on with the below config variable: Copy [Dealmaking] // When set to true, the user is responsible for publishing deals manually. // The values of MaxDealsPerPublishMsg and PublishMsgPeriod will be // ignored, and deals will remain in the pending state until manually published. ManualDealPublish bool ### [hashtag](https://boost.filecoin.io/configuration/manual-publish-storage-deal-message#querying-the-deals-pending-to-be-published) Querying the deals pending to be published The deals which have not been published can be queried using the Graphql endpoint of the Boost. The curl for the query: `curl -X POST -H "Content-Type: application/json" -d '{"query":"query { dealPublish{ ManualPSD Deals {ID IsLegacy ClientAddress ProviderAddress CreatedAt PieceCid PieceSize ProviderCollateral StartEpoch EndEpoch ClientPeerID PublishCid Transfer { Type Size Params ClientID} Message }}}"}' http://localhost:8080/graphql/query | jq` ### [hashtag](https://boost.filecoin.io/configuration/manual-publish-storage-deal-message#decision) Decision SPs can create a decision script which queries the Graphql endpoint using the above curl and take a decision on which deal to be published. ### [hashtag](https://boost.filecoin.io/configuration/manual-publish-storage-deal-message#publishing-deals) Publishing deals Once the decision has been taken on which deals to be published, SPs can use the Grapqhql endpoint to Publish the deals. The mutation is: `publishPendingDeals(ids: [ID!]!): [ID!]!` which takes an array of deal UUIDs to be published and an error message is returned. The following curl can be used to publish the deal. `curl -X POST -H "Content-Type: application/json" -d '{"query":"mutation {publishPendingDeals(ids: [\"d9f849f1-d5d8-4bfc-b034-2866bddfc8cb\", \"a8eb58ef-7381-4251-ae7a-1227c032c0b9\"])}"}' http://localhost:8080/graphql/query | jq` **Successful output** **Failure output** ### [hashtag](https://boost.filecoin.io/configuration/manual-publish-storage-deal-message#publish-all-deals) Publish all deals The publish all button in UI would have the same functionality as before. Same would be true for the Graphql mutation `dealPublishNow` ### [hashtag](https://boost.filecoin.io/configuration/manual-publish-storage-deal-message#ui-screenshots) UI screenshots ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252Fg2bOX1bJvrWcxjxRZKNA%252FScreenshot%25202023-07-19%2520at%252011.09.14%2520AM.png%3Falt%3Dmedia%26token%3D7d551b28-2a54-4de5-9817-6e26c6684472&width=768&dpr=3&quality=100&sign=9479b453&sv=2) Publish Page in Boost UI [PreviousHTTP indexer announcementchevron-left](https://boost.filecoin.io/configuration/http-indexer-announcement) [NextMonitoringchevron-right](https://boost.filecoin.io/monitoring) Last updated 2 years ago * [Querying the deals pending to be published](https://boost.filecoin.io/configuration/manual-publish-storage-deal-message#querying-the-deals-pending-to-be-published) * [Decision](https://boost.filecoin.io/configuration/manual-publish-storage-deal-message#decision) * [Publishing deals](https://boost.filecoin.io/configuration/manual-publish-storage-deal-message#publishing-deals) * [Publish all deals](https://boost.filecoin.io/configuration/manual-publish-storage-deal-message#publish-all-deals) * [UI screenshots](https://boost.filecoin.io/configuration/manual-publish-storage-deal-message#ui-screenshots) sun-brightdesktopmoon Copy { "data": { "publishPendingDeals": [\ "d9f849f1-d5d8-4bfc-b034-2866bddfc8cb",\ "a8eb58ef-7381-4251-ae7a-1227c032c0b9"\ ] } } Copy { "errors": [\ {\ "message": "failed to get deal details from DB cc78d877-f69c-410b-80a7-b23f2fab8951: scanning deal row: sql: no rows in result set",\ "path": [\ "publishPendingDeals"\ ]\ }\ ], "data": null } sun-brightdesktopmoon --- # Index types | Boost In order to support the described retrieval use cases, LID maintains the following indexes: #### [hashtag](https://boost.filecoin.io/deployment/local-index-directory/architecture#multihash-piece-cid) multihash → \[\]piece cid To look up which pieces contain a block #### [hashtag](https://boost.filecoin.io/deployment/local-index-directory/architecture#piece-cid-sector-information-sector-id-offset-size) piece cid → sector information {sector ID, offset, size} To look up which sector a piece is in #### [hashtag](https://boost.filecoin.io/deployment/local-index-directory/architecture#piece-cid-map-less-than-mulithash-block-offset-size-greater-than) piece cid → map To look up where in the piece a block is and the block’s size ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252Fz8PaELdAj4HWi3QdJr9x%252FScreenshot%25202023-05-18%2520at%252014.01.13.png%3Falt%3Dmedia%26token%3D668ab27e-33df-4dd0-b687-8894ce267da3&width=768&dpr=3&quality=100&sign=9774a0a&sv=2) ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252FPaxZD36055G2Lis2nI31%252FScreenshot%25202023-05-18%2520at%252014.01.23.png%3Falt%3Dmedia%26token%3Dc0459136-b3d5-48ca-9ba3-bab5f7161737&width=768&dpr=3&quality=100&sign=ed356a21&sv=2) [PreviousLocal Index Directorychevron-left](https://boost.filecoin.io/deployment/local-index-directory) [NextDependencieschevron-right](https://boost.filecoin.io/deployment/local-index-directory/requirements) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # HTTP indexer announcement | Boost [hashtag](https://boost.filecoin.io/configuration/http-indexer-announcement#announce-over-http) Announce over HTTP ----------------------------------------------------------------------------------------------------------------------- `IndexProvider.HttpPublisher.AnnounceOverHttp` must be set to `true` to enable the http announcements. Once HTTP announcements are enabled, the local-index provider will continue to announce over libp2p gossipsub along with HTTP for the specific indexers. The advertisements are send to the indexer nodes defined in `DirectAnnounceURLs`. You can specify more than 1 URL to announce to multiple indexer nodes. Once an IPNI node starts processing the advertisements, it will reach out to the Boost node to fetch the data. Thus, Boost node needs to specify a public IP and port which can be used by the indexer node to query for data. Copy [IndexProvider] # Enable set whether to enable indexing announcement to the network and expose endpoints that # allow indexer nodes to process announcements. Enabled by default. # # type: bool # env var: LOTUS_INDEXPROVIDER_ENABLE #Enable = true # EntriesCacheCapacity sets the maximum capacity to use for caching the indexing advertisement # entries. Defaults to 1024 if not specified. The cache is evicted using LRU policy. The # maximum storage used by the cache is a factor of EntriesCacheCapacity, EntriesChunkSize and # the length of multihashes being advertised. For example, advertising 128-bit long multihashes # with the default EntriesCacheCapacity, and EntriesChunkSize means the cache size can grow to # 256MiB when full. # # type: int # env var: LOTUS_INDEXPROVIDER_ENTRIESCACHECAPACITY #EntriesCacheCapacity = 1024 # EntriesChunkSize sets the maximum number of multihashes to include in a single entries chunk. # Defaults to 16384 if not specified. Note that chunks are chained together for indexing # advertisements that include more multihashes than the configured EntriesChunkSize. # # type: int # env var: LOTUS_INDEXPROVIDER_ENTRIESCHUNKSIZE #EntriesChunkSize = 16384 # TopicName sets the topic name on which the changes to the advertised content are announced. # If not explicitly specified, the topic name is automatically inferred from the network name # in following format: '/indexer/ingest/' # Defaults to empty, which implies the topic name is inferred from network name. # # type: string # env var: LOTUS_INDEXPROVIDER_TOPICNAME #TopicName = "" # PurgeCacheOnStart sets whether to clear any cached entries chunks when the provider engine # starts. By default, the cache is rehydrated from previously cached entries stored in # datastore if any is present. # # type: bool # env var: LOTUS_INDEXPROVIDER_PURGECACHEONSTART #PurgeCacheOnStart = true [IndexProvider.Announce] # Make a direct announcement to a list of indexing nodes over http. # Note that announcements are already made over pubsub regardless # of this setting. # # type: bool # env var: LOTUS_INDEXPROVIDER_ANNOUNCE_ANNOUNCEOVERHTTP #AnnounceOverHttp = true # The list of URLs of indexing nodes to announce to. # # type: []string # env var: LOTUS_INDEXPROVIDER_ANNOUNCE_DIRECTANNOUNCEURLS #DirectAnnounceURLs = ["https://cid.contact/ingest/announce"] [IndexProvider.HttpPublisher] # If not enabled, requests are served over graphsync instead. # # type: bool # env var: LOTUS_INDEXPROVIDER_HTTPPUBLISHER_ENABLED #Enabled = true # Set the public hostname / IP for the index provider listener. # eg "82.129.73.111" # This is usually the same as the for the boost node. # # type: string # env var: LOTUS_INDEXPROVIDER_HTTPPUBLISHER_PUBLICHOSTNAME #PublicHostname = "82.129.73.111" # Set the port on which to listen for index provider requests over HTTP. # Note that this port must be open on the firewall. # # type: int # env var: LOTUS_INDEXPROVIDER_HTTPPUBLISHER_PORT #Port = 3401 [PreviousLegacy Deal configurationchevron-left](https://boost.filecoin.io/configuration/legacy-deal-configuration) [NextManual Publish Storage Deal Messagechevron-right](https://boost.filecoin.io/configuration/manual-publish-storage-deal-message) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Remote CommP | Boost Boost is introducing a new feature that allows computing `commP` during the deal on a lotus-worker node. This should reduce the overall resource utilisation on the Boost node. In order to enable remote commP on a Boost node, update your `config.toml`: Copy [Dealmaking] RemoteCommp = true Then restart the Boost node [PreviousDeal Filterschevron-left](https://boost.filecoin.io/configuration/deal-filters) [NextLegacy Deal configurationchevron-right](https://boost.filecoin.io/configuration/legacy-deal-configuration) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # YugabyteDB | Boost YugabyteDB is used to store the retrievals indexes. When a client makes a retrieval request, LID service is used to lookup the requested data within the Lotus miner. For more details see the [architecture of LID](https://boost.filecoin.io/deployment/local-index-directory) . Depending on the size of data your SP is holding, you can run [YugabyteDBarrow-up-right](https://docs.yugabyte.com/preview/architecture/concepts/) in a variety of ways. You can find more information about how to run YugabyteDB [herearrow-up-right](https://docs.yugabyte.com/preview/launch-and-manage/) . For production use, YugabyteDB can be deployed as a highly available (HA) cluster. The cluster can be deployed on a [Kubernetes instancearrow-up-right](https://docs.yugabyte.com/preview/deploy/kubernetes/) or as a [manual deploymentarrow-up-right](https://docs.yugabyte.com/preview/deploy/manual-deployment/) . Please ensure to read all the [pre-requisitesarrow-up-right](https://docs.yugabyte.com/preview/deploy/checklist/) before proceeding. triangle-exclamation YugabyteDB must be backed up regularly to avoid loosing deal metadata. Once this data is lost, there is no reliable way to recover it. You can then start `boostd-data` on `:8044` and connect it to the `yugabyte` with: Copy boostd-data run yugabyte --hosts 127.0.0.1 \ --connect-string="postgresql://postgres:[email protected]:5433" \ --addr 0.0.0.0:8044 circle-info The PGX driver from Yugabyte supports cluster aware Postgres connection out of the box. If you are deploying a multi-node YugabyteDB cluster, then please update your connect-string to use a cluster aware connection. With Cluster Mode: "postgresql://postgres:[\[email protected\]](https://boost.filecoin.io/cdn-cgi/l/email-protection) :5433?load\_balance=true" With Cluster Mode + No SSL: "postgresql://postgres:[\[email protected\]](https://boost.filecoin.io/cdn-cgi/l/email-protection) :5433?sslmode=disable&load\_balance=true" [hashtag](https://boost.filecoin.io/deployment/yugabytedb#yugabytedb-faq) YugabyteDB FAQ --------------------------------------------------------------------------------------------- 1. **What is YugabyteDB and why YugabyteDB?** YugabyteDB is a high-performance distributed SQL database. Built using a unique combination of high-performance document store, per-shard distributed consensus replication and multi-shard ACID transactions (inspired by Google Spanner), YugabyteDB serves both scale-out RDBMS and internet-scale OLTP workloads with low query latency, extreme resilience against failures and global data distribution. We tested multiple open source databases for the Boost use case and found YugabyteDB to be well suited as it is highly performant and scales well horizonally. 2. **How do I learn about YugabyteDB and what do I need to know about YugabyteDB as an SP?** SPs should familiarize themselves with the “[Deploy and Managearrow-up-right](https://docs.yugabyte.com/preview/launch-and-manage/) ” section of the [documentationarrow-up-right](https://docs.yugabyte.com/preview/) along with the [architecturearrow-up-right](https://docs.yugabyte.com/preview/architecture/concepts/) before deploying YugabyteDB. YugabyteDB will also be utilized by Lotus V2 architecture as well. We plan to allow SPs to connect all of their Boost instances to a single LID (YugabyteDB) with Boost v2.1.0 release. This will also allow SPs to serve retrievals from any of their miners using a single `booster-http` or `booster-bitswap` process. 3. **Which deployment should I choose?** We recommend deploying YugabyteDB either locally on bare metal or using a managed deployment. Users can choose to deploy on the cloud if they can guarantee that scaling up the DB will not be impacted by the network bandwidth and infrastructure. It is recommended that YugabyteDB is highly available so that if one of the nodes is not available, your SP operations will not be impacted. You can find some of [the example YugabyteDB deployment arrow-up-right](https://github.com/filecoin-project/boost/discussions/1797) here. Please feel free to add your experience and deployment details to the discussion. 4. **Can I convert YugabyteDB processes to systemd services?** YugabyteDB does not ship as a systemd service by default. You will need to create a new service based on the commands you are running to start YT-Master and YT-Server processes. These commands can be customized based on user requirements and infrastructure. 5. **Once YugabyteDB is deployed, do I need to perform any additional steps?** Ideally, once the deployment is complete and can be reached over the network by `boostd-data` service, users do not need to perform any additional steps. If you wish to change the default username/password, you must also update the same on the `--connect-string` of `boostd-data` service. circle-info You can find some of [the example YugabyteDB deployment arrow-up-right](https://github.com/filecoin-project/boost/discussions/1797) here. Please feel free to add your experience and deployment details to the discussion. [hashtag](https://boost.filecoin.io/deployment/yugabytedb#yugabytedb-maintenance-and-upgrades) YugabyteDB Maintenance and Upgrades --------------------------------------------------------------------------------------------------------------------------------------- Boost is not currently tied to a specific version of YugabyteDB. We recommend setting up the latest stable version of the YugabyteDB when creating a new LID instance. Over the time, users can upgrade their YugabyteDB. When upgrading: 1. Boost-related processes must be stopped beforehand. If YugabyteDB is being utilised by other services apart from Boost then those services must be stopped as well. 2. Read through and follow the [upgrade guidearrow-up-right](https://docs.yugabyte.com/preview/manage/upgrade-deployment/) and reach out to [YugabyteDB team on Slackarrow-up-right](https://communityinviter.com/apps/yugabyte-db/register) if you have any questions. As per the industry best practices, YugabyteDB should be [regularly backed uparrow-up-right](https://docs.yugabyte.com/preview/manage/backup-restore/) for redundancy. Check out [YugabyteDB docsarrow-up-right](https://docs.yugabyte.com/preview/) if you need any help with management of the database. [hashtag](https://boost.filecoin.io/deployment/yugabytedb#yugabytedb-troubleshooting) YugabyteDB Troubleshooting --------------------------------------------------------------------------------------------------------------------- If you require help with YugabyteDB, we recommend checking out the [troubleshooting guidearrow-up-right](https://docs.yugabyte.com/preview/troubleshoot/) . If the problems is not resolved then users should reach out to YugabyteDB team via [Slackarrow-up-right](https://communityinviter.com/apps/yugabyte-db/register) or [Githubarrow-up-right](https://github.com/yugabyte/yugabyte-db) for any support. You can also reach out to other users in #boost-help channel of filecoin slack to seek help from your peers. [Previousboostd-datachevron-left](https://boost.filecoin.io/deployment/boostd-data) [Nextbooster-httpchevron-right](https://boost.filecoin.io/deployment/booster-http) Last updated 1 year ago * [YugabyteDB FAQ](https://boost.filecoin.io/deployment/yugabytedb#yugabytedb-faq) * [YugabyteDB Maintenance and Upgrades](https://boost.filecoin.io/deployment/yugabytedb#yugabytedb-maintenance-and-upgrades) * [YugabyteDB Troubleshooting](https://boost.filecoin.io/deployment/yugabytedb#yugabytedb-troubleshooting) sun-brightdesktopmoon sun-brightdesktopmoon --- # Storing data on Filecoin | Boost Boost comes with a client executable, `boost`, that can be used to send a deal proposal to a Boost Storage Provider. The client is intentionally minimal meant for developer testing. It is not a full featured client and is not intended to be so. It does not require a daemon process, and can be pointed at any public Filecoin API for on-chain operations. This means that users of the client do not need to run a Filecoin node that syncs the chain. ### [hashtag](https://boost.filecoin.io/storing-data-on-filecoin#set-the-api-endpoint-environment-variable) Set the API endpoint environment variable Copy export FULLNODE_API_INFO= circle-info There are a number of public Filecoin APIs ran by a number of organisations, such as Infura, Glif, etc. For test purposes you can try: `export FULLNODE_API_INFO=https://api.node.glif.io` ### [hashtag](https://boost.filecoin.io/storing-data-on-filecoin#initialize-the-client) Initialize the client Copy boost -vv init The `init` command * Creates a Boost client repository (at `~/.boost-client` by default) * Generates a libp2p peer ID key * Generates a wallet for on-chain operations and outputs the wallet address ### [hashtag](https://boost.filecoin.io/storing-data-on-filecoin#add-funds-to-the-wallet-and-to-the-market-actor) Add funds to the wallet and to the market actor To make deals you will need to: a) add funds to the wallet b) add funds to the market actor for that wallet address ### [hashtag](https://boost.filecoin.io/storing-data-on-filecoin#make-a-storage-deal) Make a storage deal Currently, we don't distribute binaries, so you will have to build from source. When a storage provider accepts the deal, you should see output of the command similar to: ### [hashtag](https://boost.filecoin.io/storing-data-on-filecoin#check-deal-status) Check deal status You can check the `deal status` with the following command: [PreviousSetting up a monitoring stack for Boostchevron-left](https://boost.filecoin.io/monitoring/setting-up-a-monitoring-stack-for-boost) [NextRetrieving data from Filecoinchevron-right](https://boost.filecoin.io/retrieving-data-from-filecoin) Last updated 1 month ago * [Set the API endpoint environment variable](https://boost.filecoin.io/storing-data-on-filecoin#set-the-api-endpoint-environment-variable) * [Initialize the client](https://boost.filecoin.io/storing-data-on-filecoin#initialize-the-client) * [Add funds to the wallet and to the market actor](https://boost.filecoin.io/storing-data-on-filecoin#add-funds-to-the-wallet-and-to-the-market-actor) * [Make a storage deal](https://boost.filecoin.io/storing-data-on-filecoin#make-a-storage-deal) * [Check deal status](https://boost.filecoin.io/storing-data-on-filecoin#check-deal-status) sun-brightdesktopmoon Copy boost -vv deal --provider= \ --http-url= \ --commp= \ --car-size= \ --piece-size= \ --payload-cid= Copy sent deal proposal deal uuid: 9e68fb16-ff9a-488e-ad0a-1289b512d176 storage provider: f0127896 client wallet: f1sw5zjcyo4mff5cbvgsgmm8uoko6gcr4tptvtkhy payload cid: bafyaa6qshafcmalqudsbeidrunclaep6mdbipm2gjfvuosjfd6cbqd6th7bshy5hi5npxe727yjaagelucbyabasgafcmalqudsaeieapsxspo2i36no36n7yitswsxdazvziwvgj4vbp2scuxasrc6n4ejaage3r7m3saykcqeaegeavdllsbzaqcaibaaeecakrvvzam url: https://webserver/file.car commp: baga6ea4seaqh5prrl6ykov4t64k6m5giijsc44dcxtdnzsp4izjakqhs7twauiq start epoch: 1700711 end epoch: 2219111 provider collateral: 358.687 μFIL Copy boost deal-status --provider= --deal-uuid= Copy got deal status response deal uuid: 9e68fb16-ff8a-488e-ad0a-1289b512d176 deal status: Transfer Queued deal label: bafyaa6qsgafcmalqudsaeidrunclaep6mdbipm2gjfvuosjfd6cbqd6th7bshy5hi5npxe727yjaagelucbyabasgafcmalqudsaeieapsxspo2i36no36n7yitswsxdazvziwvgj4vbp2scuxasrc6n4ejaage3r7m3saykcqeaegeavdllsbzaqcaibaaeecakrvvzam publish cid: chain deal id: 0 sun-brightdesktopmoon --- # boostd-data | Boost The `boostd-data` service is a proxy to the underlying backend database that hosts the Local Index Directory (LID). Currently there is support for two stores: * LevelDB * YugabyteDB [hashtag](https://boost.filecoin.io/deployment/boostd-data#usage) Usage ---------------------------------------------------------------------------- Copy boostd-data run yugabyte --hosts \ --connect-string \ --addr We recommend creating a systemd file for this service and utilising it to easily start and stop the `boostd-data` service. Please note that this service is an independent process and is not controlled by any other Boost process. circle-info The PGX driver from Yugabyte supports cluster aware Postgres connection out of the box. If you are deploying a multi-node YugabyteDB cluster, then please update your connect-string to use a cluster aware connection. With Cluster Mode: "postgresql://postgres:[\[email protected\]](https://boost.filecoin.io/cdn-cgi/l/email-protection) :5433?load\_balance=true" With Cluster Mode + No SSL: "postgresql://postgres:[\[email protected\]](https://boost.filecoin.io/cdn-cgi/l/email-protection) :5433?sslmode=disable&load\_balance=true" circle-info \--hosts takes the IP addresses of the YugabyteDB YT-Servers. Example: \--hosts=10.0.0.1 --hosts=10.0.0.2 --hosts=10.0.0.3 \--addr is the : where `boostd-data` service should be listening on. The IP here can be a private one (recommended) and should reachable by all boost related processes. Please ensure to update your firewall configuration accordingly. [PreviousInitialisationchevron-left](https://boost.filecoin.io/deployment/local-index-directory/initialisation) [NextYugabyteDBchevron-right](https://boost.filecoin.io/deployment/yugabytedb) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # New Boost Setup | Boost triangle-exclamation If you are already running a Boost node then please follow the [migration tutorial](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2) and do not attempt the below steps. This can result in permanent data loss. circle-exclamation Make sure you have read the [**Components**](https://boost.filecoin.io/deployment) page before proceeding. Boost v2 introduces a new service called **boostd-data** which requires a database to be installed - YugabyteDB or LevelDB. [hashtag](https://boost.filecoin.io/new-boost-setup#introduction) Introduction ----------------------------------------------------------------------------------- Boost v2 introduces the Local Index Directory as a replacement for the DAG store. It scales horizontally and provides a more intuitive experience for users, by surfacing problems in the UI and providing repair functionality. ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252FAwagrPbzXPS9sskaJwwi%252FScreenshot%25202023-06-14%2520at%252013.13.54.png%3Falt%3Dmedia%26token%3D225d016b-340a-48c7-a5de-4e7b70aed0e2&width=768&dpr=3&quality=100&sign=f7aedf2c&sv=2) [hashtag](https://boost.filecoin.io/new-boost-setup#prerequisites) Prerequisites ------------------------------------------------------------------------------------- ### [hashtag](https://boost.filecoin.io/new-boost-setup#install-yugabytedb) Install YugabyteDB The Local Index Directory (LID) stores retrieval indices in YugabyteDB. Retrieval indices store the size and location of each block in the deal data. It is recommend to run YugabyteDB on a dedicated host with SSD drives. Depending on how many blocks there are in the user data, the retrieval indices may require up to 2% of the size of the unsealed data. e.g. 1 TiB of unsealed user data may require a 20 GiB index. YugabyteDB requires about the same amount of space as your DAG store requires today. You can find more information about YugabyteDB in the `Components` section: [YugabyteDBchevron-right](https://boost.filecoin.io/deployment/yugabytedb) [hashtag](https://boost.filecoin.io/new-boost-setup#instructions) Instructions ----------------------------------------------------------------------------------- Follow these instructions in order to setup a new Boost instance with `boostd-data` service: 1. Make sure you have a Lotus node and miner running 2. Create and send funds to two new wallets on the lotus node to be used for Boost. Boost currently uses two wallets for storage deals: * The publish storage deals wallet - This wallet pays the gas cost when Boost sends the `PublishStorageDeals` message. * The deal collateral wallet - When the Storage Provider accepts a deal, they must put collateral for the deal into escrow. Boost moves funds from this wallet into escrow with the `StorageMarketActor`. 1. Set the publish storage deals wallet as a control wallet 1. [Build the latest Boost V2 binaries](https://boost.filecoin.io/getting-started#building-and-installing) 2. Run the `boostd-data` service `boostd-data` is a data proxy service which abstracts the access to LID through an established API. It makes it easier to secure the underlying database and not expose it. `boostd-data` listens to a _websocket_ interface, which is the entrypoint which should be exposed to `boostd`, and`booster-http` Start the `boostd-data` service with parameters to connect to YugabyteDB on its Cassandra and PostgreSQL interfaces: circle-info The `boostd-data` service can run a separate machine from Boost as long as the service is reachable via the network from the boost node 1. Create and initialize the Boost repository Boost keeps all data in a directory called the _repository_. By default the repository is at `~/.boost`. To use a different location pass the `--boost-repo` parameter (must precede any particular command verb, e.g. `boostd --boost-repo= init`). Export the environment variables needed for `boostd init` to connect to the lotus daemon and lotus miner. Export environment variables that point to the API endpoints for the sealing and mining processes. They will be used by the `boost` node to make JSON-RPC calls to the `mining/sealing/proving` node. Run `boostd init` to create and initialize the repository: * `--api-sealer` is the API info for the lotus-miner instance that does sealing * `--api-sector-index` is the API info for the lotus-miner instance that provides storage * `--max-staging-deals-bytes` is the maximum amount of storage to be used for downloaded files (once the limit is reached Boost will reject subsequent incoming deals) 1. Update `ulimit` file descriptor limit if necessary. Boost deals will fail if the file descriptor limit for the process is not set high enough. This limit can be raised temporarily before starting the Boost process by running the command `ulimit -n 1048576`. We recommend setting it permanently by following the [Permanently Setting Your ULIMIT System Valuearrow-up-right](https://lotus.filecoin.io/kb/soft-fd-limit/) guide. 2. Before you start your boostd, it is important that it is reachable from any peer in the Filecoin network. For this, you will need a stable public IP and edit your `/config.toml` as follows: 3. Add `boostd-data` details to the `boostd` config Configure `boostd` repository config (located at `/config.toml`) to point to the exposed `boostd-data` service endpoint. Note that the connection must be configured to go over a websocket. For example: 4. Start the `boostd` process 5. Make sure that the correct _peer id_ and _multiaddr_ for your SP is set on chain, given that `boost init` generates a new identity. Use the following commands to update the values on chain: circle-info should be the same as the `AnnounceAddresses` you set in the `Libp2p` section of the config.toml of Boost can be found in the output of `boostd net id` command [hashtag](https://boost.filecoin.io/new-boost-setup#conclusion) Conclusion ------------------------------------------------------------------------------- At this stage you should have the latest version of Boost running with the Local Index Directory. Go to the **Local Index Directory** page and review the number sections: #### [hashtag](https://boost.filecoin.io/new-boost-setup#pieces) Pieces ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252F3NgBMaEMAkyGYbLRLQpd%252FScreenshot%25202023-06-19%2520at%252013.43.34.png%3Falt%3Dmedia%26token%3D1985d12c-8b93-4164-9acf-e80b33148fa9&width=768&dpr=3&quality=100&sign=da7f1c23&sv=2) Pieces section shows counters for total pieces of user data that your SP is storing as well as whether you are keeping _unsealed_ and _indexed_ copies of them. #### [hashtag](https://boost.filecoin.io/new-boost-setup#flagged-pieces) Flagged pieces ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252F3o0hRSr6XEXmnrkpZ5rm%252FScreenshot%25202023-06-19%2520at%252016.31.18.png%3Falt%3Dmedia%26token%3D6bd101e1-dd11-480a-8f22-08d185231e48&width=768&dpr=3&quality=100&sign=587e99c4&sv=2) _Flagged pieces_ are pieces that either lack an unsealed copy or are missing an index. For the sealed-only user data, you should make sure that you unseal individual sectors if you want this data to be retrievable. _Sealed only_ copies of data are not retrievable and are only being proven on-chain within the corresponding _deadline_ / _window_. Typically sealed only data is considered as archival as it is not immediately retrievable. If the client requests it, the SP sealing pipeline must first unseal it, which typically takes 1-2 hours, and only then the data becomes available. _Flagged (unsealed)_ pieces is user data that your SP is hosting, which is not indexed. circle-info We recommend that you trigger re-indexing for these pieces, so that data becomes retrievable. Check the [tutorial on re-indexing flagged unsealed pieces](https://boost.filecoin.io/tutorials/how-to-re-index-unsealed-pieces-that-are-flagged-by-lid-in-boost-v2) for more information. #### [hashtag](https://boost.filecoin.io/new-boost-setup#deal-sector-copies) Deal Sector Copies ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252FwHBAINXXZm5VFWvtL3Ge%252FScreenshot%25202023-06-19%2520at%252016.34.41.png%3Falt%3Dmedia%26token%3Dc03d3e55-1461-477d-87f4-cf5ac3a23f60&width=768&dpr=3&quality=100&sign=28135436&sv=2) Deal Sector Copies section displays counters of your sectors state - whether you keep unsealed copies for all sectors or not. Ideally the SP should keep unsealed copies for all data that should be immediately retrievable. #### [hashtag](https://boost.filecoin.io/new-boost-setup#sector-proving-state) Sector Proving State ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252FjiB1nOKeKaaLyBt2usHK%252FScreenshot%25202023-06-19%2520at%252016.34.46.png%3Falt%3Dmedia%26token%3D495b803e-8ff5-436e-b6f1-b670aa4b95b9&width=768&dpr=3&quality=100&sign=f52b4e6c&sv=2) Sector Proving State section displays counters of your active and inactive sectors - active sectors are those that are actively proven on-chain, inactive sectors are those that you might have failed to publish a _WindowPoSt_ for, or are expired or removed. [PreviousInstallationchevron-left](https://boost.filecoin.io/getting-started) [NextConfigurationchevron-right](https://boost.filecoin.io/configuration) Last updated 1 month ago * [Introduction](https://boost.filecoin.io/new-boost-setup#introduction) * [Prerequisites](https://boost.filecoin.io/new-boost-setup#prerequisites) * [Install YugabyteDB](https://boost.filecoin.io/new-boost-setup#install-yugabytedb) * [Instructions](https://boost.filecoin.io/new-boost-setup#instructions) * [Conclusion](https://boost.filecoin.io/new-boost-setup#conclusion) sun-brightdesktopmoon Copy export PUBLISH_STORAGE_DEALS_WALLET=`lotus wallet new bls` export COLLAT_WALLET=`lotus wallet new bls` lotus send --from mywallet $PUBLISH_STORAGE_DEALS_WALLET 10 lotus send --from mywallet $COLLAT_WALLET 10 Copy lotus-miner actor control set --really-do-it $PUBLISH_STORAGE_DEALS_WALLET Copy ./boostd-data run yugabyte \ --hosts \ --connect-string="postgresql://:@:5433" \ --addr 0.0.0.0:8044 Copy export $(lotus auth api-info --perm=admin) export $(lotus-miner auth api-info --perm=admin) Copy export APISEALER=`echo $MINER_API_INFO` export APISECTORINDEX=`echo $MINER_API_INFO` Copy boostd --vv init \ --api-sealer=$APISEALER \ --api-sector-index=$APISECTORINDEX \ --wallet-publish-storage-deals=$PUBLISH_STORAGE_DEALS_WALLET \ --wallet-deal-collateral=$COLLAT_WALLET \ --max-staging-deals-bytes=50000000000 Copy [Libp2p] ListenAddresses = ["/ip4/0.0.0.0/tcp/24001"] # choose a fixed port AnnounceAddresses = ["/ip4//tcp/24001"] # important! Copy [LocalIndexDirectory] ServiceApiInfo = "ws://:8044" Copy boostd --vv run Copy lotus-miner actor set-addrs lotus-miner actor set-peer-id sun-brightdesktopmoon --- # HTTP Transfer limit | Boost Boost provides a capability to limit the number of simultaneous http transfer in progress to download the deal data from the clients. This new configuration has been introduced in the `ConfigVersion = 3` of the boost configuration file. ### [hashtag](https://boost.filecoin.io/configuration/http-transfer-limit#configuration-variables) Configuration Variables #### [hashtag](https://boost.filecoin.io/configuration/http-transfer-limit#http-variables) HTTP variables Copy # The maximum number of concurrent storage deal HTTP downloads. # Note that this is a soft maximum; if some downloads stall, # more downloads are allowed to start. # # type: uint64 # env var: LOTUS_DEALMAKING_HTTPTRANSFERMAXCONCURRENTDOWNLOADS #HttpTransferMaxConcurrentDownloads = 20 # The period between checking if downloads have stalled. # # type: Duration # env var: LOTUS_DEALMAKING_HTTPTRANSFERSTALLCHECKPERIOD #HttpTransferStallCheckPeriod = "30s" # The time that can elapse before a download is considered stalled (and # another concurrent download is allowed to start). # # type: Duration # env var: LOTUS_DEALMAKING_HTTPTRANSFERSTALLTIMEOUT #HttpTransferStallTimeout = "5m0s" #### [hashtag](https://boost.filecoin.io/configuration/http-transfer-limit#storage-variables) Storage variables ### [hashtag](https://boost.filecoin.io/configuration/http-transfer-limit#how-transferlimiter-works) How TransferLimiter works The `transferLimiter` maintains a queue of transfers with a soft upper limit on the number of concurrent transfers. To prevent slow or stalled transfers from blocking up the queue there are a couple of mitigations: The queue is ordered such that we * start transferring data for the oldest deal first * prefer to start transfers with peers that don't have any ongoing transfer * once the soft limit is reached, don't allow any new transfers with peers that have existing stalled transfers ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252FcOCgj4kFB5Y4ePlc7aQl%252Fimage.png%3Falt%3Dmedia%26token%3D5b244c3e-1cae-4163-b9aa-acbf60db6b33&width=768&dpr=3&quality=100&sign=89e33a7b&sv=2) How TransferLimiter works Note that peers are distinguished by their host (eg foo.bar:8080) not by libp2p peer ID. For example, if there is * one active transfer with peer A * one pending transfer (peer A) * one pending transfer (peer B) The algorithm will prefer to start a transfer with peer B than peer A. This helps to ensure that slow peers don't block the transfer queue. The limit on the number of concurrent transfers is soft. Example: if there is a limit of 5 concurrent transfers and there are * three active transfers * two stalled transfers then two more transfers are permitted to start (as long as they're not with one of the stalled peers) [PreviousUI Settingschevron-left](https://boost.filecoin.io/configuration/ui-settings) [NextDeal Filterschevron-right](https://boost.filecoin.io/configuration/deal-filters) Last updated 2 years ago * [Configuration Variables](https://boost.filecoin.io/configuration/http-transfer-limit#configuration-variables) * [How TransferLimiter works](https://boost.filecoin.io/configuration/http-transfer-limit#how-transferlimiter-works) sun-brightdesktopmoon Copy # The maximum allowed disk usage size in bytes of downloaded deal data # that has not yet been passed to the sealing node by boost. # When the client makes a new deal proposal to download data from a host, # boost checks this config value against the sum of: # - the amount of data downloaded in the staging area # - the amount of data that is queued for download # - the amount of data in the proposed deal # If the total amount would exceed the limit, boost rejects the deal. # Set this value to 0 to indicate there is no limit. # # type: int64 # env var: LOTUS_DEALMAKING_MAXSTAGINGDEALSBYTES MaxStagingDealsBytes = 50000000000 # The percentage of MaxStagingDealsBytes that is allocated to each host. # When the client makes a new deal proposal to download data from a host, # boost checks this config value against the sum of: # - the amount of data downloaded from the host in the staging area # - the amount of data that is queued for download from the host # - the amount of data in the proposed deal # If the total amount would exceed the limit, boost rejects the deal. # Set this value to 0 to indicate there is no limit per host. # # type: uint64 # env var: LOTUS_DEALMAKING_MAXSTAGINGDEALSPERCENTPERHOST #MaxStagingDealsPercentPerHost = 0 sun-brightdesktopmoon --- # Setting up a monitoring stack for Boost | Boost [hashtag](https://boost.filecoin.io/monitoring/setting-up-a-monitoring-stack-for-boost#docs-internal-guid-09f21b17-7fff-fabd-2034-91b1f0465aa1) Background --------------------------------------------------------------------------------------------------------------------------------------------------------------- The monitoring stack we will use includes: * Prometheus - collects metrics and powers dashboards in Grafana * Tempo - collects traces and powers traces search in Grafana with Jaeger * Grafana - provides visualization tools and dashboards for all metrics and traces Lotus and Boost are already instrumented to produce traces and stats for Prometheus to collect. The Boost team also packages a set of Grafana dashboards that are automatically provisioned as part of this setup. circle-info The monitoring setup should be done on the same node where `boostd` and other Boost services are running. If you are running multiple Boost services (like `boostd-data`/`booster-http`) across multiple machines then any one of those machine can be used. [hashtag](https://boost.filecoin.io/monitoring/setting-up-a-monitoring-stack-for-boost#prerequisites) Prerequisites ------------------------------------------------------------------------------------------------------------------------ This setup has been tested on macOS and on Linux. We haven’t tested it on Windows, so YMMV. All the monitoring stack containers run in Docker. [hashtag](https://boost.filecoin.io/monitoring/setting-up-a-monitoring-stack-for-boost#steps) Steps -------------------------------------------------------------------------------------------------------- 1. **Install Docker** We have tested this setup with Docker 20.10.23 on macOS and Ubuntu. [https://docs.docker.com/engine/install/arrow-up-right](https://docs.docker.com/engine/install/) 1. **DNS resolution for Prometheus** Update `extra_hosts` in `docker-compose.yaml` for `prometheus`, so that the Prometheus container can reach all its targets - `boostd`, `lotus-miner`, `booster-bitswap`, `booster-http`, etc. [https://github.com/filecoin-project/boost/blob/main/docker/monitoring/docker-compose.yaml#L54-L58arrow-up-right](https://github.com/filecoin-project/boost/blob/main/docker/monitoring/docker-compose.yaml#L54-L58) Depending on where your Filecoin processes (`boostd`, `lotus`, `lotus-miner`, `booster-bitswap`, etc.) are running, you need to confirm that they are reachable from Prometheus so that it can scrape their metrics. By default the setup expects to find them within the same Docker network, so if you are running them elsewhere (i.e. on the \`host\` network), make sure to update the docker-compose file for the same. 1. **Loki plugin for Docker** The `loki` plugin for docker is required to allow collecting logs from the services running on docker itself [https://grafana.com/docs/loki/latest/send-data/docker-driver/arrow-up-right](https://grafana.com/docs/loki/latest/send-data/docker-driver/) 1. **Storage location** The prometheus and tempo services requires access to local storage to ensure all historical data is safe across the docker restarts. This path can be defined in the [env filearrow-up-right](https://github.com/filecoin-project/boost/blob/main/docker/monitoring/.env) . The default value is `$HOME/.boost-monitoring` Please ensure that the storage directory exists and has permission 0775 or 0777 depending on your user. 1. **Update metrics endpoints for Prometheus** In case, you are running all Boost services and monitoring stack on the same node, you can skip this step. If any of your Boost services like `boostd-data` or `booster-http` are running on a different host then we must modify the prometheus configuration to update the endpoint to be scraped. Please edit [https://github.com/filecoin-project/boost/blob/main/docker/monitoring/prometheus.yamlarrow-up-right](https://github.com/filecoin-project/boost/blob/main/docker/monitoring/prometheus.yaml) file and update the IP addresses and port to scrap the metrics. Example: 1. **Start the monitoring stack using docker-compose** [hashtag](https://boost.filecoin.io/monitoring/setting-up-a-monitoring-stack-for-boost#verify-prometheus-targets) Verify Prometheus Targets ------------------------------------------------------------------------------------------------------------------------------------------------ Confirm that Prometheus targets are scraped at [http://localhost:9190arrow-up-right](http://localhost:9190/) Targets circle-info You can also access the Grafana on the localhost interface on a remote server, you can open an SSH tunnel from your local machine: ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2Flh4.googleusercontent.com%2FsUGTMb0S5lp2sog0DdG9ecSRmCPDtMSmyAIwQ-k-91Pw1IUahhCiudKXkmKUJTHajXrPMR6RGLMmFEU9szwRlvcf9B4DgAIBuAaLTPB7-XaGPvtCrg2A9XGdhDIaTxZYi5wdbpTLdpsC98YcrorNVA2R1K7NLlK2xlk6RJRZ0jJ6qZ28QW543N0Imsrbzg&width=300&dpr=3&quality=100&sign=529b6595&sv=2) If you are running software firewall like \`ufw\`, you might need to modify your iptables and allow access from the Prometheus container / network to the Filecoin stack network, for example: `sudo docker network inspect monitoring # note the Subnet for the network sudo ufw allow from 172.18.0.0/16` [hashtag](https://boost.filecoin.io/monitoring/setting-up-a-monitoring-stack-for-boost#grafana-dashboards) Grafana dashboards ---------------------------------------------------------------------------------------------------------------------------------- Go to Grafana at [http://localhost:3333arrow-up-right](http://localhost:3333/) and inspect the dashboards. circle-info You can also access the Grafana on the localhost interface on a remote server, you can open an SSH tunnel from your local machine: circle-info The default username and password for Grafana dashboard is "admin" and "admin" respectively. ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2Flh4.googleusercontent.com%2FUat6-bsTHiMvzwm854YT2W9M7fN5uiqdvQ8oFs1WsWqn98716AO56VCWD8Pok98W019mrjZVRM6fFv2qZ_hY18OR0GmCMa5eVR45mh0OT8ZbX47HT3_lLhuEvNt3CRy9eJ-t2O0De2hf3UDOsYERCgSaDR_atH9czc1gkdUbo6dDC5ZYP75xaLJugAo8VQ&width=300&dpr=3&quality=100&sign=3c2be738&sv=2) [PreviousMonitoringchevron-left](https://boost.filecoin.io/monitoring) [NextStoring data on Filecoinchevron-right](https://boost.filecoin.io/storing-data-on-filecoin) Last updated 2 years ago * [Background](https://boost.filecoin.io/monitoring/setting-up-a-monitoring-stack-for-boost#docs-internal-guid-09f21b17-7fff-fabd-2034-91b1f0465aa1) * [Prerequisites](https://boost.filecoin.io/monitoring/setting-up-a-monitoring-stack-for-boost#prerequisites) * [Steps](https://boost.filecoin.io/monitoring/setting-up-a-monitoring-stack-for-boost#steps) * [Verify Prometheus Targets](https://boost.filecoin.io/monitoring/setting-up-a-monitoring-stack-for-boost#verify-prometheus-targets) * [Grafana dashboards](https://boost.filecoin.io/monitoring/setting-up-a-monitoring-stack-for-boost#grafana-dashboards) sun-brightdesktopmoon Copy mkdir ~/.boost-monitoring chmod 0777 ~/.boost-monitoring Copy - job_name: 'booster-bitswap' static_configs: - targets: [ ':' ] labels: host: 'local' Copy cd docker/monitoring docker-compose up -d Copy ssh -L 9190:localhost:9190 myserver Copy ssh -L 3333:localhost:3333 myserver sun-brightdesktopmoon --- # Backup and Restore | Boost [hashtag](https://boost.filecoin.io/backup-and-restore#backup) Backup -------------------------------------------------------------------------- Boost now supports both online and offline backups. The backup command will output a backup directory containing the following files. 1. `metadata` - contains backup of leveldb 2. `boostd.db` - backup of deals database 3. `keystore` - directory containing libp2p keys 4. `token` - API token 5. `config` - directory containing all config files and `config.toml` link 6. `storage.json` - file containing storage details Backup does not back up deal logs and Local Index Directory. ### [hashtag](https://boost.filecoin.io/backup-and-restore#online-backup) Online backup You can take an online backup with the below command Copy boostd backup The online backup supports running only one instance at a time and you might see a locking error if another instance of backup is already running. ### [hashtag](https://boost.filecoin.io/backup-and-restore#offline-backup) Offline backup 1. Shutdown `boostd` before taking a backup 2. Take a backup using the command line [hashtag](https://boost.filecoin.io/backup-and-restore#restore) Restore ---------------------------------------------------------------------------- 1. Make sure that `--boost-repo` flag is set if you wish to restore to a custom location. Otherwise, it will be restored to `~/.boost` directory 2. Restore the boost repo using the command line [PreviousBitswap retrievalchevron-left](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval) [NextTutorialschevron-right](https://boost.filecoin.io/tutorials) Last updated 2 years ago * [Backup](https://boost.filecoin.io/backup-and-restore#backup) * [Online backup](https://boost.filecoin.io/backup-and-restore#online-backup) * [Offline backup](https://boost.filecoin.io/backup-and-restore#offline-backup) * [Restore](https://boost.filecoin.io/backup-and-restore#restore) sun-brightdesktopmoon Copy boostd backup --offline Copy boostd restore sun-brightdesktopmoon --- # Need help? | Boost 1. You can report any issues or bugs [herearrow-up-right](https://github.com/filecoin-project/boost/issues/new?assignees=&labels=need%2Ftriage%2Ckind%2Fbug&template=bug_report.yml) . 2. If you are having trouble, check the [Troubleshooting](https://boost.filecoin.io/troubleshooting) page for common problems and solutions. 3. If you have a question or require support, please [open a support ticketarrow-up-right](https://github.com/filecoin-project/boost/issues/new?assignees=&labels=support%2Ctriage&projects=&template=support-ticket.yml&title=%5BSupport+Ticket%5D%3A+) or join the [Filecoin Slackarrow-up-right](https://filecoin.io/slack/) and ask for support in #boost-help. 4. You can also [start a discussionarrow-up-right](https://github.com/filecoin-project/boost/discussions/new?category=ideas) about new feature and improvement ideas for the Boost. [PreviousFAQchevron-left](https://boost.filecoin.io/faq) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Experimental Features | Boost Boost is developing new market features on a regular basis as part of the overall market development. This section covers the experimental features released by boost along with details on how to use them. It is **not** recommended to run experimental features in production environments. The features should be tested as per your requirements, and any issues or requests should be reported to the team via Github or Slack. Once the new features have been tested and vetted, they will be released as part of a stable Boost release and all documentation concerning those features will be moved to an appropriate section of this site. Current experimental features are listed below. [FVM Contract Dealschevron-right](https://boost.filecoin.io/experimental-features/fvm-contract-deals) [Direct Dealschevron-right](https://boost.filecoin.io/experimental-features/direct-deals) [Data Segment Indexingchevron-right](https://boost.filecoin.io/experimental-features/data-segment-indexing) [PreviousTroubleshootingchevron-left](https://boost.filecoin.io/troubleshooting) [NextFVM Contract Dealschevron-right](https://boost.filecoin.io/experimental-features/fvm-contract-deals) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # What is Boost? | v1.x (Deprecated) | Boost Boost is a tool for Storage Providers to manage data onboarding and retrieval on the Filecoin network. It replaces the `go-fil-markets` package in lotus with a standalone binary that runs alongside a Lotus daemon and Lotus miner. Boost exposes libp2p interfaces for making storage and retrieval deals, a web interface for managing storage deals, and a GraphQL interface for accessing and updating real-time deal information. ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F968384846-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F01F3nbbmdopOdEHfj4fF%252Fuploads%252FrS0XJtSZv6qfwqwiO2nb%252FBoost%2520-%2520storage%2520deals.png%3Falt%3Dmedia%26token%3Dcf8061b4-e252-4d7d-b2cc-d0fe7c893ee7&width=768&dpr=3&quality=100&sign=4cfc961&sv=2) Web UI - Storage Deals screen ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F968384846-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F01F3nbbmdopOdEHfj4fF%252Fuploads%252Ft9UrzvV2i3gdVaPXdgi2%252FBoost%2520-%2520storage%2520space.png%3Falt%3Dmedia%26token%3D3f617a69-c959-45a5-858e-906cf8430fe2&width=768&dpr=3&quality=100&sign=1dde21c5&sv=2) Web UI - Storage Space screen ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F968384846-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F01F3nbbmdopOdEHfj4fF%252Fuploads%252FJfxsrHf96NMc6WdDbfIj%252FBoost%2520-%2520sealing%2520pipeline.png%3Falt%3Dmedia%26token%3Dbed249c9-dd03-4ec8-a7f5-21e3cb80af0d&width=768&dpr=3&quality=100&sign=883bd6c0&sv=2) Web UI - Sealing Pipeline screen [NextFeatureschevron-right](https://boost.filecoin.io/v1/features) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # HTTP retrieval | Boost `booster-http` is a binary that can be run alongside the `boostd` process in order to serve retrievals over HTTP. This can be used to serve full _pieces_ and also block data in the form of CAR files or raw block bytes. This can be further extended using a [bifrost-gatewayarrow-up-right](https://github.com/ipfs/bifrost-gateway) server if a Storage Provider wishes to also serve plain files and directories. HTTP retrievals currently have no in-built payment or permissioning mechanism. Storage Providers may wish to front the `booster-http` server with a reverse proxy such as [NGINXarrow-up-right](https://nginx.org/en/docs/) to handle operational concerns like SSL, authentication and load balancing. [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/http-retrieval#version-history) Version History ---------------------------------------------------------------------------------------------------------------------- * `booster-http` was introduced with Boost v1.2.0, able to serve full pieces using the /piece/ endpoint. * In Boost v1.7.0 `booster-http` was extended to be able to serve CAR files, IPLD blocks and a full "trusted" gateway (files and directories). * In Boost v2.1.0 `booster-http` was simplified to only serve "trustless" data: full pieces, CAR files and IPLD blocks; deferring optional "trusted" mode to [bifrost-gatewayarrow-up-right](https://github.com/ipfs/bifrost-gateway) for those Storage Providers who wish to offer this service to their users. [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/http-retrieval#trustless-vs-trusted) Trustless vs Trusted -------------------------------------------------------------------------------------------------------------------------------- "**Trustless**" refers to a mode in which the client does not need to trust a data provider in order to have confidence in the integrity and authenticity of the data they are receiving. Instead, trust is achieved by content-addressing, by having the server deliver to the client everything they require in order to verify the data they are receiving. "**Trusted**" refers to a mode in which the client must trust the data provider to deliver the correct data. This is the standard mode that most web servers operate in, and is a common method of interacting with IPFS gateways (including `bifrost-gateway`). In this mode, the client must trust that the server is delivering the correct data, and that the server is not maliciously modifying the data in transit. Typically, the client does not receive enough information in order to verify the relationship between the content received and the content-address (CID) requested, hence the need to "trust" the server. Trustless retrievals are always preferable where there does not exist a trust relationship between parties. By serving data only in a trustless mode, Storage Providers can avoid accusations of data alteration, and clients can be confident that they are receiving the data they requested. It also reduces the risk of having an HTTP endpoint as being flagged for serving illicit content. Learn more at [Gateways and gatekeepersarrow-up-right](https://blog.nft.storage/posts/2022-04-29-gateways-and-gatekeepers) . However, there are many circumstances in which a Storage Provider may wish to offer a trusted retrieval service, particularly due to convenience and interoperability with standard HTTP clients. ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/http-retrieval#piece-retrievals) Piece retrievals When performing certain actions, such as replicating deals, it can be convenient to retrieve the entire Piece (with padding) to ensure CommP integrity. When piece retrievals are enabled with `booster-http` with the `--serve-pieces` flag (enabled by default), the `/piece/` endpoint is exposed. Copy curl 'http://{booster-http exposed url}/piece/bagaSomePieceCID' -o bagaSomePieceCID.piece ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/http-retrieval#payload-retrievals-car-and-raw) Payload retrievals (CAR and raw) `booster-http` supports the [IPFS Trustless Gatewayarrow-up-right](https://specs.ipfs.tech/http-gateways/trustless-gateway/) specification for both CAR and raw IPLD data contained within deal payloads. When `booster-http` is run with the `--serve-cars` flag (enabled by default), the `/ipfs/` endpoint is exposed. To download a CAR file containing all the blocks linked under a given root CID, you can either pass an `Accept: application/vnd.ipld.car` header, or a `?format=car` query parameter to the `/ipfs/` endpoint. Likewise, to download raw single IPLD block bytes, you can either pass an `Accept: application/vnd.ipld.raw` header, or a `?format=raw` query parameter to the `/ipfs/` endpoint. The `/ipfs/` endpoint must be followed by a CID and one of the indicators above, for example: Alternatively, using `Accept` header: It is also possible to use [Lassiearrow-up-right](https://github.com/filecoin-project/lassie) , a Trustless retrieval client, to perform and verify retrievals from `booster-http`: See the [IPFS Trustless Gatewayarrow-up-right](https://specs.ipfs.tech/http-gateways/trustless-gateway/) specification for more information on the ranges of queries possible with this protocol; most of which are implemented by Lassie as a client. ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/http-retrieval#trusted-plain-file-directory-retrievals) Trusted (plain file/directory) retrievals For Storage Providers that wish to offer trusted access to payload data, in the same way that an IPFS Gateway bridges IPLD data to standard browser-accessible files and directories, [bifrost-gatewayarrow-up-right](https://github.com/ipfs/bifrost-gateway) can be used to translate `booster-http`'s trustless data into trusted data. This is a separate process that must be configured to communicate with `booster-http`. See [Trusted HTTP Gateway Setup](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#trusted-http-gateway-setup) for full setup instructions. [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/http-retrieval#local-setup) Local Setup -------------------------------------------------------------------------------------------------------------- SPs should try a local setup and test their HTTP retrievals before proceeding to run `booster-http` in production. The `booster-http` binary is built and installed with `boostd` binary. If you are planning to run `booster-http` on a different node, you can build and install the new binary. Otherwise, skip to step 3. 1. Clone the Boost repo and checkout the same release as your `boostd` version 1. Build and install the new binary 1. Collect the token information for lotus-miner and lotus daemon API 1. Start the `booster-http` server with the above details circle-info You can run multiple `booster-http` processes on the same machine by using a different port for each instance with the `--port` flag. You can also run multiple instances of the `booster-http` on different machines. [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/http-retrieval#running-public-boost-http-retrieval) Running Public Boost HTTP Retrieval -------------------------------------------------------------------------------------------------------------------------------------------------------------- By default, the `booster-http` server listens to `0.0.0.0` port `7777`. This can be changed with `--address` and `--port`; for example, you may wish to change the `--address` to `127.0.0.1` to restrict it to localhost-only. When exposing `booster-http` to a public address for general retrievals, it is recommended that you run an HTTP reverse proxy such as [NGINXarrow-up-right](https://nginx.org/en/docs/) to handle operational concerns such as: * SSL * Authentication and authorization * Compression * Load balancing * Logging `booster-http` can provide some of this functionality itself, but dedicated reverse proxy software is recommended for scalable production deployments. See [HTTP Reverse Proxy Setup](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#reverse-proxy-setup) for more information and example configuration. ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/http-retrieval#making-http-retrieval-discoverable) Making HTTP Retrieval Discoverable To enable public discovery of the Boost HTTP server, Storage Providers should set the domain root in boostd's `config.toml`. Under the `[DealMaking]` section, set `HTTPRetrievalMultiaddr` to the public domain root in multi-address format. Example `config.toml` section: Clients can determine if a Storage Provider offers HTTP retrieval by running: Clients can also check the HTTP URL scheme version and supported queries [PreviousRetrieving data from Filecoinchevron-left](https://boost.filecoin.io/retrieving-data-from-filecoin) [NextAdvanced Configuration of booster-httpchevron-right](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration) Last updated 1 year ago * [Version History](https://boost.filecoin.io/retrieving-data-from-filecoin/http-retrieval#version-history) * [Trustless vs Trusted](https://boost.filecoin.io/retrieving-data-from-filecoin/http-retrieval#trustless-vs-trusted) * [Piece retrievals](https://boost.filecoin.io/retrieving-data-from-filecoin/http-retrieval#piece-retrievals) * [Payload retrievals (CAR and raw)](https://boost.filecoin.io/retrieving-data-from-filecoin/http-retrieval#payload-retrievals-car-and-raw) * [Trusted (plain file/directory) retrievals](https://boost.filecoin.io/retrieving-data-from-filecoin/http-retrieval#trusted-plain-file-directory-retrievals) * [Local Setup](https://boost.filecoin.io/retrieving-data-from-filecoin/http-retrieval#local-setup) * [Running Public Boost HTTP Retrieval](https://boost.filecoin.io/retrieving-data-from-filecoin/http-retrieval#running-public-boost-http-retrieval) * [Making HTTP Retrieval Discoverable](https://boost.filecoin.io/retrieving-data-from-filecoin/http-retrieval#making-http-retrieval-discoverable) sun-brightdesktopmoon Copy # Fetch a complete DAG curl 'http://{booster-http exposed url}/ipfs/bafySomePayloadCID?format=car' -o bafySomePayloadCID.car # Fetch an individual block's bytes curl 'http://{booster-http exposed url}/ipfs/bafySomeBlockCID?format=raw' -o bafySomeBlockCID.block Copy # Fetch a complete DAG curl -H 'Accept: application/vnd.ipld.car' 'http://{booster-http exposed url}/ipfs/bafySomePayloadCID' -o bafySomePayloadCID.car # Fetch an individual block's bytes curl -H 'Accept: application/vnd.ipld.car' 'http://{booster-http exposed url}/ipfs/bafySomeBlockCID' -o bafySomeBlockCID.block Copy lassie fetch --provider 'http://{booster-http exposed url}' bafySomePayloadCID Copy curl 'http://{bifrost-gateway exposed url}/ipfs/{content ID}/{optional path to resource}' -o myimage.png Copy git clone https://github.com/filecoin-project/boost.git cd boost git checkout Copy make build sudo make build Copy export ENV_FULLNODE_API_INFO=`lotus auth api-info --perm=admin` export FULLNODE_API_INFO=`echo $ENV_FULLNODE_API_INFO | awk '{split($0,a,"="); print a[2]}'` Copy export ENV_MINER_API_INFO=`lotus-miner auth api-info --perm=admin` export MINER_API_INFO=`echo $ENV_MINER_API_INFO | awk '{split($0,a,"="); print a[2]}'` Copy booster-http run --api-lid="ws://:8044" --api-fullnode=$FULLNODE_API_INFO --api-storage=$MINER_API_INFO Copy [DealMaking] HTTPRetrievalMultiaddr = "/dns/foo.com/tcp/443/https" Copy boost provider retrieval-transports Copy # Version curl 'http://{bifrost-gateway exposed url}/info' # Supported queries curl 'http://{bifrost-gateway exposed url}/' sun-brightdesktopmoon --- # FAQ | Boost **Is there a way to stop** `**boostd**` **daemon?** You can use the regular Unix OS signals **Is Boost compatible with the Lotus client? Can a client use** `**lotus client deal**` **to send a deal to Boost storage providers or do they have to use the boost client?** No. Boost no longer supports deals from lotus client. Boost will work with any client if the client uses deal making protocol `/fil/storage/mk/1.2.0` or newer. **Does Boost provide retrieval functionality?** Yes, Boost provides 3 protocols for retrievals as of now. By default, Boost has Graphsync retrieval enabled. SPs can run Bitswap and HTTP retrievals by running [`booster-bitswap`](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval) and [`booster-http`](https://boost.filecoin.io/retrieving-data-from-filecoin/http-retrieval) respectively. **Does Boost client have retrieval functionality?** Yes, Boost client supports retrieval over graphsync protocol. But we highly recommend, using [`Lassie`arrow-up-right](https://docs.filecoin.io/basics/how-retrieval-works/basic-retrieval/#lassie) client for Filecoin/IPFS retrievals. **Can Boost make verified deals?** Yes, payments for deals can be made either from a regular wallet, or from DataCap. Deals that are paid for with DataCap are called `verified` deals. **Can I run both Boost and markets at the same time?** No, Boost replaces the legacy markets process. [PreviousData Segment Indexingchevron-left](https://boost.filecoin.io/experimental-features/data-segment-indexing) [NextNeed help?chevron-right](https://boost.filecoin.io/need-help) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Legacy Deal configuration | Boost ### [hashtag](https://boost.filecoin.io/configuration/legacy-deal-configuration#dealmaking-section) Dealmaking section This section controls parameters for making storage and retrieval deals: Copy [LotusDealmaking] # When enabled, the miner can accept online deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDERONLINESTORAGEDEALS #ConsiderOnlineStorageDeals = true # When enabled, the miner can accept offline deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDEROFFLINESTORAGEDEALS #ConsiderOfflineStorageDeals = true # When enabled, the miner can accept retrieval deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDERONLINERETRIEVALDEALS #ConsiderOnlineRetrievalDeals = true # When enabled, the miner can accept offline retrieval deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDEROFFLINERETRIEVALDEALS #ConsiderOfflineRetrievalDeals = true # When enabled, the miner can accept verified deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDERVERIFIEDSTORAGEDEALS #ConsiderVerifiedStorageDeals = true # When enabled, the miner can accept unverified deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDERUNVERIFIEDSTORAGEDEALS #ConsiderUnverifiedStorageDeals = true # A list of Data CIDs to reject when making deals # # type: []cid.Cid # env var: LOTUS_LOTUSDEALMAKING_PIECECIDBLOCKLIST #PieceCidBlocklist = [] # Maximum expected amount of time getting the deal into a sealed sector will take # This includes the time the deal will need to get transferred and published # before being assigned to a sector # # type: Duration # env var: LOTUS_LOTUSDEALMAKING_EXPECTEDSEALDURATION #ExpectedSealDuration = "24h0m0s" # Maximum amount of time proposed deal StartEpoch can be in future # # type: Duration # env var: LOTUS_LOTUSDEALMAKING_MAXDEALSTARTDELAY #MaxDealStartDelay = "336h0m0s" # When a deal is ready to publish, the amount of time to wait for more # deals to be ready to publish before publishing them all as a batch # # type: Duration # env var: LOTUS_LOTUSDEALMAKING_PUBLISHMSGPERIOD # PublishMsgPeriod = "40m0s" # The maximum number of deals to include in a single PublishStorageDeals # message # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_MAXDEALSPERPUBLISHMSG #MaxDealsPerPublishMsg = 8 # The maximum collateral that the provider will put up against a deal, # as a multiplier of the minimum collateral bound # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_MAXPROVIDERCOLLATERALMULTIPLIER #MaxProviderCollateralMultiplier = 2 # The maximum allowed disk usage size in bytes of staging deals not yet # passed to the sealing node by the markets service. 0 is unlimited. # # type: int64 # env var: LOTUS_LOTUSDEALMAKING_MAXSTAGINGDEALSBYTES # MaxStagingDealsBytes = 100000000000 # The maximum number of parallel online data transfers for storage deals # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_SIMULTANEOUSTRANSFERSFORSTORAGE #SimultaneousTransfersForStorage = 20 # The maximum number of simultaneous data transfers from any single client # for storage deals. # Unset by default (0), and values higher than SimultaneousTransfersForStorage # will have no effect; i.e. the total number of simultaneous data transfers # across all storage clients is bound by SimultaneousTransfersForStorage # regardless of this number. # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_SIMULTANEOUSTRANSFERSFORSTORAGEPERCLIENT #SimultaneousTransfersForStoragePerClient = 0 # The maximum number of parallel online data transfers for retrieval deals # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_SIMULTANEOUSTRANSFERSFORRETRIEVAL #SimultaneousTransfersForRetrieval = 20 # Minimum start epoch buffer to give time for sealing of sector with deal. # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_STARTEPOCHSEALINGBUFFER #StartEpochSealingBuffer = 480 # A command used for fine-grained evaluation of storage deals # see https://lotus.filecoin.io/storage-providers/advanced-configurations/market/#using-filters-for-fine-grained-storage-and-retrieval-deal-acceptance for more details # # type: string # env var: LOTUS_LOTUSDEALMAKING_FILTER #Filter = "" # A command used for fine-grained evaluation of retrieval deals # see https://lotus.filecoin.io/storage-providers/advanced-configurations/market/#using-filters-for-fine-grained-storage-and-retrieval-deal-acceptance for more details # # type: string # env var: LOTUS_LOTUSDEALMAKING_RETRIEVALFILTER #RetrievalFilter = "" [LotusDealmaking.RetrievalPricing] # env var: LOTUS_LOTUSDEALMAKING_RETRIEVALPRICING_STRATEGY #Strategy = "default" [LotusDealmaking.RetrievalPricing.Default] # env var: LOTUS_LOTUSDEALMAKING_RETRIEVALPRICING_DEFAULT_VERIFIEDDEALSFREETRANSFER #VerifiedDealsFreeTransfer = true [LotusDealmaking.RetrievalPricing.External] # env var: LOTUS_LOTUSDEALMAKING_RETRIEVALPRICING_EXTERNAL_PATH #Path = "" `ExpectedSealDuration` is an estimate of how long sealing will take and is used to reject deals whose start epoch might be earlier than the expected completion of sealing. It can be estimated by [benchmarkingarrow-up-right](https://lotus.filecoin.io/storage-providers/operate/benchmarks/) or by [pledging a sectorarrow-up-right](https://lotus.filecoin.io/storage-providers/operate/sector-pledging/) . circle-exclamation The final value of `ExpectedSealDuration` should equal `(TIME_TO_SEAL_A_SECTOR + WaitDealsDelay) * 1.5`. This equation ensures that the miner does not commit to having the sector sealed too soon `StartEpochSealingBuffer` allows `lotus-miner` to seal a sector before a certain epoch. For example: if the current epoch is 1000 and a deal within a sector must start on epoch 1500, then `lotus-miner` must wait until the current epoch is 1500 before it can start sealing that sector. However, if Boost sets `StartEpochSealingBuffer` to 500, the `lotus-miner` can start sealing the sector at epoch 1000. If there are multiple deals in a sector, the deal with a start time closest to the current epoch is what `StartEpochSealingBuffer` will be based off. So, if the sector in our example has three deals that start on epoch 1000, 1200, and 1400, then `lotus-miner` will start sealing the sector at epoch 500. #### [hashtag](https://boost.filecoin.io/configuration/legacy-deal-configuration#publishing-several-deals-in-one-message) Publishing several deals in one message The `PublishStorageDeals` message can publish multiple deals in a single message. When a deal is ready to be published, Boost will wait up to `PublishMsgPeriod` for other deals to be ready before sending the `PublishStorageDeals` message. However, once `MaxDealsPerPublishMsg` is ready, Boost will immediately publish all the deals. For example, if `PublishMsgPeriod` is 1 hour: * At 1:00 pm, deal 1 is ready to publish. Boost will wait until 2:00 pm for other deals to be ready before sending `PublishStorageDeals`. * At 1:30 pm, Deal 2 is ready to publish * At 1:45 pm, Deal 3 is ready to publish * At 2:00pm, Boost publishes Deals 1, 2, and 3 in a single `PublishStorageDeals` message. If `MaxDealsPerPublishMsg` is 2, then in the above example, when deal 2 is ready to be published at 1:30, Boost would immediately publish Deals 1 & 2 in a single `PublishStorageDeals` message. Deal 3 would be published in a subsequent `PublishStorageDeals` message. triangle-exclamation If any of the deals in the `PublishStorageDeals` fails validation upon execution, or if the start epoch has passed, all deals will fail to be published ### [hashtag](https://boost.filecoin.io/configuration/legacy-deal-configuration#using-filters-for-fine-grained-storage-and-retrieval-deal-acceptance) Using filters for fine-grained storage and retrieval deal acceptance Your use case might demand very precise and dynamic control over a combination of deal parameters. Boost provides two IPC hooks allowing you to name a command to execute for every deal before the miner accepts it: * `Filter` for storage deals. * `RetrievalFilter` for retrieval deals. The executed command receives a JSON representation of the deal parameters on standard input, and upon completion, its exit code is interpreted as: * `0`: success, proceed with the deal. * `non-0`: failure, reject the deal. The most trivial filter rejecting any retrieval deal would be something like: `RetrievalFilter = "/bin/false"`. `/bin/false` is binary that immediately exits with a code of `1`. [This Perl scriptarrow-up-right](https://gist.github.com/ribasushi/53b7383aeb6e6f9b030210f4d64351d5/9bd6e898f94d20b50e7c7586dc8b8f3a45dab07c#file-dealfilter-pl) lets the miner deny specific clients and only accept deals that are set to start relatively soon. You can also use a third party content policy framework like [CIDgravityarrow-up-right](https://www.cidgravity.com/) or `bitscreen` by Murmuration Labs: [PreviousRemote CommPchevron-left](https://boost.filecoin.io/configuration/remote-commp) [NextHTTP indexer announcementchevron-right](https://boost.filecoin.io/configuration/http-indexer-announcement) Last updated 2 years ago * [Dealmaking section](https://boost.filecoin.io/configuration/legacy-deal-configuration#dealmaking-section) * [Using filters for fine-grained storage and retrieval deal acceptance](https://boost.filecoin.io/configuration/legacy-deal-configuration#using-filters-for-fine-grained-storage-and-retrieval-deal-acceptance) sun-brightdesktopmoon Copy # grab filter program go get -u -v github.com/Murmuration-Labs/bitscreen # add it to both filters Filter = "/path/to/go/bin/bitscreen" RetrievalFilter = "/path/to/go/bin/bitscreen" sun-brightdesktopmoon --- # FVM Contract Deals | Boost With the release of FVM, it is now possible for smart contracts to make deal proposals on-chain. This is made possible though the DealProposal FRC. DataDAOs, as well as other clients who want to store data on Filecoin, can now deploy a smart contract on the FVM which adheres to the DealProposal FRC, and make deal proposals that are visible to every storage provider who monitors the chain. [hashtag](https://boost.filecoin.io/experimental-features/fvm-contract-deals#how-to-enable-fvm-monitoring-in-order-to-process-storage-deal-proposals-published-on-chain) How to enable FVM monitoring in order to process storage deal proposals published on-chain? ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Boost already has support for the DealProposal FRC. The code for FVM monitoring resides in the latest release of the Boost. It should be used with caution for production use. SPs must [enable FEVM on lotus daemonarrow-up-right](https://lotus.filecoin.io/lotus/configure/ethereum-rpc/) before proceeding to the next step. In order to enable contract deals, you have to edit your `config.toml` and enable chain monitoring. By default it is disabled. Here is an example configuration: Copy [ContractDeals] Enabled = true AllowlistContracts = [] From = "0x0000000000000000000000000000000000000000" `AllowlistContracts` field could be left empty if you want to accept deals from any client. If you only want to accept deals from some clients, you can specify their contract addresses in the field. `From` field should be set to your SP's FEVM address. Some clients may implement a whitelist which allows specific SPs to accept deal proposals from their contract. This field will help those clients identify your SP and match it to their whitelist. ### [hashtag](https://boost.filecoin.io/experimental-features/fvm-contract-deals#how-contract-deals-work-in-boost) How contract deals work in Boost 1. A contract publishes a `DealProposalCreate` event on the chain. 2. Boost monitors the chain for such events from all the clients by default. When such an event is detected, we go and fetch the data for the deal. 3. Deal is then run through the basic deal validation filters like clients has enough funds, SP has enough funds etc. 4. Once deal passes the validation, we create a new deal handler in Boost and pass this deal for execution like other Boost deals. [PreviousExperimental Featureschevron-left](https://boost.filecoin.io/experimental-features) [NextDirect Dealschevron-right](https://boost.filecoin.io/experimental-features/direct-deals) Last updated 2 years ago * [How to enable FVM monitoring in order to process storage deal proposals published on-chain?](https://boost.filecoin.io/experimental-features/fvm-contract-deals#how-to-enable-fvm-monitoring-in-order-to-process-storage-deal-proposals-published-on-chain) * [How contract deals work in Boost](https://boost.filecoin.io/experimental-features/fvm-contract-deals#how-contract-deals-work-in-boost) sun-brightdesktopmoon sun-brightdesktopmoon --- # Tutorials | Boost [How to upgrade from Boost v1 to Boost v2chevron-right](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2) [How to re-index unsealed pieces that are flagged by LID in Boost v2chevron-right](https://boost.filecoin.io/tutorials/how-to-re-index-unsealed-pieces-that-are-flagged-by-lid-in-boost-v2) [How to upgrade from v2.0.0 to v2.1.0chevron-right](https://boost.filecoin.io/tutorials/how-to-upgrade-from-v2.0.0-to-v2.1.0) [How to upgrade from v2.1.x to v2.2.0chevron-right](https://boost.filecoin.io/tutorials/how-to-upgrade-from-v2.1.x-to-v2.2.0) [Start and stop Boost processeschevron-right](https://boost.filecoin.io/tutorials/start-and-stop-boost-processes) [How to store files with Boost on Filecoinchevron-right](https://boost.filecoin.io/tutorials/how-to-store-files-with-boost-on-filecoin) [Using filters for storage and retrieval dealschevron-right](https://boost.filecoin.io/tutorials/using-filters-for-storage-and-retrieval-deals) [Migrate from Lotus to Boostchevron-right](https://boost.filecoin.io/tutorials/upgrade-from-lotus-to-boost) [How to onboard data using DDO dealschevron-right](https://boost.filecoin.io/tutorials/how-to-onboard-data-using-ddo-deals) [PreviousBackup and Restorechevron-left](https://boost.filecoin.io/backup-and-restore) [NextHow to upgrade from Boost v1 to Boost v2chevron-right](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2) Last updated 1 month ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Features | v1.x (Deprecated) | Boost [hashtag](https://boost.filecoin.io/v1/features#make-storage-deals-with-http-data-transfer) Make storage deals with HTTP data transfer ------------------------------------------------------------------------------------------------------------------------------------------- Boost supports multiple options for data transfer when making storage deals, including HTTP. Clients can host their CAR file on an HTTP server, such as S3, and provide that URL when proposing the storage deal. Once accepted, Boost will automatically fetch the CAR file from the specified URL. See [As a client](https://boost.filecoin.io/v1/getting-started/boost-client) for more details. [hashtag](https://boost.filecoin.io/v1/features#web-ui) Web UI ------------------------------------------------------------------- Boost comes with a web interface that can be used to manage deals, watch disk usage, monitor funds, adjust settings and more. ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F968384846-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F01F3nbbmdopOdEHfj4fF%252Fuploads%252FiIIvOGSVFslO9lR8NBD8%252FScreen%2520Shot%25202022-03-23%2520at%252010.12.26%2520AM.png%3Falt%3Dmedia%26token%3D43e6fc9d-a9be-480b-9cc0-2c9b41ffc361&width=768&dpr=3&quality=100&sign=c51c6224&sv=2) Boost Web UI [hashtag](https://boost.filecoin.io/v1/features#backwards-compatibility-with-go-fil-markets-package) Backwards compatibility with go-fil-markets package ------------------------------------------------------------------------------------------------------------------------------------------------------------- Boost supports the same endpoints as `go-fil-markets` package for making storage and retrieval deals, getting the storage and retrieval ask, and getting the status of ongoing deals. This ensures that clients running lotus can make deals with Storage Providers running boost. [hashtag](https://boost.filecoin.io/v1/features#a-client-for-proposing-deals-that-doesnt-require-a-lotus-node) A client for proposing deals that doesn't require a Lotus node ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Boost comes with a client that can be used to make storage deals, and can be configured to point at a public Filecoin API endpoint. That means clients don't need to run a Filecoin node or sync from chain. See [As a client](https://boost.filecoin.io/v1/getting-started/boost-client) for details. [PreviousWhat is Boost?chevron-left](https://boost.filecoin.io/v1) [NextArchitecturechevron-right](https://boost.filecoin.io/v1/boost-architecture) Last updated 2 years ago * [Make storage deals with HTTP data transfer](https://boost.filecoin.io/v1/features#make-storage-deals-with-http-data-transfer) * [Web UI](https://boost.filecoin.io/v1/features#web-ui) * [Backwards compatibility with go-fil-markets package](https://boost.filecoin.io/v1/features#backwards-compatibility-with-go-fil-markets-package) * [A client for proposing deals that doesn't require a Lotus node](https://boost.filecoin.io/v1/features#a-client-for-proposing-deals-that-doesnt-require-a-lotus-node) sun-brightdesktopmoon sun-brightdesktopmoon --- # Data Segment Indexing | Boost A large majority of users onboard data onto the Filecoin network via an Aggregator, a third party combining small pieces of data into a singular large deal. Today the work done by aggregators is unverifiable and unprovable. The user relies on the Aggregator to perform the work correctly and at the same time, it is impossible to prove to a third party that a given piece of data was included in a deal which is a highly requested functionality for user-programmable data use cases. [FRC 58arrow-up-right](https://github.com/filecoin-project/FIPs/blob/master/FRCs/frc-0058.md) enables the data aggregators to produce a Proof of Data Segment Inclusion certifying proper aggregation of Client's data. The produced proof assures: * an inclusion of Client's data within the on-chain deal * the Client's data can be trivially discovered within the deal to enable retrieval * malicious behaviour of an Aggregator or another user, whose data was aggregated, does not interfere with retrievability of Client's data ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252FfvSONvM93naEeyBMEwT3%252FScreenshot%25202023-10-18%2520at%252012.25.40%2520PM.png%3Falt%3Dmedia%26token%3Da39316e9-8d61-4859-b45f-f3f23f27cfd5&width=768&dpr=3&quality=100&sign=4696cdb7&sv=2) A Filecoin piece containing aggregated deals This is a critical link in enabling and exposing small pieces of data to the FEVM ecosystem. In the majority of cases, small pieces of data undergo an aggregation process, combing them into a large deal for acceptance by a Storage Provider. Without the proposed proof, data within aggregated deals becomes a second class citizen in Filecoin ecosystem. A significant portion of the F(E)VM use-case is enabling the ability to process and reason about the data stored by Filecoin Storage Providers. The Proof of Data Segment Inclusion allows to apply this new capability on segments of data which are too small to be on-boarded in their own deals due to economic constraints. ### [hashtag](https://boost.filecoin.io/experimental-features/data-segment-indexing#how-to-enable-data-segment-indexing-on-storage-provider) How to enable Data Segment Indexing on Storage Provider? After upgrading to Boost v2.1.0-rc1, users can build `boostd` using the branch `feat/noncar-files`. Once the new binary is used to start the `boostd` process, the feature is automatically enabled on the storage provider side. ### [hashtag](https://boost.filecoin.io/experimental-features/data-segment-indexing#retrievals-for-aggregated-car-deals) Retrievals for aggregated car deals The attached index at the end of the aggregated cars allow Boost to index the aggregated deals correctly. Once the deals are indexed, client can retrieve any payload CIDs from that deal using one of the 3 available data transfer protocols. ### [hashtag](https://boost.filecoin.io/experimental-features/data-segment-indexing#how-to-create-an-aggregated-file) How to create an aggregated file Client can use the [mkpiece utilityarrow-up-right](https://github.com/willscott/mkpiece) to generate an aggregated car file for the deal making. The utility takes multiple car files and generates the resulting aggregated file on the standard output. circle-info Please note that each car file is padded to the nearest 2^n bytes. So, the resultant aggregated file can be much larger than the original car files. Example: car1 - 4.5 GiB - Padded to 8 GiB car2 - 10 GiB - Padded to 16 GiB car3 - 5 GiB - Padded to 8 GiB Total car size = 4.5+10+5 = 19.5 GiB Aggregated car size = 8+16+8 = 32 GiB This aggregated file can be used to generate the piece CID and size for a Boost deal. [PreviousDirect Dealschevron-left](https://boost.filecoin.io/experimental-features/direct-deals) [NextFAQchevron-right](https://boost.filecoin.io/faq) Last updated 2 years ago * [How to enable Data Segment Indexing on Storage Provider?](https://boost.filecoin.io/experimental-features/data-segment-indexing#how-to-enable-data-segment-indexing-on-storage-provider) * [Retrievals for aggregated car deals](https://boost.filecoin.io/experimental-features/data-segment-indexing#retrievals-for-aggregated-car-deals) * [How to create an aggregated file](https://boost.filecoin.io/experimental-features/data-segment-indexing#how-to-create-an-aggregated-file) sun-brightdesktopmoon Copy mkpiece a.car b.car c.car ... > out.dat Copy boostx commp out.dat sun-brightdesktopmoon --- # Bitswap retrieval | Boost `booster-bitswap` is a binary that runs alongside the `boostd` process, to serve retrievals over the Bitswap protocol. This feature of boost provides a number of tools for managing a production grade Bitswap retrieval service for a Storage Provider's content. circle-info There is currently no payment method in booster-bitswap. This endpoint is intended to serve free content. [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#why-enable-retrievals-via-bitswap) Why enable retrievals via Bitswap? -------------------------------------------------------------------------------------------------------------------------------------------------------------- Bitswap retrieval introduces interoperability between IPFS and Filecoin, as it enables clients to retrieve Filecoin data over IPFS. This expands the reach of the Filecoin network considerably, increasing the value proposition for users to store data on the Filecoin network. This benefits the whole community, including SPs. Users will be able to access data directly via IPFS, as well as benefit from retrieval markets (e.g. Saturn) and compute over data projects (e.g. Bacalhau). [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#booster-bitswap-modes) `booster-bitswap` modes --------------------------------------------------------------------------------------------------------------------------------------- There are two primary "modes" for exposing `booster-bitswap` to the internet. 1. In `private mode` the `booster-bitswap` peer ID is not publicly accessible to the internet. Instead, public Bitswap traffic goes to `boostd` itself, which then acts as a reverse proxy, forwarding that traffic on to `booster-bitswap`. This is similar to the way one might configure Nginx as a reverse proxy for an otherwise private web server. `private mode` is simpler to setup but may produce greater load on `boostd` as a protocol proxy. 2. In `public mode` the public internet firewall must be configured to forward traffic directly to the `booster-bitswap` instance. `boostd` is configured to announce the public address of `booster-bitswap` to the network indexer (the network indexer is the service that clients can query to discover where to retrieve content). This mode offers greater flexibility and performance. You can even setup `booster-bitswap` to run over a separate internet connection from `boostd`. However, it might require additional configuration and changes to your overall network infrastructure. ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252FPs0hzcxFhvacpaJLN7xU%252Fbitswap.png%3Falt%3Dmedia%26token%3D22ca8f57-c37a-4d4e-a383-0ddf6070a22d&width=768&dpr=3&quality=100&sign=73901afe&sv=2) Bitswap architecture and possible configurations ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#demo-configuration) Demo configuration You can configure booster-bitswap in the demo mode and familiarise yourself with the configuration. Once you are confident and familiar with the options, please go ahead and configure `booster-bitswap` for [production use](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#setup-booster-bitswap-to-serve-retrievals-publicly) . The `booster-bitswap` binary is built and installed with `boostd` binary. If you are planning to run `booster-bitswap` on a different node, you can build and install the new binary. Otherwise, skip to step 3. 1\. Clone the Boost repo and checkout the same release as your `boostd` version 2\. Build and install the `booster-bitswap` binary: 3\. Initialize `booster-bitswap`: 4\. Record the peer ID output by `booster-bitswap init` -- we will need this peer id later 5\. Run booster-bitswap 6\. By default, booster-bitswap runs on port 8888. You can use `--port` to override this behaviour 7\. Fetching over bitswap by running Where `peerID` is the peer id recorded when you ran `booster-bitswap init` and `rootCID` is the CID of a data CID known to be stored on your SP. [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#setup-booster-bitswap-to-serve-retrievals) Setup `booster-bitswap` to serve retrievals ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- As described above, `booster-bitswap` can be configured to serve the retrievals in 2 modes. We recommend using `public mode` to avoid greater load on `boostd` as a protocol proxy. ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#private-mode) Private Mode The `booster-bitswap` binary is built and installed with `boostd` binary. If you are planning to run `booster-bitswap` on a different node, you can build and install the new binary. Otherwise, skip to step 3. 1\. Clone the Boost repo and checkout the same release as your `boostd` version 2\. Build and install the `booster-bitswap` binary: 3\. Initialize `booster-bitswap`: 4\. Record the peer ID output by `booster-bitswap init` -- we will need this peer id later 5\. Stop `boostd` and edit ~/.boost/config.toml to set the peer ID for bitswap 6\. Start `boostd` service again 7\. Collect the token information for lotus-miner and lotus daemon API 8\. Run booster-bitswap circle-info You can get a `boostd` multiaddress by running `boostd net listen` and using any of the returned addresses 9\. By default, booster-bitswap runs on port 8888. You can use `--port` to override this behaviour 10\. Try to fetch a payload CID over bitswap to verify your configuration ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#public-mode) Public mode The `booster-bitswap` binary is built and installed with `boostd` binary. If you are planning to run `booster-bitswap` on a different node, you can build and install the new binary. Otherwise, skip to step 3. 1\. Clone the Boost repo and checkout the same release as your `boostd` version 2\. Build and install the `booster-bitswap` binary: 3\. Initialize `booster-bitswap`: 4\. Record the peer ID output by `booster-bitswap init` -- we will need this peer id later 5\. Stop `boostd` and edit ~/.boost/config.toml to set the peer ID for bitswap circle-info The libp2p private key file for booster-bitswap can generally be found at /libp2p.key The reason boost needs to know the public multiaddresses and libp2p private key for `booster-bitswap` is so it can properly announce these records to the network indexer. 6\. Start `boostd` service again 7\. Collect the token information for lotus-miner and lotus daemon API 8\. Run booster-bitswap 9\. By default, booster-bitswap runs on port 8888. You can use `--port` to override this behaviour 10\. Try to fetch a payload CID over bitswap to verify your configuration [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#booster-bitswap-configuration) `booster-bitswap` configuration ------------------------------------------------------------------------------------------------------------------------------------------------------- `booster-bitswap` provides a number of performance and safety tools for managing a production grade bitswap server without overloading your infrastructure. ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#bitswap-server-performance) Bitswap server performance Depending on your hardware you may wish to increase or decrease the default parameters for the bitswap server internals. In the following example we are increasing the worker count for various components up to 600. This will utilize more CPU and I/O, but improve the performance of retrievals. See the command line help docs for details on each parameter. ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#badbits-filtering) BadBits filtering Booster-bitswap is automatically setup to deny all requests for CIDs that are on the BadBits Denylist. The default badbits list can be override or addition badbits list can be provided to the `booster-bitswap` instance. #### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#to-override-the-default-badbits-list) To override the default badbits list #### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#to-provide-additional-badbits-list) To provide additional badbits list ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#request-filtering) Request filtering `booster-bitswap` provides a number of controls for filtering requests and limiting resource usage. These are expressed in a JSON configuration file `/retrievalconfig.json` circle-info You can create a new `retrievalconfig.json` file if one does not exists To make changes to the current configuration, you need to edit the `retrievalconfig.json` file and restart `booster-bitswap` for the changes to take affect. All configs are optional and absent parameters generally default to no filtering at all for the given parameter. You can also configure `booster-bitswap` to fetch your retrieval config from a remote HTTP API, possibly one provided by a third party configuration tool like [CIDGravityarrow-up-right](https://cidgravity.com/) . To do this, start `booster-bitswap` with the --api-filter-endpoint {url} option where URL is the HTTP URL for an API serving the above JSON format. Optionally, add --api-filter-auth {authheader}, if you need to pass a value for the HTTP Authorization header with your API When you setup with an API endpoint, `booster-bitswap` will update its local configuration from the API every five minutes, so you won't have to restart `booster-bitswap` to make a change. Please, be aware that the remote config will overwrite, rather than merge, with the local config. ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#bandwidth-limiting) Bandwidth limiting Limiting bandwidth within booster-bitswap will not provide the optimal user experience. Dependent on individual setup, setting up limitations within the software could have a larger impact on the storage provider operations. Therefore, we recommend storage providers to set up their own bandwidth limitations using existing tools. There are multiple options to setup bandwidth limitating. 1. At the ISP level - dedicated bandwidth is provided to the node running booster-bitswap. 2. At the router level - we recommend configuring the bandwidth at the router level as it provides more flexibility and can be updated as needed. To configure the bandwidth on your router, please check with your manufacturer. 3. Limit the bandwidth using different tools available in Linux. Here are some of the examples of such tools. Please feel free to use any other tools not listed here and open a Github issue to add your example to this page. #### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#tc) TC [TCarrow-up-right](https://man7.org/linux/man-pages/man8/tc.8.html) is used to configure Traffic Control in the Linux kernel. There are examples available online detailing how to configure rate limiting using TC. You can use the below commands to run a very basic configuration. #### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#trickle) Trickle [Tricklearrow-up-right](https://wiki.archlinux.org/title/trickle) is a portable lightweight user space bandwidth shaper, that either runs in collaborative mode (together with trickled) or in standalone mode. You can read more about rate limiting with trickle [herearrow-up-right](https://www.tecmint.com/limit-linux-network-bandwidth-usage-with-trickle/) . Here's a starting point for configuration in trickle to rate limit the booster-bitswap service. #### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#wondershaper) Wondershaper Another way of controlling network traffic is to limit bandwidth on individual network interface cards (NICs). [Wondershaperarrow-up-right](https://github.com/magnific0/wondershaper) is a small Bash script that uses the tc command-line utility in the background to let you regulate the amount of data flowing through a particular NIC. As you can imagine, while you can use wondershaper on a machine with a single NIC, its real advantage is on a machine with multiple NICs. Just like trickle, wondershaper is available in the official repositories of mainstream distributions. To limit network traffic with wondershaper, specify the NIC on which you wish to restrict traffic with the download and upload speed in kilobits per second. For example, [PreviousAdvanced Configuration of booster-httpchevron-left](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration) [NextBackup and Restorechevron-right](https://boost.filecoin.io/backup-and-restore) Last updated 2 years ago * [Why enable retrievals via Bitswap?](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#why-enable-retrievals-via-bitswap) * [booster-bitswap modes](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#booster-bitswap-modes) * [Demo configuration](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#demo-configuration) * [Setup booster-bitswap to serve retrievals](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#setup-booster-bitswap-to-serve-retrievals) * [Private Mode](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#private-mode) * [Public mode](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#public-mode) * [booster-bitswap configuration](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#booster-bitswap-configuration) * [Bitswap server performance](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#bitswap-server-performance) * [BadBits filtering](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#badbits-filtering) * [Request filtering](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#request-filtering) * [Bandwidth limiting](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval#bandwidth-limiting) sun-brightdesktopmoon Copy git clone https://github.com/filecoin-project/boost.git cd boost git checkout Copy make build sudo make build Copy booster-bitswap init Copy booster-bitswap run --api-lid="ws://:8044" Copy booster-bitswap fetch /ip4/127.0.0.1/tcp/8888/p2p/{peerID} {rootCID} outfile.car Copy git clone https://github.com/filecoin-project/boost.git cd boost git checkout Copy make build sudo make build Copy booster-bitswap init Copy [Dealmaking] BitswapPeerID ="{peer id for booster bitswap you recorded earlier}" Copy export `lotus auth api-info --perm=admin` Copy export `lotus-miner auth api-info --perm=admin` Copy booster-bitswap run --api-lid="ws://:8044" --proxy={boostd multiaddress} --api-fullnode=$FULLNODE_API_INFO --api-storage=$MINER_API_INFO Copy git clone https://github.com/filecoin-project/boost.git cd boost git checkout Copy make build sudo make build Copy booster-bitswap init Copy [DealMaking] BitswapPeerID ="{peer id for bosoter bitswap you recorded earlier}" BitswapPublicAddresses = ["/ip4/{booster-bitswap public IP}/tcp/{booster-bitswap public port}"] BitswapPrivKeyFile = "{path to libp2p private key file for booster bitswap}" Copy export `lotus auth api-info --perm=admin` Copy export `lotus-miner auth api-info --perm=admin` Copy booster-bitswap run --api-lid="ws://:8044" --api-fullnode=$FULLNODE_API_INFO --api-storage=$MINER_API_INFO Copy booster-bitswap run --api-lid="ws://:8044" \ --engine-blockstore-worker-count=600 \ --engine-task-worker-count=600 \ --max-outstanding-bytes-per-peer=33554432 \ --target-message-size=1048576 \ --task-worker-count=600 Copy booster-bitswap run --api-lid="ws://:8044" --badbits-denylists Copy booster-bitswap run --api-lid="ws://:8044" --badbits-denylists https://badbits.dwebops.pub/denylist.json Copy { "AllowDenyList": { // list of peers to either deny or allow (denying all others) "Type": "allowlist", // "allowlist" or "denylist" "PeerIDs": [\ "Qma9T5YraSnpRDZqRR4krcSJabThc8nwZuJV3LercPHufi",\ "QmYyQSo1c1Ym7orWxLYvCrM2EmxFTANf8wXmmE7DWjhx5N"\ ] }, "UnderMaintenance": false, // when set to true, denies all requests "StorageProviderLimits": { "Bitswap": { "SimultaneousRequests": 100, // bitswap block requests served at the same time across peers "SimultaneousRequestsPerPeer": 10, // bitswap block requests served at the same time for a single peer "MaxBandwidth": "100mb" // human readable size metric, per second } } } Copy booster-bitswap run --api-lid="ws://:8044" --api-filter-endpoint --api-filter-auth Copy sudo tc qdisc add dev root handle 1: htb sudo tc class add dev parent 1: classid 1:20 htb rate 100mibit sudo tc qdisc add dev parent 1:20 handle 20: sfq perturb 10 sudo tc filter add dev parent 1: protocol ip prio 1 basic match 'cmp(u16 at 0 layer transport eq 8888)' flowid 1:20 Copy [booster-bitswap] Priority = Time-Smoothing = Length-Smoothing = Copy wondershaper enp5s0 4096 1024 sun-brightdesktopmoon --- # Hardware requirements | v1.x (Deprecated) | Boost The hardware requirements for Boost are tied to the sealer part of the Lotus deployment it is attached to. Depending on how much data you need to onboard, and how many deals you need to make with clients, hardware requirements in terms of CPU and Disk will vary. [hashtag](https://boost.filecoin.io/v1/hardware-requirements#general-hardware-requirements) General hardware requirements ------------------------------------------------------------------------------------------------------------------------------ ### [hashtag](https://boost.filecoin.io/v1/hardware-requirements#cpu) CPU A miner will need an **8+ core CPU**. We strongly recommend a CPU model with support for _Intel SHA Extensions_: AMD since Zen microarchitecture, or Intel since Ice Lake. Lack of SHA Extensions results in a very significant slow down. The most significant computation that Boost has to do is the _Piece CID calculation (also known as Piece Commitment or CommP)_. When Boost receives data from a client, it calculates the _Merkle root_ out of the hashes of the _Piece_ (padded .car file). The resulting root of the clean binary Merkle tree is the _Piece CID._ ### [hashtag](https://boost.filecoin.io/v1/hardware-requirements#ram) RAM **2 GiB of RAM** are needed at the very least. ### [hashtag](https://boost.filecoin.io/v1/hardware-requirements#disk) Disk Boost stores all data received from clients before Piece CID is calculated and compared against deal parameters received from clients. Next, deals are published on-chain, and Boost waits for a number of epoch confirmations before proceeding to pass data to the Lotus sealing subsystem. This means that depending on the throughput of your operation, you must have disk space for at least a few staged sectors. For small deployments **100 GiB of disk are needed at the very least** if we assume that Boost is to keep three 32 GiB sectors before passing them to the sealing subsystem. We recommend using **NVME disk** for Boost. As Dagstore grows in size, the overall performance might slow down due to slow disk. [Previouslibp2p Protocolschevron-left](https://boost.filecoin.io/v1/boost-architecture/libp2p-protocols) [NextGetting startedchevron-right](https://boost.filecoin.io/v1/getting-started) Last updated 2 years ago * [General hardware requirements](https://boost.filecoin.io/v1/hardware-requirements#general-hardware-requirements) * [CPU](https://boost.filecoin.io/v1/hardware-requirements#cpu) * [RAM](https://boost.filecoin.io/v1/hardware-requirements#ram) * [Disk](https://boost.filecoin.io/v1/hardware-requirements#disk) sun-brightdesktopmoon sun-brightdesktopmoon --- # Troubleshooting | Boost [hashtag](https://boost.filecoin.io/troubleshooting#failed-to-connect-to-peer) Failed to connect to peer ------------------------------------------------------------------------------------------------------------- If the client cannot connect to Boost running on a Storage provider, with an error similar to the following: Copy failed to connect to peer : failed to dial : * failed to negotiate security protocol: peer id mismatch: expected , but remote key matches The problem is that: * The SP registered their peer id and address on chain. eg "Register the peer id `123abcd` at address `/ip4/123.456.12.345/tcp/1234`" Copy lotus-miner actor set-peer-id 123abcd lotus-miner actor set-addrs /ip4/123.456.12.345/tcp/1234 * The SP changed their peer id locally but didn't update the peer id on chain. * The client wants to make a storage deal with peer `123abcd`. The client looks on chain for the address of peer `123abcd` and sees peer `123abcd` has registered an address `/ip4/123.456.12.345/tcp/1234`. * The client sends a deal proposal for peer `123abcd` to the SP at address `/ip4/123.456.12.345/tcp/1234`. * The SP has changed their peer ID, so the SP responds to the deal proposal request with an error: `peer id mismatch` To fix the problem, the SP should register the new peer id on chain: Copy lotus-miner actor set-peer-id [hashtag](https://boost.filecoin.io/troubleshooting#update-storage-providers-on-chain-address) Update storage provider's on chain address ---------------------------------------------------------------------------------------------------------------------------------------------- Clients would not be able to connect to Boost running on a Storage provider after an IP change. This happens as clients lookup the registered peer id and address on chain for a SP. When a SP changes their IP or address locally, they must update the same on chain. The SP should register the new peer id on chain using the following lotus-miner command circle-exclamation Please make sure to use the public IP and port of the Boost node and not `lotus-miner` node if your miner and `boostd` runs on a separate machine. The on chain address change requires access to the worker key and thus the command lives in `lotus-miner` instead of Boost. [hashtag](https://boost.filecoin.io/troubleshooting#error-in-lotus-miner-info-output) Error in _lotus-miner info_ output ----------------------------------------------------------------------------------------------------------------------------- After migrating to Boost, following error is seen when running `lotus-miner info` : #### [hashtag](https://boost.filecoin.io/troubleshooting#problem) Problem: `lotus-miner` is making a call on `lotus-market` process which has been replaced by Boost, but `lotus-miner` is not aware of the new market process. #### [hashtag](https://boost.filecoin.io/troubleshooting#solution) Solution: Export the MARKETS\_API\_INFO variable on your lotus-miner node. [hashtag](https://boost.filecoin.io/troubleshooting#cql-timeout) CQL Timeout --------------------------------------------------------------------------------- Users might find log lines indicating client timeouts when reading or writing to the YugabyteDB's Cassandra API #### [hashtag](https://boost.filecoin.io/troubleshooting#problem-1) Problem: The CQL timeouts can be caused by multiple issues: 1. YugabyteDB prerequisites are not met. This can be due to lack of hardware resources or incorrect hardware types. 2. Slow network connection between `boostd-data` service and YugabyteDB. 3. A low client timeout value. 4. Multiple indexing operation in parallel for large indices (tens of millions of records within the piece index). 5. YugabyteDB has not been scaled up to support the workload in case of extreme workloads. 6. Make sure that you don't have `commp` hash computations running locally for incoming deals, while you also have YugabyteDB on the same node. YugabyteDB is CPU intensive, if another process (such as `boostd` `commp` computation) takes over its resources, it is possible that you are seeing `cql timeout` errors. #### [hashtag](https://boost.filecoin.io/troubleshooting#solution-1) Solution: 1. We recommend verifying that all prerequisites for YugabyteDB are met. 2. Verify that you are running the latest version of Boost. If you are running an older version, we recommend you to upgrade to latest stable version and check issue persists. 3. Reduce the parallel indexing operation to 1 from the default of 4 by updating the value of `ParallelAddIndexLimit` in `config.toml` file 4. Consider moving CommP computations to remote nodes if you are running YugabyteDB on the same host as `boostd` using the `RemoteCommp` flag in `config.toml` file 5. Open a support ticket if issue persists even after trying the above steps. Please make sure to include details about your YugabyteDB hardware and `DEBUG` logs from `boostd` and `boostd-data` service. [PreviousHow to onboard data using DDO dealschevron-left](https://boost.filecoin.io/tutorials/how-to-onboard-data-using-ddo-deals) [NextExperimental Featureschevron-right](https://boost.filecoin.io/experimental-features) Last updated 1 month ago * [Failed to connect to peer](https://boost.filecoin.io/troubleshooting#failed-to-connect-to-peer) * [Update storage provider's on chain address](https://boost.filecoin.io/troubleshooting#update-storage-providers-on-chain-address) * [Error in lotus-miner info output](https://boost.filecoin.io/troubleshooting#error-in-lotus-miner-info-output) * [CQL Timeout](https://boost.filecoin.io/troubleshooting#cql-timeout) sun-brightdesktopmoon Copy lotus-miner actor set-addrs /ip4//tcp/ Copy ERROR: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference Copy export MARKETS_API_INFO= Copy Error: getting pieces count: gocql: no response received from cassandra within timeout period sun-brightdesktopmoon --- # How to store files with Boost on Filecoin | Boost First, you need to initialise a new Boost client and also set the endpoint for a public Filecoin node. In this example we are using [https://glif.ioarrow-up-right](https://glif.io/) Copy export FULLNODE_API_INFO=https://api.node.glif.io boost init The `init` command will output your new wallet address, and warn you that the market actor is not initialised. Copy boost init boost/init_cmd.go:53 default wallet set {"wallet": "f3wfbcudimjcqtfztfhoskgls5gmkfx3kb2ubpycgo7a2ru77temduoj2ottwzlxbrbzm4jycrtu45deawbluq"} boost/init_cmd.go:60 wallet balance {"value": "0"} boost/init_cmd.go:65 market actor is not initialised, you must add funds to it in order to send online deals Then you need to send funds to the wallet, and add funds to the market actor (in the example below we are adding `1 FIL`). You can use the `boostx` utilities to add funds to the market actor: Copy boostx market-add 1 You can confirm that the market actor has funds by running `boost init` again. After that you need to generate a `car` file for data you want to store on Filecoin, and note down its `payload-cid.` We recommend using [`go-car`arrow-up-right](https://github.com/ipld/go-car/releases/latest) CLI to generate the car file. Copy car create -f my-data.car --version 1 car root my-data.car bafykbzacedzjq6jvlqnclrseq8pp5lypa6ozuqgug3wjie3orh67berkwv7e4 Then you need to calculate the `commp` and `piece size` for the generated `car` file: Place the generated `car` file on a public HTTP server, so that a storage provider can later fetch it. Finally, trigger an online storage deal with a given storage provider: [PreviousStart and stop Boost processeschevron-left](https://boost.filecoin.io/tutorials/start-and-stop-boost-processes) [NextUsing filters for storage and retrieval dealschevron-right](https://boost.filecoin.io/tutorials/using-filters-for-storage-and-retrieval-deals) Last updated 2 years ago sun-brightdesktopmoon Copy boostx commp ./my-data.car CommP CID: baga6ea4seaqjaxked6ovoj5f3bdisfeuwtjhrzh3s34mg5cyzevgoebe7tdckdi Piece size: 2097152 Car file size: 1101978 Copy export FULLNODE_API_INFO=https://api.node.glif.io boost deal --verified=false \ --provider=f0026876 \ --http-url=https://public-http-server.com/my-data.car \ --commp=baga6ea4seaqjaxked6ovoj5f3bdisfeuwtjhrzh3s34mg5cyzevgoebe7tdckdi \ --car-size=1101978 \ --piece-size=2097152 \ --payload-cid=bafykbzacedzjq6jvlqnclrseq8pp5lypa6ozuqgug3wjie3orh67berkwv7e4 sun-brightdesktopmoon --- # UI Settings | Boost [hashtag](https://boost.filecoin.io/configuration/ui-settings#configuration) Configuration ----------------------------------------------------------------------------------------------- Copy [Graphql] ListenAddress = "127.0.0.1" Port = 8080 [Monitoring] MpoolAlertEpochs = 30 By default, the web UI listens on the localhost interface on port 8080. We recommend keeping the UI listening on localhost or some internal IP within your private network to avoid accidentally exposing it to the internet. You can access the web UI listening on the localhost interface on a remote server, you can open an SSH tunnel from your local machine: Copy ssh -L 8080:localhost:8080 myserver [hashtag](https://boost.filecoin.io/configuration/ui-settings#settings-page) Settings Page ----------------------------------------------------------------------------------------------- ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252FVz30HqRteslfKXt9wChP%252FScreenshot%25202022-06-08%2520at%25207.25.55%2520PM.png%3Falt%3Dmedia%26token%3De340720f-275f-44e7-b8a7-a2085039b347&width=768&dpr=3&quality=100&sign=db611f1e&sv=2) Settings Page Screenshot Parameter Example Description Price / epoch / Gib 500000000 Asking price for a deal in atto fils. This price is per epoch per GiB of data in a deal Verified Price / epoch / Gib 500000000 Asking price for a verified deal in atto fils. This price is per epoch per GiB of data in a deal Min Piece Size 256 Minimum size of a piece that storage provider will accept in bytes Max Piece Size 34359738368 Maximum size of a piece that storage provider will accept in bytes [PreviousConfigurationchevron-left](https://boost.filecoin.io/configuration) [NextHTTP Transfer limitchevron-right](https://boost.filecoin.io/configuration/http-transfer-limit) Last updated 1 month ago * [Configuration](https://boost.filecoin.io/configuration/ui-settings#configuration) * [Settings Page](https://boost.filecoin.io/configuration/ui-settings#settings-page) sun-brightdesktopmoon sun-brightdesktopmoon --- # How to re-index unsealed pieces that are flagged by LID in Boost v2 | Boost As explained in the tutorial [how to upgrade from Boost v1 to Boost v2](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2) , in Boost v2 the Local Index Directory periodically checks all pieces that the SP stores and confirms if there exists an _unsealed copy_ of the data and whether it is _indexed_. If the index is missing, the piece is _flagged_, meaning that the operator of the SP should fix it if they wish to make it retrievable. ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252FgNG1W4X8tDNx3MYhmAZv%252Fflagged-pieces.png%3Falt%3Dmedia%26token%3D959af3ed-1596-481a-b6d2-4138105923a6&width=768&dpr=3&quality=100&sign=309796f8&sv=2) Fixing individual flagged unsealed pieces from the Boost Web UI is possible directly from the Web UI. If the SP has a lot of flagged pieces, you can automate the re-indexing of pieces with the following commands: #### [hashtag](https://boost.filecoin.io/tutorials/how-to-re-index-unsealed-pieces-that-are-flagged-by-lid-in-boost-v2#fetch-the-piececids-of-all-the-flagged-unsealed-pieces-from-lid) Fetch the `piececid`s of all the flagged unsealed pieces from LID #### [hashtag](https://boost.filecoin.io/tutorials/how-to-re-index-unsealed-pieces-that-are-flagged-by-lid-in-boost-v2#trigger-re-indexing-of-each-piececid) Trigger re-indexing of each `piececid` #### [hashtag](https://boost.filecoin.io/tutorials/how-to-re-index-unsealed-pieces-that-are-flagged-by-lid-in-boost-v2#automatically-re-index-marked-pieces-only) Automatically re-index marked pieces only This little script will fetch up to a thousand flagged pieces that are not processed yet, and will process them 4 at a time. Change the -P4 parameter to change the concurrency. [PreviousHow to upgrade from Boost v1 to Boost v2chevron-left](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2) [NextHow to upgrade from v2.0.0 to v2.1.0chevron-right](https://boost.filecoin.io/tutorials/how-to-upgrade-from-v2.0.0-to-v2.1.0) Last updated 1 year ago sun-brightdesktopmoon Copy // GraphQL query to get the first 100 flagged unsealed pieces from LID. // If you have more than 100 flagged unsealed pieces, you need to modify the query // and adjust the pagination parameters curl -X POST -H "Content-Type: application/json" -d '{"query":"query { piecesFlagged(hasUnsealedCopy: true, limit: 100) { totalCount pieces { CreatedAt PieceCid IndexStatus { Status Error } } } }" }' http://localhost:8080/graphql/query | jq -r ".data.piecesFlagged.pieces[] | .PieceCid" Copy // Re-index a specific piececid. // Assumes that the piececid has an unsealed copy. boostd lid gen-index Copy curl -X POST -H "Content-Type: application/json" -d '{"query":"query { piecesFlagged(hasUnsealedCopy: true, limit: 1000) { totalCount pieces { CreatedAt PieceCid IndexStatus { Status Error } } } }" }' http://localhost:8080/graphql/query | jq -rc ".data.piecesFlagged.pieces[]" | grep Registered | jq -r ".PieceCid" | xargs -P4 -L1 boostd lid gen-index sun-brightdesktopmoon --- # Database | v1.x (Deprecated) | Boost Boost stores metadata about deals in a sqlite database in the root directory of the Boost repo. To open the database use a sqlite client: `sqlite3 boost.db` The database tables are * `Deals` metadata about Boost storage deals (eg deal proposal) and their current state (eg checkpoint) * `FundsLogs` log of each change in funds reserved for a deal * `FundsTagged` how much FIL is tagged for deal collateral and publish message for a deal * `StorageLogs` log of each change in storage reserved for a deal * `StorageTagged` how much storage is tagged for a deal Boost keeps a separate database just for deal logs, so as to make it easier to manage log data separately from deal metadata. The logs database is named `boost.logs.db` and it has a single table `DealLogs` that stores logs for each deal, indexed by uuid. ### [hashtag](https://boost.filecoin.io/v1/boost-architecture/getting-started-1#migrations) Migrations Boost uses `goose` ([https://pressly.github.io/goose/arrow-up-right](https://pressly.github.io/goose/) ) tool and library for handling sqlite3 migrations. `goose` can be installed following the instructions at [https://pressly.github.io/goose/installation/arrow-up-right](https://pressly.github.io/goose/installation/) Migrations in Boost are stored in the `/db/migrations` directory. Boost handles database migrations on start-up. If a user is running an older version of Boost, migrations up to the latest version are automatically applied on start-up. Developers can use `goose` to inspect and apply migrations using the CLI: Copy ➜ ~ goose Usage: goose [OPTIONS] DRIVER DBSTRING COMMAND ... Commands: up Migrate the DB to the most recent version available up-by-one Migrate the DB up by 1 up-to VERSION Migrate the DB to a specific VERSION down Roll back the version by 1 down-to VERSION Roll back to a specific VERSION redo Re-run the latest migration reset Roll back all migrations status Dump the migration status for the current DB version Print the current version of the database create NAME [sql|go] Creates new migration file with the current timestamp fix Apply sequential ordering to migrations [PreviousRepositorychevron-left](https://boost.filecoin.io/v1/boost-architecture/architecture) [NextDAG storechevron-right](https://boost.filecoin.io/v1/boost-architecture/dagstore) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # How to upgrade from v2.0.0 to v2.1.0 | Boost Boost v2.1.0 supports [connecting multiple miners to a single LID instancearrow-up-right](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2#connecting-multiple-boost-instances-to-a-single-lid) . This would enable the SP to serve retrievals from all the connected miners via a single `boostd`, `booster-bitswap` or `booster-http` instance. To enable this feature, we need to update our existing LID tables to add minerID to the piece metadata. 1. Pull the new stable release v2.1.0 or RC candidate v2.1.0-rcx 2. Rebuild the binaries with the new version 3. Stop `booster-http`, `booster-bitswap`, `boostd` and `boostd-data` in that order 4. Start `boostd-data` service first. If you are using systemd service files to manage the process, then please start it manually without using the systemd files. circle-exclamation The Postgres will need to be updated to not use SSL mode. Otherwise, you might see error connecting to the DB. The updated connect string should like below `postgresql://:@:5433?sslmode=disable` circle-info The PGX driver from Yugabyte supports cluster aware Postgres connection out of the box. If you are deploying a multi-node YugabyteDB cluster, then please update your connect-string to use a cluster aware connection. With Cluster Mode: "postgresql://postgres:[\[email protected\]](https://boost.filecoin.io/cdn-cgi/l/email-protection) :5433?load\_balance=true" With Cluster Mode + No SSL: "postgresql://postgres:[\[email protected\]](https://boost.filecoin.io/cdn-cgi/l/email-protection) :5433?sslmode=disable&load\_balance=true" 1. Once `boostd-data` service starts, it will throw the below error and quit the process Copy The database needs to be migrated. Run `boostd-data run yugabyte-migrate` 1. If you do not see the above error and process does not exit, then you do not require a migration. At this point, please skip to step 9. 2. Run the one time migration. Copy boostd-data run yugabyte-migrate --hosts --connect-string --miner-address circle-info Please ensure to use the minerID of already connected miner. Other miners can only be connected once the migration is complete. 1. Once the migration is finished (few seconds to 2 minutes), SPs using systemd to maintain `boostd-data` service should also update their `connect-string` in the systemd service files. 2. Start the `boostd-data` process (or service) and start your `boostd` instance. Go to the UI and confirm that you can see your minerID on the top left side of the page and the LID page is being populated correctly. 3. Start `booster-http` and `booster-bitswap` services. 4. If you wish to connected other miners to the upgraded LID service, then please follow the [migration guide for each miner](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2) . [PreviousHow to re-index unsealed pieces that are flagged by LID in Boost v2chevron-left](https://boost.filecoin.io/tutorials/how-to-re-index-unsealed-pieces-that-are-flagged-by-lid-in-boost-v2) [NextHow to upgrade from v2.1.x to v2.2.0chevron-right](https://boost.filecoin.io/tutorials/how-to-upgrade-from-v2.1.x-to-v2.2.0) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # libp2p Protocols | Boost [hashtag](https://boost.filecoin.io/deployment/libp2p-protocols#propose-storage-deal-protocol) Propose Storage Deal Protocol --------------------------------------------------------------------------------------------------------------------------------- The client makes a deal proposal over `v1.2.0` or `v1.2.1` of the Propose Storage Deal Protocol: - `/fil/storage/mk/1.2.0` or - `/fil/storage/mk/1.2.1` It is a request / response protocol, where the request and response are CBOR-marshalled. There are two new fields in the request of `v1.2.1` of the protocol, described in the table below. ### [hashtag](https://boost.filecoin.io/deployment/libp2p-protocols#request) Request Field Type Description DealUUID uuid A uuid for the deal specified by the client IsOffline boolean Indicates whether the deal is online or offline ClientDealProposal ClientDealProposal Same as `.DealProposal` DealDataRoot cid The root cid of the CAR file. Same as `.Piece.Root` Transfer.Type string eg "http" Transfer.ClientID string Any id the client wants (useful for matching logs between client and server) Transfer.Params byte array Interpreted according to `Type`. eg for "http" `Transfer.Params` contains the http headers as JSON Transfer.Size integer The size of the data that is sent across the network SkipIPNIAnnounce (v1.2.1) boolean Whether the provider should announce the deal to IPNI or not (default: false) RemoveUnsealedCopy (v1.2.1) boolean Whether the provider should keep an unsealed copy of the deal (default: false) ### [hashtag](https://boost.filecoin.io/deployment/libp2p-protocols#response) Response Field Type Description Accepted boolean Indicates whether the deal proposal was accepted Message string A message about why the deal proposal was rejected [hashtag](https://boost.filecoin.io/deployment/libp2p-protocols#storage-deal-status-protocol) Storage Deal Status Protocol ------------------------------------------------------------------------------------------------------------------------------- The client requests the status of a deal over `v1.2.0` of the Storage Deal Status Protocol: `/fil/storage/status/1.2.0` It is a request / response protocol, where the request and response are CBOR-marshalled. ### [hashtag](https://boost.filecoin.io/deployment/libp2p-protocols#request-1) Request Field Type Description DealUUID uuid The uuid of the deal Signature [Signaturearrow-up-right](https://github.com/filecoin-project/go-state-types/blob/057cdfb837f7a0309c1607c7c4640f315e51d7af/crypto/signature.go#L36-L39) A signature over the uuid with the client's wallet ### [hashtag](https://boost.filecoin.io/deployment/libp2p-protocols#response-1) Response Field Type Description DealUUID uuid The uuid of the deal Error string Non-empty if there's an error getting the deal status IsOffline boolean Indicates whether the deal is online or offline TransferSize integer The total size of the transfer in bytes NBytesReceived integer The number of bytes that have been downloaded DealStatus.Error string Non-empty if the deal has failed DealStatus.Status string The [checkpointarrow-up-right](https://github.com/filecoin-project/boost/blob/4fb17ba117784479e09db4012a3abf9862b8afd9/storagemarket/types/dealcheckpoints/checkpoints.go#L7-L15) that the deal has reached DealStatus.Proposal DealProposal SignedProposalCid cid cid of the client deal proposal + signature PublishCid cid The cid of the publish message, if the deal has been published ChainDealID integer The ID of the deal on chain, if it's been published [Previousbooster-bitswapchevron-left](https://boost.filecoin.io/deployment/booster-bitswap) [NextHardware requirementschevron-right](https://boost.filecoin.io/hardware-requirements) Last updated 2 years ago * [Propose Storage Deal Protocol](https://boost.filecoin.io/deployment/libp2p-protocols#propose-storage-deal-protocol) * [Request](https://boost.filecoin.io/deployment/libp2p-protocols#request) * [Response](https://boost.filecoin.io/deployment/libp2p-protocols#response) * [Storage Deal Status Protocol](https://boost.filecoin.io/deployment/libp2p-protocols#storage-deal-status-protocol) * [Request](https://boost.filecoin.io/deployment/libp2p-protocols#request-1) * [Response](https://boost.filecoin.io/deployment/libp2p-protocols#response-1) sun-brightdesktopmoon sun-brightdesktopmoon --- # Direct Deals | Boost [hashtag](https://boost.filecoin.io/experimental-features/direct-deals#set-up-a-devnet-to-make-direct-deals) Set up a devnet to make Direct Deals ------------------------------------------------------------------------------------------------------------------------------------------------------ ### [hashtag](https://boost.filecoin.io/experimental-features/direct-deals#create-images) Create images #### [hashtag](https://boost.filecoin.io/experimental-features/direct-deals#create-lotus-image) Create Lotus image: Copy git clone https://github.com/filecoin-project/lotus lotus-direct-deal cd lotus-direct-deal git checkout nonsense/ddo-integration-and-allocation-and-claim-apis docker build --build-arg FFI_BUILD_FROM_SOURCE=1 --build-arg GOFLAGS=-tags=debug -t filecoin/lotus-all-in-one:feat-ddo-integration-debug --target lotus-all-in-one -f Dockerfile . #### [hashtag](https://boost.filecoin.io/experimental-features/direct-deals#create-boost-images) Create Boost images: Copy git clone https://github.com/filecoin-project/boost boost-direct-deal cd boost-direct-deal make docker/lotus ffi_from_source=1 build_lotus=0 lotus_version=feat-ddo-integration make docker/lotus-miner ffi_from_source=1 build_lotus=0 lotus_version=feat-ddo-integration make docker/boost docker/booster-http docker/booster-bitswap ffi_from_source=1 build_lotus=0 lotus_version=feat-ddo-integration ### [hashtag](https://boost.filecoin.io/experimental-features/direct-deals#start-docker-devnet) Start docker devnet Copy make devnet/up ### [hashtag](https://boost.filecoin.io/experimental-features/direct-deals#setup-notaries-and-grant-datacap) Setup notaries and grant datacap 1. Wait till the Boost UI is reachable at http://localhost and then, open a terminal to the `boost` container 2. Setup notary and add balance to client market actor 3. Grant the datacap to the client [hashtag](https://boost.filecoin.io/experimental-features/direct-deals#make-a-deal) Make a deal ---------------------------------------------------------------------------------------------------- [hashtag](https://boost.filecoin.io/experimental-features/direct-deals#stop-the-devnet) Stop the devnet ------------------------------------------------------------------------------------------------------------ [PreviousFVM Contract Dealschevron-left](https://boost.filecoin.io/experimental-features/fvm-contract-deals) [NextData Segment Indexingchevron-right](https://boost.filecoin.io/experimental-features/data-segment-indexing) Last updated 2 years ago * [Set up a devnet to make Direct Deals](https://boost.filecoin.io/experimental-features/direct-deals#set-up-a-devnet-to-make-direct-deals) * [Create images](https://boost.filecoin.io/experimental-features/direct-deals#create-images) * [Start docker devnet](https://boost.filecoin.io/experimental-features/direct-deals#start-docker-devnet) * [Setup notaries and grant datacap](https://boost.filecoin.io/experimental-features/direct-deals#setup-notaries-and-grant-datacap) * [Make a deal](https://boost.filecoin.io/experimental-features/direct-deals#make-a-deal) * [Stop the devnet](https://boost.filecoin.io/experimental-features/direct-deals#stop-the-devnet) sun-brightdesktopmoon Copy docker compose exec boost /bin/bash Copy export `lotus auth api-info --perm=admin` boost init lotus send --from=`lotus wallet default` `boost wallet default` 100 NOTARY_1=`cat $BOOST_PATH/notary_1` lotus send $NOTARY_1 10 lotus wallet list sleep 10 boostx market-add 5 Copy lotus filplus grant-datacap --from=$NOTARY_1 `boost wallet default` 100000000 Copy chunks=50 links=10 FILE=`boostx generate-rand-car -c=$chunks -l=$links -s=5120000 /app/public/ | awk '{print $NF}'` PAYLOAD_CID=$(find "$FILE" | xargs -I{} basename {} | sed 's/\.car//') COMMP_CID=`boostx commp $FILE 2> /dev/null | grep CID | cut -d: -f2 | xargs` echo $PAYLOAD_CID echo $COMMP_CID boost allocate --miner=t01000 --piece-info=$COMMP_CID=8388608 boost list-allocations boostd import-direct --client-addr=`boost wallet default` --allocation-id=2 $COMMP_CID /app/public/$PAYLOAD_CID.car Copy make devnet/down sun-brightdesktopmoon --- # How to upgrade from v2.1.x to v2.2.0 | Boost Boost v2.2.0 enables support for Direct Data Onbording (DDO) and no longer support legacy deals in any form. These changes resulted in some breaking changes in the configuration. Please follow the below steps to upgrade. 1. Please save you current configuration to a file with `boostd config updated --diff` 2. Note down the asking price setting from the `boostd` UI. 3. Pull the new stable release v2.2.0 or RC candidate v2.2.0-rcx 4. Rebuild the binaries with the new version 5. Stop `booster-http`, `booster-bitswap`, `boostd` and `boostd-data` in that order 6. Start `boostd-data` service first. 7. Start `boostd` after upgrade and shutdown after a successful start 8. Review the `/config.toml` for configuration parameters saved in step 1 and update them if they have been reset to default values. 9. Start `boostd`, `booster-http` and `booster-bitswap` services. 10. Restore the asking price setting from UI. [PreviousHow to upgrade from v2.0.0 to v2.1.0chevron-left](https://boost.filecoin.io/tutorials/how-to-upgrade-from-v2.0.0-to-v2.1.0) [NextStart and stop Boost processeschevron-right](https://boost.filecoin.io/tutorials/start-and-stop-boost-processes) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Tutorials | v1.x (Deprecated) | Boost [How to store files with Boost on Filecoinchevron-right](https://boost.filecoin.io/v1/tutorials/how-to-store-files-with-boost-on-filecoin) [Using filters for storage and retrieval dealschevron-right](https://boost.filecoin.io/v1/tutorials/using-filters-for-storage-and-retrieval-deals) [Setting up a monitoring stack for Boostchevron-right](https://boost.filecoin.io/v1/tutorials/setting-up-a-monitoring-stack-for-boost) [PreviousBackup and Restorechevron-left](https://boost.filecoin.io/v1/backup-and-restore) [NextHow to store files with Boost on Filecoinchevron-right](https://boost.filecoin.io/v1/tutorials/how-to-store-files-with-boost-on-filecoin) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Roll back to Lotus markets service process | v1.x (Deprecated) | Boost circle-exclamation Before you begin migration from Lotus markets service process to Boost, make sure you have a backup of your Lotus repository, by following the [Lotus documentationarrow-up-right](https://lotus.filecoin.io/storage-providers/operate/backup-and-restore/) . You can also do a full backup of the Lotus markets repository directory. 1. If you haven't made any legacy deals with Boost: 1. Stop `boostd` 2. Run your `lotus-miner` `markets` service process as you previously did 2. If you have made new legacy deals with Boost, and want to migrate them back: 1. Stop `boostd` 2. Copy the `dagstore` directory from `boost` repository to `markets` repository. 3. Export Boost deals datastore keys/values: `lotus-shed market export-datastore --repo --backup-dir Wrote backup file to /markets.datastore.backup` 4. Import the exported deals datastore keys/values from `boost` to `lotus markets`: `lotus-shed market import-datastore --repo --backup-path Completed importing from backup file ` [PreviousMigrate a Lotus markets service process to Boostchevron-left](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-lotus-markets-service-process-to-boost) [NextConfigurationchevron-right](https://boost.filecoin.io/v1/configuration) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Migrate from Lotus to Boost | v1.x (Deprecated) | Boost A storage provider can run the lotus as a monolith process where everything is handled by a single `lotus-miner` process or separate the mining and market subsystems on different machines. Boost supports migration from both monolith and a split-market miner. You can follow the below guides to migrate to boost. [Migrate a monolith lotus-miner to Boostchevron-right](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-monolith-lotus-miner-to-boost) [Migrate a Lotus markets service process to Boostchevron-right](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-lotus-markets-service-process-to-boost) ### [hashtag](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost#rollback) **Rollback** triangle-exclamation Please note that Boost uses a SQLite database for the deal metadata and logs. Once Boost has been enabled, the new deals cannot be rolled back to the Lotus markets. If you decide to roll back after making Boost deals, you will lose all the metadata for the deal made with Boost. However, this will have no impact on the sealed data itself. [Roll back to Lotus markets service processchevron-right](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/roll-back-to-legacy-markets) [PreviousAs a clientchevron-left](https://boost.filecoin.io/v1/getting-started/boost-client) [NextMigrate a monolith lotus-miner to Boostchevron-right](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-monolith-lotus-miner-to-boost) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Backup and Restore | v1.x (Deprecated) | Boost [hashtag](https://boost.filecoin.io/v1/backup-and-restore#backup) Backup ----------------------------------------------------------------------------- Boost now supports both online and offline backups. The backup command will output a backup directory containing the following files. 1. `metadata` - contains backup of leveldb 2. `boostd.db` - backup of deals database 3. `keystore` - directory containing libp2p keys 4. `token` - API token 5. `config` - directory containing all config files and `config.toml` link 6. `storage.json` - file containing storage details Backup does not backs up the deal logs and dagstore. ### [hashtag](https://boost.filecoin.io/v1/backup-and-restore#online-backup) Online backup You can take an online backup with the below command Copy boostd backup The online backup supports running only one instance at a time and you might see a locking error if another instance of backup is already running. ### [hashtag](https://boost.filecoin.io/v1/backup-and-restore#offline-backup) Offline backup 1. Shutdown `boostd` before taking a backup 2. Take a backup using the command line circle-info Boost offline backup does not include Dagstore and user can copy dagstore directory to a backup location manually. Dagstore can be reinitialized if there is no backup. [hashtag](https://boost.filecoin.io/v1/backup-and-restore#restore) Restore ------------------------------------------------------------------------------- 1. Make sure that `--boost-repo` flag is set if you wish to restore to a custom location. Otherwise, it will be restored to `~/.boost` directory 2. Restore the boost repo using the command line circle-info Once restore is complete, Dagstore can be manually copied inside the boost repo to restore it. [PreviousHTTP indexer announcementchevron-left](https://boost.filecoin.io/v1/configuration/http-indexer-announcement) [NextTutorialschevron-right](https://boost.filecoin.io/v1/tutorials) Last updated 2 years ago * [Backup](https://boost.filecoin.io/v1/backup-and-restore#backup) * [Online backup](https://boost.filecoin.io/v1/backup-and-restore#online-backup) * [Offline backup](https://boost.filecoin.io/v1/backup-and-restore#offline-backup) * [Restore](https://boost.filecoin.io/v1/backup-and-restore#restore) sun-brightdesktopmoon Copy boostd backup --offline Copy boostd restore sun-brightdesktopmoon --- # Architecture | v1.x (Deprecated) | Boost The `boostd` executable runs as a daemon alongside a lotus node and lotus miner. This daemon replaces the current markets subsystem in the lotus miner. The boost daemon exposes a libp2p interface for storage and retrieval deals. It performs on-chain operations by making API calls to the lotus node. The daemon hands off downloaded data to the lotus miner for sealing via API calls to the lotus miner. `boostd` has a web interface for fund management and deal monitoring. The web interface is a react app that consumes a graphql interface exposed by the daemon. ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F968384846-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F01F3nbbmdopOdEHfj4fF%252Fuploads%252FmkqlvMUatADe4tg1Zxmf%252FBoost%2520Interfaces.png%3Falt%3Dmedia%26token%3Daff4d843-2ec5-4c06-b5fb-69e42a97fad3&width=768&dpr=3&quality=100&sign=b6d2e612&sv=2) ### [hashtag](https://boost.filecoin.io/v1/boost-architecture#storage-deal-flow) Storage Deal Flow The typical flow for a Storage Deal is: 1. The Client puts funds in escrow with the Storage Market Actor on chain. 2. The Client uploads a CAR file to a web server. 3. The Client sends a storage deal proposal to Boost with the URL of the CAR file. 4. Boost checks that the client has enough funds in escrow to pay for storing the file. 5. Boost accepts the storage deal proposal. 6. Boost downloads the CAR file from the web server. 7. Boost publishes the deal on chain. 8. The client checks that the deal was successfully published on chain. ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F968384846-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F01F3nbbmdopOdEHfj4fF%252Fuploads%252FejOvmtPVTaizkC9AyKUJ%252FBoost%2520Flow.png%3Falt%3Dmedia%26token%3Dd2cf5ade-8e33-4336-ba95-47c3fbf40894&width=768&dpr=3&quality=100&sign=3de5bd67&sv=2) Boost exposes a libp2p interface to listen for storage deal proposals from clients. This is similar to the libp2p interface exposed by the lotus market subsystem. Boost communicates with the lotus node over its JSON-RPC API for on-chain operations like checking client funds and publishing the deal. Once the deal has been published, Boost hands off the downloaded file to `lotus-miner` for sealing. [PreviousFeatureschevron-left](https://boost.filecoin.io/v1/features) [NextRepositorychevron-right](https://boost.filecoin.io/v1/boost-architecture/architecture) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # How to onboard data using DDO deals | Boost 1. First, you need to initialise a new Boost client and also set the endpoint for a public Filecoin node. In this example we are using [https://glif.ioarrow-up-right](https://glif.io/) Copy export FULLNODE_API_INFO=https://api.node.glif.io boost init 2. The `init` command will output your new wallet address, and warn you that the market actor is not initialised. Copy boost init boost/init_cmd.go:53 default wallet set {"wallet": "f3wfbcudimjcqtfztfhoskgls5gmkfx3kb2ubpycgo7a2ru77temduoj2ottwzlxbrbzm4jycrtu45deawbluq"} boost/init_cmd.go:60 wallet balance {"value": "0"} boost/init_cmd.go:65 market actor is not initialised, you must add funds to it in order to send online deals 3. Now, you need to send some funds and Datacap to the wallet. 4. You can confirm that the market actor has funds and Datacap by running `boost wallet list`. After that you need to generate a `car` file for data you want to store on Filecoin, and note down its `payload-cid.` We recommend using [`go-car`arrow-up-right](https://github.com/ipld/go-car/releases/latest) CLI to generate the car file. Copy boostx generate-rand-car -c=50 -l=$links -s=5120000 . Payload CID: bafykbzacedr7avw5yvxgjftkhfgzgbinq523csw3ir5hyukx2ulewaigyjdrm, written to: <$CWD>/bafykbzacedr7avw5yvxgjftkhfgzgbinq523csw3ir5hyukx2ulewaigyjdrm.car 5. Then you need to calculate the `commp` and `piece size` for the generated `car` file: boostx generate-rand-car -c=50 -l=$links -s=5120000 . Copy boostx commp bafykbzacedr7avw5yvxgjftkhfgzgbinq523csw3ir5hyukx2ulewaigyjdrm.car CommP CID: baga6ea4seaqjpldhlgodxw2vjj6g46xra7jthe2g37kt7577ep5euxipkupfsly Piece size: 8388608 Car file size: 7657847 6. Create a new verified allocation for this piece using `boost` client. You can use other method to create allocations as long as piece details match the generated commP. Copy boost allocate --miner=t01013 --piece-info=baga6ea4seaqjpldhlgodxw2vjj6g46xra7jthe2g37kt7577ep5euxipkupfsly=8388608 --wallet t3tejq3lb3szsq7spvttqohsfpsju2jof2dbive2qujgz2idqaj2etuolzgbmro3owsmpuebmoghwxgt6ricvq about to send message with the following gas costs max fee: 0.00000000512550864 FIL (absolute maximum amount you are willing to pay to get your transaction confirmed) gas fee cap: 0.00000000000000012 FIL gas limit: 42712572 gas premium: 0.000000000000000023 FIL basefee: 0.0000000000000001 FIL Proceed? Yes [y] / No [n]: y 2024-03-11T18:11:59.794Z INFO boost boost/direct_deal.go:112 submitted data cap allocation message {"cid": "bafy2bzacecyxmx4uyuqfy6xdnlaba2aamlghcujt2asvqmkz6trilnttcouoi"} 2024-03-11T18:11:59.794Z INFO boost boost/direct_deal.go:113 waiting for message to be included in a block AllocationID Client Miner PieceCid PieceSize TermMin TermMax Expiration 31825 1011 1013 baga6ea4seaqjpldhlgodxw2vjj6g46xra7jthe2g37kt7577ep5euxipkupfsly 8388608 518400 5256000 1601277 7. Import the piece for the newly create allocation using `boostd` Copy boostd import-direct --client-addr=t3tejq3lb3szsq7spvttqohsfpsju2jof2dbive2qujgz2idqaj2etuolzgbmro3owsmpuebmoghwxgt6ricvq --allocation-id=31825 baga6ea4seaqjpldhlgodxw2vjj6g46xra7jthe2g37kt7577ep5euxipkupfsly ~/bafykbzacedr7avw5yvxgjftkhfgzgbinq523csw3ir5hyukx2ulewaigyjdrm.car Direct data import scheduled for execution 8. Watch the `boostd` UI to verify that the new DDO deal reaches "Complete" and "Claim Verified" state. [PreviousMigrate from Lotus to Boostchevron-left](https://boost.filecoin.io/tutorials/upgrade-from-lotus-to-boost) [NextTroubleshootingchevron-right](https://boost.filecoin.io/troubleshooting) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # As a client | v1.x (Deprecated) | Boost Boost comes with a client executable, `boost`, that can be used to send a deal proposal to a Boost server. The client is intentionally minimal meant for developer testing. It is not a full featured client and is not intended to be so. It does not require a daemon process, and can be pointed at any public Filecoin API for on-chain operations. This means that users of the client do not need to run a Filecoin node that syncs the chain. ### [hashtag](https://boost.filecoin.io/v1/getting-started/boost-client#set-the-api-endpoint-environment-variable) Set the API endpoint environment variable Copy export FULLNODE_API_INFO= circle-info There are a number of public Filecoin APIs ran by a number of organisations, such as Infura, Glif, etc. For test purposes you can try: `export FULLNODE_API_INFO=https://api.node.glif.io` ### [hashtag](https://boost.filecoin.io/v1/getting-started/boost-client#initialize-the-client) Initialize the client Copy boost -vv init The `init` command * Creates a Boost client repository (at `~/.boost-client` by default) * Generates a libp2p peer ID key * Generates a wallet for on-chain operations and outputs the wallet address ### [hashtag](https://boost.filecoin.io/v1/getting-started/boost-client#add-funds-to-the-wallet-and-to-the-market-actor) Add funds to the wallet and to the market actor To make deals you will need to: a) add funds to the wallet b) add funds to the market actor for that wallet address ### [hashtag](https://boost.filecoin.io/v1/getting-started/boost-client#make-a-storage-deal) Make a storage deal Currently, we don't distribute binaries, so you will have to build from source. When a storage provider accepts the deal, you should see output of the command similar to: ### [hashtag](https://boost.filecoin.io/v1/getting-started/boost-client#check-deal-status) Check deal status You can check the `deal status` with the following command: [PreviousAs a storage providerchevron-left](https://boost.filecoin.io/v1/getting-started/getting-started) [NextMigrate from Lotus to Boostchevron-right](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost) Last updated 2 years ago * [Set the API endpoint environment variable](https://boost.filecoin.io/v1/getting-started/boost-client#set-the-api-endpoint-environment-variable) * [Initialize the client](https://boost.filecoin.io/v1/getting-started/boost-client#initialize-the-client) * [Add funds to the wallet and to the market actor](https://boost.filecoin.io/v1/getting-started/boost-client#add-funds-to-the-wallet-and-to-the-market-actor) * [Make a storage deal](https://boost.filecoin.io/v1/getting-started/boost-client#make-a-storage-deal) * [Check deal status](https://boost.filecoin.io/v1/getting-started/boost-client#check-deal-status) sun-brightdesktopmoon Copy boost -vv deal --provider= \ --http-url= \ --commp= \ --car-size= \ --piece-size= \ --payload-cid= Copy sent deal proposal deal uuid: 9e68fb16-ff9a-488e-ad0a-1289b512d176 storage provider: f0127896 client wallet: f1sw5zjcyo4mff5cbvgsgmm8uoko6gcr4tptvtkhy payload cid: bafyaa6qshafcmalqudsbeidrunclaep6mdbipm2gjfvuosjfd6cbqd6th7bshy5hi5npxe727yjaagelucbyabasgafcmalqudsaeieapsxspo2i36no36n7yitswsxdazvziwvgj4vbp2scuxasrc6n4ejaage3r7m3saykcqeaegeavdllsbzaqcaibaaeecakrvvzam url: https://webserver/file.car commp: baga6ea4seaqh5prrl6ykov4t64k6m5giijsc44dcxtdnzsp4izjakqhs7twauiq start epoch: 1700711 end epoch: 2219111 provider collateral: 358.687 μFIL Copy boost deal-status --provider= --deal-uuid= Copy got deal status response deal uuid: 9e68fb16-ff8a-488e-ad0a-1289b512d176 deal status: Transfer Queued deal label: bafyaa6qsgafcmalqudsaeidrunclaep6mdbipm2gjfvuosjfd6cbqd6th7bshy5hi5npxe727yjaagelucbyabasgafcmalqudsaeieapsxspo2i36no36n7yitswsxdazvziwvgj4vbp2scuxasrc6n4ejaage3r7m3saykcqeaegeavdllsbzaqcaibaaeecakrvvzam publish cid: chain deal id: 0 sun-brightdesktopmoon --- # Start and stop Boost processes | Boost [hashtag](https://boost.filecoin.io/tutorials/start-and-stop-boost-processes#start) Start ---------------------------------------------------------------------------------------------- 1. Start YugabyteDB and wait for services to come online. 2. Start `boostd-data` process. 3. Start `boostd` process. 4. Start `booster-http` and `booster-bitswap` processes. [hashtag](https://boost.filecoin.io/tutorials/start-and-stop-boost-processes#stop) Stop -------------------------------------------------------------------------------------------- 1. Stop `booster-http` and `booster-bitswap` processes. 2. Stop `boostd` process. 3. Stop `boostd-data` process. 4. Finally, you can stop YugabyteDB services. circle-info YugabyteDB services are not required to be stopped if your maintenance work does not directly make changes to the DB itself. The DB can simply be disconnected from boost process by shutting down the `boostd-data` process. [PreviousHow to upgrade from v2.1.x to v2.2.0chevron-left](https://boost.filecoin.io/tutorials/how-to-upgrade-from-v2.1.x-to-v2.2.0) [NextHow to store files with Boost on Filecoinchevron-right](https://boost.filecoin.io/tutorials/how-to-store-files-with-boost-on-filecoin) Last updated 2 years ago * [Start](https://boost.filecoin.io/tutorials/start-and-stop-boost-processes#start) * [Stop](https://boost.filecoin.io/tutorials/start-and-stop-boost-processes#stop) sun-brightdesktopmoon sun-brightdesktopmoon --- # Need help? | v1.x (Deprecated) | Boost 1. You can report any issues or bugs [herearrow-up-right](https://github.com/filecoin-project/boost/issues/new?assignees=&labels=need%2Ftriage%2Ckind%2Fbug&template=bug_report.yml) . 2. If you are having trouble, check the [Troubleshooting](https://boost.filecoin.io/v1/troubleshooting) page for common problems and solutions. 3. If you have a question, please join the [Filecoin Slackarrow-up-right](https://filecoin.io/slack/) and ask in #fil-help or #fil-lotus-help or #boost-help or start [a discussionarrow-up-right](https://github.com/filecoin-project/boost/discussions) . 4. You can also [start a discussionarrow-up-right](https://github.com/filecoin-project/boost/discussions/new?category=ideas) about new feature and improvement ideas for the Boost. [PreviousFAQchevron-left](https://boost.filecoin.io/v1/faq) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Configuration | Boost [hashtag](https://boost.filecoin.io/configuration#sample-config-file) Sample config file --------------------------------------------------------------------------------------------- Copy # The version of the config file (used for migrations) # # type: int # env var: LOTUS__CONFIGVERSION ConfigVersion = 4 # The connect string for the sealing RPC API (lotus miner) # # type: string # env var: LOTUS__SEALERAPIINFO SealerApiInfo = "" # The connect string for the sector index RPC API (lotus miner) # # type: string # env var: LOTUS__SECTORINDEXAPIINFO SectorIndexApiInfo = "" [API] # Binding address for the Lotus API # # type: string # env var: LOTUS_API_LISTENADDRESS #ListenAddress = "/ip4/127.0.0.1/tcp/2345/http" # type: string # env var: LOTUS_API_REMOTELISTENADDRESS #RemoteListenAddress = "" # type: Duration # env var: LOTUS_API_TIMEOUT #Timeout = "30s" [Backup] # Note that in case of metadata corruption it might be much harder to recover # your node if metadata log is disabled # # type: bool # env var: LOTUS_BACKUP_DISABLEMETADATALOG #DisableMetadataLog = false [Libp2p] # Binding address for the libp2p host - 0 means random port. # Format: multiaddress; see https://multiformats.io/multiaddr/ # # type: []string # env var: LOTUS_LIBP2P_LISTENADDRESSES # ListenAddresses = ["/ip4/0.0.0.0/tcp/24001", "/ip6/::/tcp/24001"] # Addresses to explicitally announce to other peers. If not specified, # all interface addresses are announced # Format: multiaddress # # type: []string # env var: LOTUS_LIBP2P_ANNOUNCEADDRESSES # AnnounceAddresses = [] # Addresses to not announce # Format: multiaddress # # type: []string # env var: LOTUS_LIBP2P_NOANNOUNCEADDRESSES # NoAnnounceAddresses = [] # When not disabled (default), lotus asks NAT devices (e.g., routers), to # open up an external port and forward it to the port lotus is running on. # When this works (i.e., when your router supports NAT port forwarding), # it makes the local lotus node accessible from the public internet # # type: bool # env var: LOTUS_LIBP2P_DISABLENATPORTMAP # DisableNatPortMap = false # ConnMgrLow is the number of connections that the basic connection manager # will trim down to. # # type: uint # env var: LOTUS_LIBP2P_CONNMGRLOW # ConnMgrLow = 350 # ConnMgrHigh is the number of connections that, when exceeded, will trigger # a connection GC operation. Note: protected/recently formed connections don't # count towards this limit. # # type: uint # env var: LOTUS_LIBP2P_CONNMGRHIGH # ConnMgrHigh = 400 # ConnMgrGrace is a time duration that new connections are immune from being # closed by the connection manager. # # type: Duration # env var: LOTUS_LIBP2P_CONNMGRGRACE # ConnMgrGrace = "20s" [Pubsub] # Run the node in bootstrap-node mode # # type: bool # env var: LOTUS_PUBSUB_BOOTSTRAPPER #Bootstrapper = false # type: string # env var: LOTUS_PUBSUB_REMOTETRACER #RemoteTracer = "" [Storage] # The maximum number of concurrent fetch operations to the storage subsystem # # type: int # env var: LOTUS_STORAGE_PARALLELFETCHLIMIT # ParallelFetchLimit = 10 [Dealmaking] # When enabled, the miner can accept online deals # # type: bool # env var: LOTUS_DEALMAKING_CONSIDERONLINESTORAGEDEALS #ConsiderOnlineStorageDeals = true # When enabled, the miner can accept offline deals # # type: bool # env var: LOTUS_DEALMAKING_CONSIDEROFFLINESTORAGEDEALS #ConsiderOfflineStorageDeals = true # When enabled, the miner can accept retrieval deals # # type: bool # env var: LOTUS_DEALMAKING_CONSIDERONLINERETRIEVALDEALS #ConsiderOnlineRetrievalDeals = true # When enabled, the miner can accept offline retrieval deals # # type: bool # env var: LOTUS_DEALMAKING_CONSIDEROFFLINERETRIEVALDEALS #ConsiderOfflineRetrievalDeals = true # When enabled, the miner can accept verified deals # # type: bool # env var: LOTUS_DEALMAKING_CONSIDERVERIFIEDSTORAGEDEALS #ConsiderVerifiedStorageDeals = true # When enabled, the miner can accept unverified deals # # type: bool # env var: LOTUS_DEALMAKING_CONSIDERUNVERIFIEDSTORAGEDEALS #ConsiderUnverifiedStorageDeals = true # A list of Data CIDs to reject when making deals # # type: []cid.Cid # env var: LOTUS_DEALMAKING_PIECECIDBLOCKLIST #PieceCidBlocklist = [] # Maximum expected amount of time getting the deal into a sealed sector will take # This includes the time the deal will need to get transferred and published # before being assigned to a sector # # type: Duration # env var: LOTUS_DEALMAKING_EXPECTEDSEALDURATION #ExpectedSealDuration = "24h0m0s" # Maximum amount of time proposed deal StartEpoch can be in future # # type: Duration # env var: LOTUS_DEALMAKING_MAXDEALSTARTDELAY #MaxDealStartDelay = "336h0m0s" # The maximum collateral that the provider will put up against a deal, # as a multiplier of the minimum collateral bound # # type: uint64 # env var: LOTUS_DEALMAKING_MAXPROVIDERCOLLATERALMULTIPLIER #MaxProviderCollateralMultiplier = 2 # The maximum allowed disk usage size in bytes of downloaded deal data # that has not yet been passed to the sealing node by boost. # When the client makes a new deal proposal to download data from a host, # boost checks this config value against the sum of: # - the amount of data downloaded in the staging area # - the amount of data that is queued for download # - the amount of data in the proposed deal # If the total amount would exceed the limit, boost rejects the deal. # Set this value to 0 to indicate there is no limit. # # type: int64 # env var: LOTUS_DEALMAKING_MAXSTAGINGDEALSBYTES # MaxStagingDealsBytes = 500000000 # The percentage of MaxStagingDealsBytes that is allocated to each host. # When the client makes a new deal proposal to download data from a host, # boost checks this config value against the sum of: # - the amount of data downloaded from the host in the staging area # - the amount of data that is queued for download from the host # - the amount of data in the proposed deal # If the total amount would exceed the limit, boost rejects the deal. # Set this value to 0 to indicate there is no limit per host. # # type: uint64 # env var: LOTUS_DEALMAKING_MAXSTAGINGDEALSPERCENTPERHOST # MaxStagingDealsPercentPerHost = 50 # Minimum start epoch buffer to give time for sealing of sector with deal. # # type: uint64 # env var: LOTUS_DEALMAKING_STARTEPOCHSEALINGBUFFER #StartEpochSealingBuffer = 480 # The amount of time to keep deal proposal logs for before cleaning them up. # # type: Duration # env var: LOTUS_DEALMAKING_DEALPROPOSALLOGDURATION #DealProposalLogDuration = "24h0m0s" # The amount of time to keep retrieval deal logs for before cleaning them up. # # type: Duration # env var: LOTUS_DEALMAKING_RETRIEVALLOGDURATION #RetrievalLogDuration = "24h0m0s" # A command used for fine-grained evaluation of storage deals # see https://boost.filecoin.io/configuration/deal-filters for more details # # type: string # env var: LOTUS_DEALMAKING_FILTER #Filter = "" # A command used for fine-grained evaluation of retrieval deals # see https://boost.filecoin.io/configuration/deal-filters for more details # # type: string # env var: LOTUS_DEALMAKING_RETRIEVALFILTER #RetrievalFilter = "" # The maximum amount of time a transfer can take before it fails # # type: Duration # env var: LOTUS_DEALMAKING_MAXTRANSFERDURATION #MaxTransferDuration = "24h0m0s" # Whether to do commp on the Boost node (local) or on the Sealer (remote) # # type: bool # env var: LOTUS_DEALMAKING_REMOTECOMMP #RemoteCommp = false # The maximum number of commp processes to run in parallel on the local # boost process # # type: uint64 # env var: LOTUS_DEALMAKING_MAXCONCURRENTLOCALCOMMP #MaxConcurrentLocalCommp = 1 # The public multi-address for retrieving deals with booster-http. # Note: Must be in multiaddr format, eg /dns/foo.com/tcp/443/https # # type: string # env var: LOTUS_DEALMAKING_HTTPRETRIEVALMULTIADDR #HTTPRetrievalMultiaddr = "" # The maximum number of concurrent storage deal HTTP downloads. # Note that this is a soft maximum; if some downloads stall, # more downloads are allowed to start. # # type: uint64 # env var: LOTUS_DEALMAKING_HTTPTRANSFERMAXCONCURRENTDOWNLOADS HttpTransferMaxConcurrentDownloads = 5 # The period between checking if downloads have stalled. # # type: Duration # env var: LOTUS_DEALMAKING_HTTPTRANSFERSTALLCHECKPERIOD #HttpTransferStallCheckPeriod = "30s" # The time that can elapse before a download is considered stalled (and # another concurrent download is allowed to start). # # type: Duration # env var: LOTUS_DEALMAKING_HTTPTRANSFERSTALLTIMEOUT #HttpTransferStallTimeout = "5m0s" # The peed id used by booster-bitswap. To set, copy the value # printed by running 'booster-bitswap init'. If this value is set, # Boost will: # - listen on bitswap protocols on its own peer id and forward them # to booster bitswap # - advertise bitswap records to the content indexer # - list bitswap in available transports on the retrieval transport protocol # # type: string # env var: LOTUS_DEALMAKING_BITSWAPPEERID # BitswapPeerID = "" # The deal logs older than DealLogDurationDays are deleted from the logsDB # to keep the size of logsDB in check. Set the value as "0" to disable log cleanup # # type: int # env var: LOTUS_DEALMAKING_DEALLOGDURATIONDAYS #DealLogDurationDays = 30 [Dealmaking.RetrievalPricing] # env var: LOTUS_DEALMAKING_RETRIEVALPRICING_STRATEGY #Strategy = "default" [Dealmaking.RetrievalPricing.Default] # env var: LOTUS_DEALMAKING_RETRIEVALPRICING_DEFAULT_VERIFIEDDEALSFREETRANSFER #VerifiedDealsFreeTransfer = true [Dealmaking.RetrievalPricing.External] # env var: LOTUS_DEALMAKING_RETRIEVALPRICING_EXTERNAL_PATH #Path = "" [Wallets] # The "owner" address of the miner # # type: string # env var: LOTUS_WALLETS_MINER Miner = "" # The wallet used to send PublishStorageDeals messages. # Must be a control or worker address of the miner. # # type: string # env var: LOTUS_WALLETS_PUBLISHSTORAGEDEALS PublishStorageDeals = "" # The wallet used as the source for storage deal collateral # # type: string # env var: LOTUS_WALLETS_DEALCOLLATERAL #DealCollateral = "" # Deprecated: Renamed to DealCollateral # # type: string # env var: LOTUS_WALLETS_PLEDGECOLLATERAL PledgeCollateral = "" [Graphql] # The port that the graphql server listens on # # type: uint64 # env var: LOTUS_GRAPHQL_PORT #Port = 8080 [LotusDealmaking] # When enabled, the miner can accept online deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDERONLINESTORAGEDEALS #ConsiderOnlineStorageDeals = true # When enabled, the miner can accept offline deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDEROFFLINESTORAGEDEALS #ConsiderOfflineStorageDeals = true # When enabled, the miner can accept retrieval deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDERONLINERETRIEVALDEALS #ConsiderOnlineRetrievalDeals = true # When enabled, the miner can accept offline retrieval deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDEROFFLINERETRIEVALDEALS #ConsiderOfflineRetrievalDeals = true # When enabled, the miner can accept verified deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDERVERIFIEDSTORAGEDEALS #ConsiderVerifiedStorageDeals = true # When enabled, the miner can accept unverified deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDERUNVERIFIEDSTORAGEDEALS #ConsiderUnverifiedStorageDeals = true # A list of Data CIDs to reject when making deals # # type: []cid.Cid # env var: LOTUS_LOTUSDEALMAKING_PIECECIDBLOCKLIST #PieceCidBlocklist = [] # Maximum expected amount of time getting the deal into a sealed sector will take # This includes the time the deal will need to get transferred and published # before being assigned to a sector # # type: Duration # env var: LOTUS_LOTUSDEALMAKING_EXPECTEDSEALDURATION #ExpectedSealDuration = "24h0m0s" # Maximum amount of time proposed deal StartEpoch can be in future # # type: Duration # env var: LOTUS_LOTUSDEALMAKING_MAXDEALSTARTDELAY #MaxDealStartDelay = "336h0m0s" # When a deal is ready to publish, the amount of time to wait for more # deals to be ready to publish before publishing them all as a batch # # type: Duration # env var: LOTUS_LOTUSDEALMAKING_PUBLISHMSGPERIOD #PublishMsgPeriod = "40m0s" # The maximum number of deals to include in a single PublishStorageDeals # message # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_MAXDEALSPERPUBLISHMSG #MaxDealsPerPublishMsg = 8 # The maximum collateral that the provider will put up against a deal, # as a multiplier of the minimum collateral bound # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_MAXPROVIDERCOLLATERALMULTIPLIER #MaxProviderCollateralMultiplier = 2 # The maximum allowed disk usage size in bytes of staging deals not yet # passed to the sealing node by the markets service. 0 is unlimited. # # type: int64 # env var: LOTUS_LOTUSDEALMAKING_MAXSTAGINGDEALSBYTES MaxStagingDealsBytes = 100000000000 # The maximum number of parallel online data transfers for storage deals # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_SIMULTANEOUSTRANSFERSFORSTORAGE #SimultaneousTransfersForStorage = 20 # The maximum number of simultaneous data transfers from any single client # for storage deals. # Unset by default (0), and values higher than SimultaneousTransfersForStorage # will have no effect; i.e. the total number of simultaneous data transfers # across all storage clients is bound by SimultaneousTransfersForStorage # regardless of this number. # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_SIMULTANEOUSTRANSFERSFORSTORAGEPERCLIENT #SimultaneousTransfersForStoragePerClient = 0 # The maximum number of parallel online data transfers for retrieval deals # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_SIMULTANEOUSTRANSFERSFORRETRIEVAL #SimultaneousTransfersForRetrieval = 20 # Minimum start epoch buffer to give time for sealing of sector with deal. # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_STARTEPOCHSEALINGBUFFER #StartEpochSealingBuffer = 480 # A command used for fine-grained evaluation of storage deals # see https://lotus.filecoin.io/storage-providers/advanced-configurations/market/#using-filters-for-fine-grained-storage-and-retrieval-deal-acceptance for more details # # type: string # env var: LOTUS_LOTUSDEALMAKING_FILTER Filter = "" # A command used for fine-grained evaluation of retrieval deals # see https://lotus.filecoin.io/storage-providers/advanced-configurations/market/#using-filters-for-fine-grained-storage-and-retrieval-deal-acceptance for more details # # type: string # env var: LOTUS_LOTUSDEALMAKING_RETRIEVALFILTER #RetrievalFilter = "" [LotusDealmaking.RetrievalPricing] # env var: LOTUS_LOTUSDEALMAKING_RETRIEVALPRICING_STRATEGY #Strategy = "default" [LotusDealmaking.RetrievalPricing.Default] # env var: LOTUS_LOTUSDEALMAKING_RETRIEVALPRICING_DEFAULT_VERIFIEDDEALSFREETRANSFER #VerifiedDealsFreeTransfer = true [LotusDealmaking.RetrievalPricing.External] # env var: LOTUS_LOTUSDEALMAKING_RETRIEVALPRICING_EXTERNAL_PATH #Path = "" [LotusFees] # The maximum fee to pay when sending the PublishStorageDeals message # # type: types.FIL # env var: LOTUS_LOTUSFEES_MAXPUBLISHDEALSFEE MaxPublishDealsFee = "0.5 FIL" # The maximum fee to pay when sending the AddBalance message (used by legacy markets) # # type: types.FIL # env var: LOTUS_LOTUSFEES_MAXMARKETBALANCEADDFEE #MaxMarketBalanceAddFee = "0.007 FIL" [DAGStore] # Path to the dagstore root directory. This directory contains three # subdirectories, which can be symlinked to alternative locations if # need be: # - ./transients: caches unsealed deals that have been fetched from the # storage subsystem for serving retrievals. # - ./indices: stores shard indices. # - ./datastore: holds the KV store tracking the state of every shard # known to the DAG store. # Default value: /dagstore (split deployment) or # /dagstore (monolith deployment) # # type: string # env var: LOTUS_DAGSTORE_ROOTDIR #RootDir = "" # The maximum amount of indexing jobs that can run simultaneously. # 0 means unlimited. # Default value: 5. # # type: int # env var: LOTUS_DAGSTORE_MAXCONCURRENTINDEX #MaxConcurrentIndex = 5 # The maximum amount of unsealed deals that can be fetched simultaneously # from the storage subsystem. 0 means unlimited. # Default value: 0 (unlimited). # # type: int # env var: LOTUS_DAGSTORE_MAXCONCURRENTREADYFETCHES #MaxConcurrentReadyFetches = 0 # The maximum amount of unseals that can be processed simultaneously # from the storage subsystem. 0 means unlimited. # Default value: 0 (unlimited). # # type: int # env var: LOTUS_DAGSTORE_MAXCONCURRENTUNSEALS #MaxConcurrentUnseals = 0 # The maximum number of simultaneous inflight API calls to the storage # subsystem. # Default value: 100. # # type: int # env var: LOTUS_DAGSTORE_MAXCONCURRENCYSTORAGECALLS #MaxConcurrencyStorageCalls = 100 # The time between calls to periodic dagstore GC, in time.Duration string # representation, e.g. 1m, 5m, 1h. # Default value: 1 minute. # # type: Duration # env var: LOTUS_DAGSTORE_GCINTERVAL #GCInterval = "1m0s" [IndexProvider] # Enable set whether to enable indexing announcement to the network and expose endpoints that # allow indexer nodes to process announcements. Enabled by default. # # type: bool # env var: LOTUS_INDEXPROVIDER_ENABLE #Enable = true # EntriesCacheCapacity sets the maximum capacity to use for caching the indexing advertisement # entries. Defaults to 1024 if not specified. The cache is evicted using LRU policy. The # maximum storage used by the cache is a factor of EntriesCacheCapacity, EntriesChunkSize and # the length of multihashes being advertised. For example, advertising 128-bit long multihashes # with the default EntriesCacheCapacity, and EntriesChunkSize means the cache size can grow to # 256MiB when full. # # type: int # env var: LOTUS_INDEXPROVIDER_ENTRIESCACHECAPACITY #EntriesCacheCapacity = 1024 # EntriesChunkSize sets the maximum number of multihashes to include in a single entries chunk. # Defaults to 16384 if not specified. Note that chunks are chained together for indexing # advertisements that include more multihashes than the configured EntriesChunkSize. # # type: int # env var: LOTUS_INDEXPROVIDER_ENTRIESCHUNKSIZE #EntriesChunkSize = 16384 # TopicName sets the topic name on which the changes to the advertised content are announced. # If not explicitly specified, the topic name is automatically inferred from the network name # in following format: '/indexer/ingest/' # Defaults to empty, which implies the topic name is inferred from network name. # # type: string # env var: LOTUS_INDEXPROVIDER_TOPICNAME #TopicName = "" # PurgeCacheOnStart sets whether to clear any cached entries chunks when the provider engine # starts. By default, the cache is rehydrated from previously cached entries stored in # datastore if any is present. # # type: bool # env var: LOTUS_INDEXPROVIDER_PURGECACHEONSTART #PurgeCacheOnStart = false [ContractDeals] Enabled = true circle-info [hashtag](https://boost.filecoin.io/configuration#sealer) Sealer --------------------------------------------------------------------- Parameter Example Description SealerApiInfo "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBbGxvdyI6WyJyZwdyIiwid3JpdGUiLCJzaWduIiwiYWRtaW4iXX0.nbSvy11-tSUbXqo465hZqzTohGDfSdgh28C4irkmE10:/ip4/0.0.0.0/tcp/2345/http" Miner API info passed during boost init. Requires admin permissions. Connect string for the miner/sealer instance API endpoint SectorIndexApiInfo "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBbGxvdyI6WyJyZwdyIiwid3JpdGUiLCJzaWduIiwiYWRtaW4iXX0.nbSvy11-tSUbXqo465hZqzTohGDfSdgh28C4irkmE10:/ip4/0.0.0.0/tcp/2345/http" Miner API info passed during boost init. Requires admin permissions. Connect string for the miner/sealer instance API endpoint [hashtag](https://boost.filecoin.io/configuration#api) API --------------------------------------------------------------- Parameter Example Description ListenAddress "/ip4/127.0.0.1/tcp/1288/http" \# Format: multiaddress Address Boost API will be listening on. No need to update unless you are planning to make API calls from outside the boost node RemoteListenAddress "0.0.0.0:1288" Address boost API can reached at from outside. No need to update unless you are planning to make API calls from outside the boost node Timeout "30s" RPC timeout value [hashtag](https://boost.filecoin.io/configuration#libp2p) Libp2p --------------------------------------------------------------------- Parameter Example Description ListenAddresses \# Format: multiaddress \["/ip4/209.94.92.3/tcp/24001"\] Binding address for the libp2p host - 0 means random port. AnnounceAddresses \# Format: multiaddress \["/ip4/209.94.92.3/tcp/24001"\] Addresses to explicitly announce to other peers. If not specified, all interface addresses are announced. On chain address need to be updated when this address is changed # lotus-miner actor set-addrs /ip4//tcp/24001 NoAnnounceAddresses \# Format: multiaddress \["/ip4/209.94.92.3/tcp/24001"\] Addresses to not announce. Can be used if you want to announce addresses with exceptions ConnMgrLow 150 ConnMgrLow is the number of connections that the basic connection manager will trim down to. Too low number can cause frequent connectivity issues ConnMgrHigh 200 ConnMgrHigh is the number of connections that, when exceeded, will trigger a connection GC operation Note: protected/recently formed connections don't count towards this limit. A high limit can cause very high resource utilization ConnMgrGrace "20s" ConnMgrGrace is a time duration that new connections are immune from being closed by the connection manager. [hashtag](https://boost.filecoin.io/configuration#storage) Storage ----------------------------------------------------------------------- Parameter Example Description ParallelFetchLimit 10 Upper bound on how many sectors can be fetched in parallel by the storage system at a time [hashtag](https://boost.filecoin.io/configuration#dealmaking) Dealmaking ----------------------------------------------------------------------------- `Dealmaking` section handles deal making configuration explicitly for boost deal that uses the new `/fil/storage/mk/1.2.0` protocol. [hashtag](https://boost.filecoin.io/configuration#wallets) Wallets ----------------------------------------------------------------------- Parameter Example Description Miner f032187 Miner ID PublishStorageDeals f3syzhufifmnbzcznoquhy4mlxo3byetqlamzbeijk62bjpoohrj3wiphkgxe3yjrlh5dmxlca3zqxp3yvd33a #BLS wallet address This value is taken during init with `--wallet-publish-storage-deals.` This wallet is used to send PublishDeal messages. It can be hosted on the remote daemon node and does not require to be present locally. DealCollateral f3syzhufifmnbzcznoquhy4mlxo3byetqlamzbeijk62bjpoohrj3wiphkgxe3yjrlh5dmxlca3zqxp3yvd33a #BLS wallet address This value is taken during init with `--wallet-deal-collateral`. This wallet is used to provide collateral for the deal. Funds from this wallet are moved to market actor and locked during the deal duration. It can be hosted on the remote daemon node and does not require to be present locally. [hashtag](https://boost.filecoin.io/configuration#lotusfees) LotusFees --------------------------------------------------------------------------- Parameter Example Description MaxPublishDealsFee "0.05 FIL" Maximum fee user is willing to pay for a PublishDeal message MaxMarketBalanceAddFee "0.007 FIL" The maximum fee to pay when sending the AddBalance message (used by legacy markets) [hashtag](https://boost.filecoin.io/configuration#dagstore) DAGStore ------------------------------------------------------------------------- Parameter Example Description RootDir Empty If a custom value is specified, boost instance will refuse to start. This will be deprecated and removed in the future. MaxConcurrentIndex 5 The maximum amount of indexing jobs that can run simultaneously. 0 means unlimited. MaxConcurrentReadyFetches 0 The maximum amount of unsealed deals that can be fetched simultaneously from the storage subsystem. 0 means unlimited. MaxConcurrentUnseals 0 The maximum amount of unseals that can be processed simultaneously from the storage subsystem. 0 means unlimited. MaxConcurrencyStorageCalls 100 The maximum number of simultaneous inflight API calls to the storage subsystem. GCInterval "1m0s" The time between calls to periodic dagstore GC, in time.Duration string representation, e.g. 1m, 5m, 1h. [hashtag](https://boost.filecoin.io/configuration#indexprovider) IndexProvider ----------------------------------------------------------------------------------- Parameter Example Description Enable True/False Enabled or disable the index-provider subsystem EntriesCacheCapacity 5 EntriesCacheCapacity sets the maximum capacity to use for caching the indexing advertisement entries. Defaults to 1024 if not specified. The cache is evicted using LRU policy. The maximum storage used by the cache is a factor of EntriesCacheCapacity, EntriesChunkSize and the length of multihashes being advertised. EntriesChunkSize 0 EntriesChunkSize sets the maximum number of multihashes to include in a single entries chunk. Defaults to 16384 if not specified. Note that chunks are chained together for indexing advertisements that include more multihashes than the configured EntriesChunkSize. TopicName "" TopicName sets the topic name on which the changes to the advertised content are announced. If not explicitly specified, the topic name is automatically inferred from the network name in following format: '/indexer/ingest/' PurgeCacheOnStart 100 PurgeCacheOnStart sets whether to clear any cached entries chunks when the provider engine starts. By default, the cache is rehydrated from previously cached entries stored in datastore if any is present. GCInterval "1m0s" The time between calls to periodic dagstore GC, in time.Duration string representation, e.g. 1m, 5m, 1h. Advertising 128-bit long multihashes with the default EntriesCacheCapacity, and EntriesChunkSize means the cache size can grow to 256MiB when full. [PreviousNew Boost Setupchevron-left](https://boost.filecoin.io/new-boost-setup) [NextUI Settingschevron-right](https://boost.filecoin.io/configuration/ui-settings) Last updated 1 month ago * [Sample config file](https://boost.filecoin.io/configuration#sample-config-file) * [Sealer](https://boost.filecoin.io/configuration#sealer) * [API](https://boost.filecoin.io/configuration#api) * [Libp2p](https://boost.filecoin.io/configuration#libp2p) * [Storage](https://boost.filecoin.io/configuration#storage) * [Dealmaking](https://boost.filecoin.io/configuration#dealmaking) * [Wallets](https://boost.filecoin.io/configuration#wallets) * [LotusFees](https://boost.filecoin.io/configuration#lotusfees) * [DAGStore](https://boost.filecoin.io/configuration#dagstore) * [IndexProvider](https://boost.filecoin.io/configuration#indexprovider) sun-brightdesktopmoon Copy This guide covers all the configuration in use by boostd process. Some of the configuration parameters found in the config.toml file are not used in boost and thus are not covered here. These configuration parameters can be ignored sun-brightdesktopmoon --- # FVM Contract Deals | v1.x (Deprecated) | Boost With the release of FVM, it is now possible for smart contracts to make deal proposals on-chain. This is made possible though the DealProposal FRC. DataDAOs, as well as other clients who want to store data on Filecoin, can now deploy a smart contract on the FVM which adheres to the DealProposal FRC, and make deal proposals that are visible to every storage provider who monitors the chain. [hashtag](https://boost.filecoin.io/v1/experimental-features/fvm-contract-deals#how-to-enable-fvm-monitoring-in-order-to-process-storage-deal-proposals-published-on-chain) How to enable FVM monitoring in order to process storage deal proposals published on-chain? ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Boost already has support for the DealProposal FRC. The code for FVM monitoring resides in the latest release of the Boost. It should be used with caution for production use. SPs must [enable FEVM on lotus daemonarrow-up-right](https://lotus.filecoin.io/lotus/configure/ethereum-rpc/) before proceeding to the next step. To build for mainnet: Copy git clone https://github.com/filecoin-project/boost.git cd boost git checkout make build In order to enable DealProposal FRC, you have to edit your `config.toml` and enable contract deal monitoring. By default it is disabled. Here is an example configuration: Copy [ContractDeals] Enabled = true AllowlistContracts = [] From = "0x0000000000000000000000000000000000000000" `AllowlistContracts` field could be left empty if you want to accept deals from any client. If you only want to accept deals from some clients, you can specify their contract addresses in the field. `From` field should be set to your SP's FEVM address. Some clients may implement a whitelist which allows specific SPs to accept deal proposals from their contract. This field will help those clients identify your SP and match it to their whitelist. ### [hashtag](https://boost.filecoin.io/v1/experimental-features/fvm-contract-deals#how-contract-deals-work-in-boost) How contract deals work in Boost 1. A contract publishes a `DealProposalCreate` event on the chain. 2. Boost monitors the chain for such events from all the clients by default. When such an event is detected, we go and fetch the data for the deal. 3. Deal is then run through the basic deal validation filters like clients has enough funds, SP has enough funds etc. 4. Once deal passes the validation, we create a new deal handler in Boost and pass this deal for execution like other Boost deals. [PreviousExperimental Featureschevron-left](https://boost.filecoin.io/v1/experimental-features) [NextLocal Index Directorychevron-right](https://boost.filecoin.io/v1/experimental-features/local-index-directory) Last updated 2 years ago * [How to enable FVM monitoring in order to process storage deal proposals published on-chain?](https://boost.filecoin.io/v1/experimental-features/fvm-contract-deals#how-to-enable-fvm-monitoring-in-order-to-process-storage-deal-proposals-published-on-chain) * [How contract deals work in Boost](https://boost.filecoin.io/v1/experimental-features/fvm-contract-deals#how-contract-deals-work-in-boost) sun-brightdesktopmoon sun-brightdesktopmoon --- # How to store files with Boost on Filecoin | v1.x (Deprecated) | Boost First, you need to initialise a new Boost client and also set the endpoint for a public Filecoin node. In this example we are using [https://glif.ioarrow-up-right](https://glif.io/) Copy export FULLNODE_API_INFO=https://api.node.glif.io boost init The `init` command will output your new wallet address, and warn you that the market actor is not initialised. Copy boost init boost/init_cmd.go:53 default wallet set {"wallet": "f3wfbcudimjcqtfztfhoskgls5gmkfx3kb2ubpycgo7a2ru77temduoj2ottwzlxbrbzm4jycrtu45deawbluq"} boost/init_cmd.go:60 wallet balance {"value": "0"} boost/init_cmd.go:65 market actor is not initialised, you must add funds to it in order to send online deals Then you need to send funds to the wallet, and add funds to the market actor (in the example below we are adding `1 FIL`). You can use the `boostx` utilities to add funds to the market actor: Copy boostx market-add 1 You can confirm that the market actor has funds by running `boost init` again. After that you need to generate a `car` file for data you want to store on Filecoin, and note down its `payload-cid.` We recommend using [`go-car`arrow-up-right](https://github.com/ipld/go-car/releases/latest) CLI to generate the car file. Copy car create -f my-data.car --version 1 car root my-data.car bafykbzacedzjq6jvlqnclrseq8pp5lypa6ozuqgug3wjie3orh67berkwv7e4 Then you need to calculate the `commp` and `piece size` for the generated `car` file: Place the generated `car` file on a public HTTP server, so that a storage provider can later fetch it. Finally, trigger an online storage deal with a given storage provider: [PreviousTutorialschevron-left](https://boost.filecoin.io/v1/tutorials) [NextUsing filters for storage and retrieval dealschevron-right](https://boost.filecoin.io/v1/tutorials/using-filters-for-storage-and-retrieval-deals) Last updated 2 years ago sun-brightdesktopmoon Copy boostx commp ./my-data.car CommP CID: baga6ea4seaqjaxked6ovoj5f3bdisfeuwtjhrzh3s34mg5cyzevgoebe7tdckdi Piece size: 2097152 Car file size: 1101978 Copy export FULLNODE_API_INFO=https://api.node.glif.io boost deal --verified=false \ --provider=f0026876 \ --http-url=https://public-http-server.com/my-data.car \ --commp=baga6ea4seaqjaxked6ovoj5f3bdisfeuwtjhrzh3s34mg5cyzevgoebe7tdckdi \ --car-size=1101978 \ --piece-size=2097152 \ --payload-cid=bafykbzacedzjq6jvlqnclrseq8pp5lypa6ozuqgug3wjie3orh67berkwv7e4 sun-brightdesktopmoon --- # UI Settings | v1.x (Deprecated) | Boost [hashtag](https://boost.filecoin.io/v1/configuration/ui-settings#configuration) Configuration -------------------------------------------------------------------------------------------------- Copy [Graphql] ListenAddress = "127.0.0.1" Port = 8080 [Monitoring] MpoolAlertEpochs = 30 By default, the web UI listens on the localhost interface on port 8080. We recommend keeping the UI listening on localhost or some internal IP within your private network to avoid accidentally exposing it to the internet. You can access the web UI listening on the localhost interface on a remote server, you can open an SSH tunnel from your local machine: Copy ssh -L 8080:localhost:8080 myserver [hashtag](https://boost.filecoin.io/v1/configuration/ui-settings#settings-page) Settings Page -------------------------------------------------------------------------------------------------- ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F968384846-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F01F3nbbmdopOdEHfj4fF%252Fuploads%252FqnnEksLAxyYWM7omUJ5C%252FScreenshot%25202022-06-08%2520at%25207.25.55%2520PM.png%3Falt%3Dmedia%26token%3De340720f-275f-44e7-b8a7-a2085039b347&width=768&dpr=3&quality=100&sign=c119470b&sv=2) Settings Page Screenshot Parameter Example Description Price / epoch / Gib 500000000 Asking price for a deal in atto fils. This price is per epoch per GiB of data in a deal Verified Price / epoch / Gib 500000000 Asking price for a verified deal in atto fils. This price is per epoch per GiB of data in a deal Min Piece Size 256 Minimum size of a piece that storage provider will accept in bytes Max Piece Size 34359738368 Maximum size of a piece that storage provider will accept in bytes [PreviousConfigurationchevron-left](https://boost.filecoin.io/v1/configuration) [NextHTTP Transfer limitchevron-right](https://boost.filecoin.io/v1/configuration/http-transfer-limit) Last updated 2 years ago * [Configuration](https://boost.filecoin.io/v1/configuration/ui-settings#configuration) * [Settings Page](https://boost.filecoin.io/v1/configuration/ui-settings#settings-page) sun-brightdesktopmoon sun-brightdesktopmoon --- # Repository | v1.x (Deprecated) | Boost By default the Boost daemon repository is located at `~/.boost` It contains the following files: * `api` The local multi-address of Boost's libp2p API * `boost.db` The sqlite database with all deal metadata * `boost.logs.db` The sqlite database with the logs for deals * `config.toml` The config file with all Boost's settings * `repo.lock` A lock file created when Boost is running * `storage.json` Deprecated (needed by legacy markets) * `token` The token used when calling Boost's JSON RPC endpoints It has the following directories: * `dagstore` Contains indexes of CAR files stored with Boost * `datastore` Contains metadata about deals for legacy markets * `deal-staging` The directory used by legacy markets for incoming data transfers * `incoming` The directory used by Boost for incoming data transfers * `journal` Contains journal events (used by legacy markets) * `keystore` Contains the secret keys used by libp2p (eg the peer ID) * `kvlog` Used by legacy markets datastore [PreviousArchitecturechevron-left](https://boost.filecoin.io/v1/boost-architecture) [NextDatabasechevron-right](https://boost.filecoin.io/v1/boost-architecture/getting-started-1) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # libp2p Protocols | v1.x (Deprecated) | Boost Boost supports the same libp2p protocols as legacy markets, and adds new versions of the protocols used to propose a storage deal and to check the deal's status. [hashtag](https://boost.filecoin.io/v1/boost-architecture/libp2p-protocols#propose-storage-deal-protocol) Propose Storage Deal Protocol -------------------------------------------------------------------------------------------------------------------------------------------- The client makes a deal proposal over `v1.2.0` or `v1.2.1` of the Propose Storage Deal Protocol: `/fil/storage/mk/1.2.0` or `/fil/storage/mk/1.2.1` It is a request / response protocol, where the request and response are CBOR-marshalled. There are two new fields in the Request of `v1.2.1` of the protocol, described in the table below. ### [hashtag](https://boost.filecoin.io/v1/boost-architecture/libp2p-protocols#request) Request Field Type Description DealUUID uuid A uuid for the deal specified by the client IsOffline boolean Indicates whether the deal is online or offline ClientDealProposal ClientDealProposal Same as `.DealProposal` DealDataRoot cid The root cid of the CAR file. Same as `.Piece.Root` Transfer.Type string eg "http" Transfer.ClientID string Any id the client wants (useful for matching logs between client and server) Transfer.Params byte array Interpreted according to `Type`. eg for "http" `Transfer.Params` contains the http headers as JSON Transfer.Size integer The size of the data that is sent across the network SkipIPNIAnnounce (v1.2.1) boolean Whether the provider should announce the deal to IPNI or not (default: false) RemoveUnsealedCopy (v1.2.1) boolean Whether the provider should keep an unsealed copy of the deal (default: false) ### [hashtag](https://boost.filecoin.io/v1/boost-architecture/libp2p-protocols#response) Response Field Type Description Accepted boolean Indicates whether the deal proposal was accepted Message string A message about why the deal proposal was rejected [hashtag](https://boost.filecoin.io/v1/boost-architecture/libp2p-protocols#storage-deal-status-protocol) Storage Deal Status Protocol ------------------------------------------------------------------------------------------------------------------------------------------ The client requests the status of a deal over `v1.2.0` of the Storage Deal Status Protocol: `/fil/storage/status/1.2.0` It is a request / response protocol, where the request and response are CBOR-marshalled. ### [hashtag](https://boost.filecoin.io/v1/boost-architecture/libp2p-protocols#request-1) Request Field Type Description DealUUID uuid The uuid of the deal Signature [Signaturearrow-up-right](https://github.com/filecoin-project/go-state-types/blob/057cdfb837f7a0309c1607c7c4640f315e51d7af/crypto/signature.go#L36-L39) A signature over the uuid with the client's wallet ### [hashtag](https://boost.filecoin.io/v1/boost-architecture/libp2p-protocols#response-1) Response Field Type Description DealUUID uuid The uuid of the deal Error string Non-empty if there's an error getting the deal status IsOffline boolean Indicates whether the deal is online or offline TransferSize integer The total size of the transfer in bytes NBytesReceived integer The number of bytes that have been downloaded DealStatus.Error string Non-empty if the deal has failed DealStatus.Status string The [checkpointarrow-up-right](https://github.com/filecoin-project/boost/blob/4fb17ba117784479e09db4012a3abf9862b8afd9/storagemarket/types/dealcheckpoints/checkpoints.go#L7-L15) that the deal has reached DealStatus.Proposal DealProposal SignedProposalCid cid cid of the client deal proposal + signature PublishCid cid The cid of the publish message, if the deal has been published ChainDealID integer The ID of the deal on chain, if it's been published [PreviousDAG storechevron-left](https://boost.filecoin.io/v1/boost-architecture/dagstore) [NextHardware requirementschevron-right](https://boost.filecoin.io/v1/hardware-requirements) Last updated 2 years ago * [Propose Storage Deal Protocol](https://boost.filecoin.io/v1/boost-architecture/libp2p-protocols#propose-storage-deal-protocol) * [Request](https://boost.filecoin.io/v1/boost-architecture/libp2p-protocols#request) * [Response](https://boost.filecoin.io/v1/boost-architecture/libp2p-protocols#response) * [Storage Deal Status Protocol](https://boost.filecoin.io/v1/boost-architecture/libp2p-protocols#storage-deal-status-protocol) * [Request](https://boost.filecoin.io/v1/boost-architecture/libp2p-protocols#request-1) * [Response](https://boost.filecoin.io/v1/boost-architecture/libp2p-protocols#response-1) sun-brightdesktopmoon sun-brightdesktopmoon --- # Migrate a Lotus markets service process to Boost | v1.x (Deprecated) | Boost circle-exclamation If you are running a `monolith` `lotus-miner`and have not yet split the `markets` service into an individual process, follow the steps in [Migrate a monolith lotus-miner to Boost](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-monolith-lotus-miner-to-boost) . If you are running a `markets` service as a separate `lotus-miner` process: 1\. Stop accepting incoming deals 2\. Wait for incoming deals to complete 3\. Shutdown the markets process 4\. Backup the markets repository 5\. Backup the markets datastore (in case you decide to roll back from Boost to Lotus) with: Copy lotus-shed market export-datastore --repo --backup-dir 6\. Make sure you have a Lotus node and miner running 7\. Create and send funds to two new wallets on the lotus node to be used for Boost Boost currently uses two wallets for storage deals: * The publish storage deals wallet - This wallet pays the gas cost when Boost sends the `PublishStorageDeals` message. circle-info If you already have a PublishStorageDeal control wallet setup then it can be reused in boost as the `PUBLISH_STORAGE_DEALS_WALLET`. * The deal collateral wallet - When the Storage Provider accepts a deal, they must put collateral for the deal into escrow. Boost moves funds from this wallet into escrow with the `StorageMarketActor`. circle-info If you already have a wallet that you want to use as the source of funds for deal collateral, then it can be reused in boost as the `COLLAT_WALLET`. Copy PUBLISH_STORAGE_DEALS_WALLET=`lotus wallet new bls` COLLAT_WALLET=`lotus wallet new bls` lotus send --from mywallet $PUBLISH_STORAGE_DEALS_WALLET 10 lotus send --from mywallet $COLLAT_WALLET 10 8\. Boost keeps all data in a directory called the repository. By default the repository is at `~/.boost`. To use a different location pass the `--boost-repo` parameter. 9\. Export the environment variables needed for `boostd migrate-markets` to connect to the lotus daemon and lotus miner. Export environment variables that point to the API endpoints for the sealing and mining processes. They will be used by the `boost` node to make JSON-RPC calls to the `mining/sealing/proving` node. 10\. Set the publish storage deals wallet as a control wallet. circle-exclamation Add the value of`PUBLISH_STORAGE_DEALS_WALLET` to the parameter `DealPublishControl` in the `Address` section of lotus-miner configuration if not present. Restart lotus-miner if configuration has been updated. 11\. Run `boostd migrate-markets` to initialize the repository and start the migration: The `migrate-markets` command * Initializes a Boost repository * Migrates markets datastore keys to Boost * Storage and retrieval deal metadata * Storage and retrieval ask data * Migrates markets libp2p keys to Boost * Migrates markets config to Boost (libp2p endpoints, settings etc) * Migrates the markets DAG store to Boost 12\. Run the `boostd` service, which will start: * libp2p listeners for storage and retrieval * the JSON RPC API * the graphql interface (used by the react front-end) * the web server for the react front-end circle-info In your firewall you will need to open the ports that libp2p listens on, so that Boost can receive storage and retrieval deals. See the `Libp2p` section of `config.toml` in the [Repository](https://boost.filecoin.io/v1/boost-architecture/architecture) ### [hashtag](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-lotus-markets-service-process-to-boost#web-ui) Web UI Open http://localhost:8080 in your browser. circle-info To access a web UI running on a remote server, you can open an SSH tunnel from your local machine: ### [hashtag](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-lotus-markets-service-process-to-boost#api-access) API Access Boost API can be accessed by setting the environment variable `BOOST_API_INFO` same as `LOTUS_MARKET_INFO`. [PreviousMigrate a monolith lotus-miner to Boostchevron-left](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-monolith-lotus-miner-to-boost) [NextRoll back to Lotus markets service processchevron-right](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/roll-back-to-legacy-markets) Last updated 2 years ago * [Web UI](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-lotus-markets-service-process-to-boost#web-ui) * [API Access](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-lotus-markets-service-process-to-boost#api-access) sun-brightdesktopmoon Copy export $(lotus auth api-info --perm=admin) export $(lotus-miner auth api-info --perm=admin) Copy export APISEALER=`echo $MINER_API_INFO` export APISECTORINDEX=`echo $MINER_API_INFO` Copy export OLD_CONTROL_ADDRESS=`lotus-miner actor control list --verbose | grep -v owner | grep -v worker | grep -v beneficiary | awk '{print $3}' | grep -v key | tr -s '\n' ' '` lotus-miner actor control set --really-do-it $PUBLISH_STORAGE_DEALS_WALLET $OLD_CONTROL_ADDRESS Copy boostd --vv migrate-markets \ --import-markets-repo=~/.my-markets-repo \ --wallet-publish-storage-deals=$PUBLISH_STORAGE_DEALS_WALLET \ --wallet-deal-collateral=$COLLAT_WALLET \ --max-staging-deals-bytes=50000000000 Copy boostd --vv run Copy ssh -L 8080:localhost:8080 myserver Copy export BOOST_API_INFO=: boostd auth api-info -perm auth sun-brightdesktopmoon --- # DAG store | v1.x (Deprecated) | Boost [hashtag](https://boost.filecoin.io/v1/boost-architecture/dagstore#directory-structure) Directory structure ---------------------------------------------------------------------------------------------------------------- By default, the dagstore root will be: * `$BOOST_PATH/dagstore` The directory structure is as follows: Copy dagstore |___ index # (1) |___ transients # (2) |___ datastore # (3) |___ .shard-registration-complete # (4) |___ .boost-shard-registration-complete # (5) 1. `index`: holds the shard indices. 2. `transients`: holds temporary shard data (unsealed pieces) while they're being indexed. 3. `datastore`: records shard state and metadata so it can survive restarts. 4. `.shard-registration-complete`: marker file that signals that initial migration for legacy markets deals is complete. 5. `.boost-shard-registration-complete`: marker file that signals that initial migration for boost deals is complete. [hashtag](https://boost.filecoin.io/v1/boost-architecture/dagstore#first-time-migration) First-time migration ------------------------------------------------------------------------------------------------------------------ When you first start your boost process without a dagstore repo, a migration process will register all shards for both legacy and Boost deals in **lazy initialization** mode. As deals come in, shards are fetched and initialized just in time to serve the retrieval. * For legacy deals, you can monitor the progress of the migration in your log output, by grepping for the keyword `migrator`. Here's example output. Notice the first line, which specifies how many deals will be evaluated (this number includes failed deals that never went on chain, and therefore will not be migrated), and the last lines (which communicate that migration completed successfully): * For Boost deals, you can do the same by grepping for the keyword `boost-migrator`. [hashtag](https://boost.filecoin.io/v1/boost-architecture/dagstore#forcing-bulk-initialization) Forcing bulk initialization -------------------------------------------------------------------------------------------------------------------------------- Forcing bulk initialization will become important in the near future, when miners begin publishing indices to the network to advertise content they have, and new retrieval features become available (e.g. automatic shard routing). Initialization places IO workload on your storage system. You can stop/start this command at your wish/convenience as proving deadlines approach and elapse, to avoid IOPS starvation or competition with window PoSt. To stop a bulk initialization(see the next paragraph), press Control-C. Shards being initialized at that time will continue in the background, but no more initializations will be performed. The next time you run the command, it will resume from where it left off. You can force bulk initialization using the `boostd dagstore initialize-all` command. This command will force initialization of every shard that is still in `ShardStateNew` state for both legacy and Boost deals. To control the operation: * You must set a concurrency level through the `--concurrency=N` flag. * A value of `0` will disable throttling and all shards will be initialized at once. ⚠️ Use with caution! * By default, only unsealed pieces will be indexed to avoid forcing unsealing jobs. To index also sealed pieces, use the `--include-sealed` flag. In our test environments, we found the migration to proceed at a rate of 400-500 shards/deals per second, on the following hardware specs: AMD Ryzen Threadripper 3970X, 256GB DDR4 3200 RAM, Samsung 970 EVO 2TB SSD, RTX3080 10GB GPU. [hashtag](https://boost.filecoin.io/v1/boost-architecture/dagstore#configuration) Configuration ---------------------------------------------------------------------------------------------------- The DAG store can be configured through the `config.toml` file of the node that runs the _boost_ subsystem. Refer to the `[DAGStore]` section. Boost ships with sane defaults: [hashtag](https://boost.filecoin.io/v1/boost-architecture/dagstore#automatic-shard-recovery-on-error) Automatic shard recovery on error -------------------------------------------------------------------------------------------------------------------------------------------- Shards can error for various reasons, e.g. if the storage system cannot serve the unsealed CAR for a deal/shard, if the shard index is accidentally deleted, etc. Boost will automatically try to recover failed shards by triggering a recovery once. You can view failed shards by using the `boostd dagstore list-shards` command, and optionally grepping for `ShardStateErrored`. [hashtag](https://boost.filecoin.io/v1/boost-architecture/dagstore#cli-commands) CLI commands -------------------------------------------------------------------------------------------------- The `boostd` executable contains a `dagstore` command with several useful subcommands: * `boostd dagstore list-shards` * `boostd dagstore initialize-shard ` * `boostd dagstore initialize-all --concurrency=10` * `boostd dagstore gc` Refer to the `--help` texts for more information. [PreviousDatabasechevron-left](https://boost.filecoin.io/v1/boost-architecture/getting-started-1) [Nextlibp2p Protocolschevron-right](https://boost.filecoin.io/v1/boost-architecture/libp2p-protocols) Last updated 2 years ago * [Directory structure](https://boost.filecoin.io/v1/boost-architecture/dagstore#directory-structure) * [First-time migration](https://boost.filecoin.io/v1/boost-architecture/dagstore#first-time-migration) * [Forcing bulk initialization](https://boost.filecoin.io/v1/boost-architecture/dagstore#forcing-bulk-initialization) * [Configuration](https://boost.filecoin.io/v1/boost-architecture/dagstore#configuration) * [Automatic shard recovery on error](https://boost.filecoin.io/v1/boost-architecture/dagstore#automatic-shard-recovery-on-error) * [CLI commands](https://boost.filecoin.io/v1/boost-architecture/dagstore#cli-commands) sun-brightdesktopmoon Copy 2021-08-09T22:06:35.701+0300 INFO dagstore.migrator dagstore/wrapper.go:286 registering shards for all active deals in sealing subsystem {"count": 453} 2021-08-09T22:06:35.701+0300 WARN dagstore.migrator dagstore/wrapper.go:335 deal has nil piece CID; skipping {"deal_id": 0} 2021-08-09T22:06:35.701+0300 INFO dagstore.migrator dagstore/wrapper.go:348 registering deal in dagstore with lazy init {"deal_id": 2208881, "piece_cid": "baga6ea4seaqhnvxy55e nveknyqhkkh7mltcrrcx35yvuxdmcbfouaafkvp6niay"} 2021-08-09T22:06:35.702+0300 INFO dagstore.migrator dagstore/wrapper.go:318 async shard registration completed successfully {"shard_key": "baga6ea4seaqhnvxy55enveknyqhkkh7mltcrrcx 35yvuxdmcbfouaafkvp6niay"} [...] 2021-08-09T22:06:35.709+0300 INFO dagstore.migrator dagstore/wrapper.go:361 finished registering all shards {"total": 44} [...] 2021-08-09T22:06:35.826+0300 INFO dagstore.migrator dagstore/wrapper.go:365 confirmed registration of all shards 2021-08-09T22:06:35.826+0300 INFO dagstore.migrator dagstore/wrapper.go:372 successfully marked migration as complete 2021-08-09T22:06:35.826+0300 INFO dagstore.migrator dagstore/wrapper.go:375 dagstore migration complete Copy [DAGStore] # Path to the dagstore root directory. This directory contains three # subdirectories, which can be symlinked to alternative locations if # need be: # - ./transients: caches unsealed deals that have been fetched from the # storage subsystem for serving retrievals. # - ./indices: stores shard indices. # - ./datastore: holds the KV store tracking the state of every shard # known to the DAG store. # Default value: /dagstore # RootDir = "" # The maximum amount of indexing jobs that can run simultaneously. # 0 means unlimited. # Default value: 5. # # type: int # MaxConcurrentIndex = 5 # The maximum amount of unsealed deals that can be fetched simultaneously # from the storage subsystem. 0 means unlimited. # Default value: 0 (unlimited). # # type: int # MaxConcurrentReadyFetches = 0 # The maximum number of simultaneous inflight API calls to the storage # subsystem. # Default value: 100. # # type: int # MaxConcurrencyStorageCalls = 100 # The time between calls to periodic dagstore GC, in time.Duration string # representation, e.g. 1m, 5m, 1h. # Default value: 1 minute. # # type: Duration # GCInterval = "1m" sun-brightdesktopmoon --- # Getting started | v1.x (Deprecated) | Boost The Boost source code repository is hosted at [github.com/filecoin-project/boostarrow-up-right](https://github.com/filecoin-project/boost) ### [hashtag](https://boost.filecoin.io/v1/getting-started#boost-and-lotus-compatibility-matrix) Boost and Lotus compatibility Matrix Boost Version Lotus Version Golang Version v1.5.0 v1.18.0 1.18.x v1.5.1, v1.5.2, v1.5.3 v1.18.0, v1.19.0 1.18.x v1.6.0, v1.6.1, v1.6.2-rc1 v1.20.x 1.18.x v1.6.3, v1.6.4 v1.22.x 1.18.x v1.6.2-rc2, v1.7.0-rc1 v1.21.0-rc1, v1.21.0-rc2 1.20.x v1.7.0, v1.7.1, v1.7.2 v1.7.3, v1.7.4 v1.23.x 1.20.x ### [hashtag](https://boost.filecoin.io/v1/getting-started#building-and-installing) Building and installing #### [hashtag](https://boost.filecoin.io/v1/getting-started#prerequisites) Prerequisites circle-exclamation Please make sure you have installed: **Go** - following [https://go.dev/learn/arrow-up-right](https://go.dev/learn/) **Rust** - following [https://www.rust-lang.org/tools/installarrow-up-right](https://www.rust-lang.org/tools/install) **Node 16.x** #### [hashtag](https://boost.filecoin.io/v1/getting-started#environment-variables-in-boost) **Environment Variables in Boost** Copy BOOST_CLIENT_REPO - repo directory for Boost client BOOSTER_BITSWAP_REP - repo directory for Booster bitswap BOOST_PATH - boost repo path FULLNODE_API_INFO - Lotus daemon node API connection string **Linux / Ubuntu** **MacOS** #### [hashtag](https://boost.filecoin.io/v1/getting-started#linux) Linux circle-exclamation Depending on your architecture, you will want to export additional environment variables: Please ignore any output or onscreen instruction during the `npm build` unless there is an error. #### [hashtag](https://boost.filecoin.io/v1/getting-started#macos) MacOS Please ignore any output or onscreen instruction during the `npm build` unless there is an error. #### [hashtag](https://boost.filecoin.io/v1/getting-started#calibration-network) **Calibration Network** To build boost for calibnet, please complete the above pre-requisites and build using the following commands. ### [hashtag](https://boost.filecoin.io/v1/getting-started#upgrading-boost) Upgrading Boost #### [hashtag](https://boost.filecoin.io/v1/getting-started#linux-1) Linux 1\. Make sure that Boost daemon is not running. Run the below commands to upgrade the binary. 2\. Please ignore any onscreen instruction during the `npm build` unless there is an error. 3\. Start the boost daemon. #### [hashtag](https://boost.filecoin.io/v1/getting-started#macos-1) MacOS 1\. Make sure that Boost daemon is not running. Run the below commands to upgrade the binary. 2\. Please ignore any onscreen instruction during the `npm build` unless there is an error. 3\. Start the boost daemon. [PreviousHardware requirementschevron-left](https://boost.filecoin.io/v1/hardware-requirements) [NextAs a storage providerchevron-right](https://boost.filecoin.io/v1/getting-started/getting-started) Last updated 2 years ago * [Boost and Lotus compatibility Matrix](https://boost.filecoin.io/v1/getting-started#boost-and-lotus-compatibility-matrix) * [Building and installing](https://boost.filecoin.io/v1/getting-started#building-and-installing) * [Upgrading Boost](https://boost.filecoin.io/v1/getting-started#upgrading-boost) sun-brightdesktopmoon Copy curl -sL https://deb.nodesource.com/setup_16.x | sudo -E bash - sudo apt-get install -y nodejs sudo apt install mesa-opencl-icd ocl-icd-opencl-dev gcc git bzr jq pkg-config curl clang build-essential hwloc libhwloc-dev wget -y Copy brew install node@16 brew install bzr jq pkg-config hwloc coreutils Copy export RUSTFLAGS="-C target-cpu=native -g" export FFI_BUILD_FROM_SOURCE=1 Copy git clone https://github.com/filecoin-project/boost cd boost git checkout make clean build sudo make install Copy export LIBRARY_PATH=$LIBRARY_PATH:/opt/homebrew/lib git clone https://github.com/filecoin-project/boost cd boost git checkout make clean build sudo make install Copy git clone https://github.com/filecoin-project/boost cd boost git checkout make clean calibnet Copy cd boost git checkout main git pull git checkout make clean build sudo make install Copy export LIBRARY_PATH=$LIBRARY_PATH:/opt/homebrew/lib cd boost git checkout main git pull git checkout make clean build sudo make install sun-brightdesktopmoon --- # GraphQL API | v1.x (Deprecated) | Boost Boost exposes a GraphQL API that is used by the Web UI to query and update information about Boost deals. The GraphQL API query endpoint is at [http://localhost:8080/graphql/queryarrow-up-right](http://localhost:8080/graphql/query) You can also run your own queries against the GraphQL API using CURL or a programming language that has a [GraphQL clientarrow-up-right](https://graphql.org/code/) . Boost has a built-in GraphQL explorer at [http://localhost:8080/graphiqlarrow-up-right](http://localhost:8080/graphiql) You can test out queries, or explore the GraphQL API by clicking on the `< Docs` link at the top right of the page: ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F968384846-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F01F3nbbmdopOdEHfj4fF%252Fuploads%252FFUl9bAj4CyUT847oNQQH%252FScreen%2520Shot%25202022-04-21%2520at%252010.06.16%2520AM.png%3Falt%3Dmedia%26token%3D861edbcf-884a-4f55-8873-615983aadfe4&width=768&dpr=3&quality=100&sign=7cd885f5&sv=2) To run a graphql query with CURL: This 1m video shows how to use these tools to build an run a GraphQL query against Boost: Use the GraphQL explorer to create a query against Boost ### [hashtag](https://boost.filecoin.io/v1/graphql-api#example-queries) Example Queries 1\. Query failed deals 2\. Cancel a deal where `ab12345c-5678-90de-12f3-45a6b78cd9ef` is the deal ID [PreviousInitialisationchevron-left](https://boost.filecoin.io/v1/experimental-features/local-index-directory/initialisation) [NextJSON-RPC APIchevron-right](https://boost.filecoin.io/v1/json-rpc-api) Last updated 2 years ago sun-brightdesktopmoon Copy curl -X POST -H "Content-Type: application/json" -d '{"query":"query { deals(offset: 5, limit: 10) { deals { ID CreatedAt PieceCid } } }"}' http://localhost:8080/graphql/query | jq Copy curl -X POST \ -H "Content-Type: application/json" \ -d '{"query":"query { deals(limit: 10, query: \"failed to get size of imported\") { deals { ID CreatedAt Message } } }"}' \ http://localhost:8080/graphql/query | jq Copy curl -X POST \ -H "Content-Type: application/json" \ -d '{"query":"mutation { dealCancel(id: \"ab12345c-5678-90de-12f3-45a6b78cd9ef\") }"}' \ http://localhost:8080/graphql/query | jq sun-brightdesktopmoon --- # Advanced Configuration of booster-http | Boost triangle-exclamation Before proceeding any further, we suggest you read about the [basics of HTTP retrieval with booster-http](https://boost.filecoin.io/retrieving-data-from-filecoin/http-retrieval) . This section is an extension of HTTP retrievals and deals with advanced configuration options. [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#configuration-options) Configuration options ----------------------------------------------------------------------------------------------------------------------------------------------- `booster-http` is an independent process that can be run on the same machine as `boostd` or on a different machine. Multiple instances can also be run with different listen addresses if required. Copy --pprof run pprof web server on localhost:6070 (default: false) --base-path value the base path at which to run the web server --address value, --addr value the listen address for the web server (default: "0.0.0.0") --port value the port the web server listens on (default: 7777) --api-lid value the endpoint for the local index directory API, eg 'http://localhost:8042' --add-index-throttle value the maximum number of add index operations that can run in parallel (default: 4) --api-fullnode value the endpoint for the full node API --api-storage value [ --api-storage value ] the endpoint for the storage node API --serve-pieces enables serving raw pieces (default: true) --serve-cars serve CAR files with the Trustless IPFS Gateway API (default: true) --compression-level value compression level to use for responses, 0-9, 0 is no compression, 9 is maximum compression; set to 0 to disable if using a reverse proxy with compression (default: 1) --log-file value path to file to append HTTP request and error logs to, defaults to stdout (-) (default: "-") --tracing enables tracing of booster-http calls (default: false) --tracing-endpoint value the endpoint for the tracing exporter (default: "http://tempo:14268/api/traces") --api-filter-endpoint value the endpoint to use for fetching a remote retrieval configuration for requests --api-filter-auth value value to pass in the authorization header when sending a request to the API filter endpoint (e.g. 'Basic ~base64 encoded user/pass~' --badbits-denylists value [ --badbits-denylists value ] the endpoints for fetching one or more custom BadBits list instead of the default one at https://badbits.dwebops.pub/denylist.json (default: "https://badbits.dwebops.pub/denylist.json") --help, -h show help ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#booster-http-boostd-and-lotus) booster-http, boostd and lotus `booster-http` must have network, or localhost access to a full `lotus` node, a `lotus-miner` and a `LID` instance. The following options are required: * `--api-lid` - the Local Index Directory (_LID_) service endpoint * `--api-fullnode` - the `lotus` full node API endpoint, discoverable by running `lotus auth api-info --perm=admin` * `--api-storage` - the `lotus-miner` API endpoint, discoverable by running `lotus-miner auth api-info --perm=admin` ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#address-and-port) Address and port `--address` and `--port` configure the listen address and port of the `booster-http` server. By default HTTP server will listen on `0.0.0.0:7777`. This can be set to other addresses and ports as required, e.g. `127.0.0.1` to serve localhost-only if running a reverse-proxy on the same server with a public listen address. ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#pieces-and-cars) Pieces and CARs `--serve-pieces` is enabled by default and allows retrieval of raw pieces on the `/piece/` endpoint of your `booster-http` server. Requests for pieces require the full piece CID appended to `/piece/`. Piece retrieval is typically performed to replicate deals, or by clients that are able to decode raw piece data. `--serve-cars` is enabled by default and allows [IPFS Trustless Gatewayarrow-up-right](https://specs.ipfs.tech/http-gateways/trustless-gateway/) retrievals on the `/ipfs/` endpoint. This is not a full "trusted" gateway, and requests must either ask for CARs containing one or more blocks from a root CID, or raw block bytes for a single CID. Requests can either pass an `Accept: application/vnd.ipld.car` header, or a `?format=car` query parameter for CAR data. Or, to download raw single IPLD block bytes, either an `Accept: application/vnd.ipld.raw` header, or a `?format=raw` query parameter. A trustless retrieval client is recommended for performing and verifying retrievals from `booster-http`. See [Lassiearrow-up-right](https://github.com/filecoin-project/lassie) for more information. Providing Lassie with the `--provider http://{booster-http exposed url}` will perform verified, trustless retrievals to your `booster-http` instance. ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#badbits-denylist) BadBits denylist `booster-http` (and `booster-bitswap`) automatically filter out known flagged content using the denylist maintained at [https://badbits.dwebops.pub/denylist.jsonarrow-up-right](https://badbits.dwebops.pub/denylist.json) . Use one or more `--badbits-denylists` flags to point to a custom, valid BadBits denylist and override the default. ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#compression) Compression By default, `booster-http` will compress responses with gzip compression. `--compression-level` can be set between values of `0` and `9`, where `0` is no compression and `9` is maximum compression. The default value is `1`, which optimises for speed over compression ratio but this may be increased if required. Compression can be disabled by setting `--compression-level 0`. If you are running a reverse proxy, such as NGINX, in front of `booster-http` that performs compression, you should disable compression in `booster-http` to avoid double-compression. ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#logging) Logging `booster-http` logs HTTP requests and errors to stdout by default. This can be overridden with `--log-file` to log to a file instead. The log file format is similar to typical NGINX or Apache log file formats and is suitable for ingestion into log aggregation tools such as Splunk or ELK. The format is as follows: Where the elements are: 1. RFC 3339 timestamp 2. Remote address 3. HTTP Method 4. Request path 5. Response status code 6. Response duration (in milliseconds) 7. Response size 8. Compression ratio (or `-` if no compression) 9. Remote user agent 10. Error (or `""` if no error) When using a reverse proxy, log output from the reverse proxy may be more suitable for storage and analysis than `booster-http` logging. [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#trusted-http-gateway-setup) Trusted HTTP Gateway Setup --------------------------------------------------------------------------------------------------------------------------------------------------------- Where a Storage Provider wishes to serve plain files, including streaming video, audio and other media, a trusted HTTP gateway can be used to translate `booster-http`'s trustless data into trusted data. This is a separate process that must be configured to communicate with `booster-http`. Be aware that trusted HTTP responses typically do not count as "successful" retrievals by reputation systems and other retrieval checkers. It is also not possible to use trusted HTTP gateways to retrieve CAR files, which are required for verified retrievals by clients such as [Lassiearrow-up-right](https://github.com/filecoin-project/lassie) which is a recommended client for Filecoin downloads. When enabling trusted HTTP gateways, it is recommended to also enable the trustless CAR gateway to allow CAR retrievals; this includes via reverse proxies (see below). ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#bifrost-gateway) bifrost-gateway [bifrost-gatewayarrow-up-right](https://github.com/ipfs/bifrost-gateway) is an IPFS Trusted Gateway server that can be used to translate `booster-http`'s trustless data into trusted data. bifrost-gateway is a separate process that must be configured to communicate with `booster-http`. When running `bifrost-gateway`, two environment variables must be set: * `PROXY_GATEWAY_URL=http://{booster-http exposed url}` to point to the `booster-http` address (without path) * `GRAPH_BACKEND=true` to instruct `bifrost-gateway` to perform full CAR retrievals rather than individual IPLD block retrievals for efficiency Additionally, `--gateway-port` may be used to override the default listen port of `8081`. A reverse proxy can be configured to talk to `bifrost-gateway`, but be aware that IPFS gateway's are typically exposed on the `/ipfs/` endpoint, which is also the endpoint of the trustless gateway which is required for standard Filecoin retrievals (e.g. using [Lassiearrow-up-right](https://github.com/filecoin-project/lassie) ). A reverse proxy combining both the `booster-http` trustless endpoint and a `bifrost-gateway` trusted endpoint must be configured to route `/ipfs/` requests to `booster-http` where the `Accept` header contains `application/vnd.ipld.car` or `application/vnd.ipld.raw`, and `/ipfs/` requests to `bifrost-gateway` where the `Accept` header contains anything else, such as `text/html` or `*/*`. Alternatively, separate reverse proxies may be configured for both `booster-http` and `bifrost-gateway`. [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#reverse-proxy-setup) Reverse Proxy Setup ------------------------------------------------------------------------------------------------------------------------------------------- Storage Providers should secure their `booster-http` before exposing it to the public. Storage Providers may use any tool available to limit who can download files, the number of requests per second, and the download bandwidth each client can use per second. [NGNIXarrow-up-right](https://nginx.org/en/docs/) is one such reverse proxy which may be used in front of a `booster-http` instance. This section provides only a basic coverage of the ways in which NGINX can set access limits, rate limits and bandwidth limits. In particular it’s possible to add limits by request token, or using JWT tokens. The examples in this section are adapted from [Deploying NGINX as an API Gatewayarrow-up-right](http://nginx.com/blog/deploying-nginx-plus-as-an-api-gateway-part-1) which goes into more detail. By default NGINX puts configuration files into `/etc/nginx` The default configuration file is `/etc/nginx/sites-available/default` ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#setup-server-block) Setup server block In this example, we are setting up an NGINX server listen on port `7575` and forward requests to `booster-http` on port `7777`. The IPFS Trustless Gateway serves content from `/ipfs/` and the piece retrieval endpoint is `/piece/`. A `location` block that matches both of these paths will forward requests to `booster-http`. Alternatively, to only forward `/ipfs/` requests to `booster-http` our `location` directive can be simplified: ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#limiting-access) Limiting Access We can limit access to the IPFS gateway using the standard `.htaccess` file. This file contains usernames and passwords. In this example we create a user named `_alice_`: Include the `.htaccess` file in the `/etc/nginx/sites-available/default` Now when we open any URL under the path `/ipfs/` we will be presented with a _Sign-in_ dialog. ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252Fx9v2pwtb5KqMwdSHJidP%252Fsecure_login.jpg%3Falt%3Dmedia%26token%3D09957b9e-b5f5-4a45-9573-99523c5b6353&width=768&dpr=3&quality=100&sign=bd54b9d4&sv=2) Login prompt when accessing \`booster-http\` url ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#rate-limiting) Rate Limiting To prevent users from making too many requests per second, we can add rate limits. 1. Create a file with the rate limiting configuration at `/etc/nginx/ipfs-gateway.conf.d/ipfs-gateway.conf` 2. Add a request zone limit to the file of 1 request per second, per client IP 1. Include `ipfs-gateway.conf` in `/etc/nginx/sites-available/default` and set the response for too many requests to _HTTP response code 429_ 1. Click the refresh button in your browser on any path under `/ipfs/` more than once per second you will see a 429 error page. ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2Flh6.googleusercontent.com%2Fd-wTXF49wBDA8q-RiTRmZ7tAW4ZnH8nNRvpMxilJwr9hgyhtRemO5cceBfveUqsDUYEA1fINOyEOrHKVX0eL4WS8U9FjoKzYNNjef6OZl_FlrlQsg-slRftPCClEJLq9MxvF7EIwQKgfhJRVKx2Fz1M&width=768&dpr=3&quality=100&sign=aafb5c7&sv=2) HTTP error 429 ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#bandwidth-limiting) Bandwidth Limiting It is also recommended to limit the amount of bandwidth that clients can take up when downloading data from `booster-http`. This ensures a fair bandwidth distribution to each client and prevents situations where one client ends up choking the `booster-http` instance. 1. Create a new .htaccess user called `_bob_` 1. Add a mapping from `.htaccess` username to bandwidth limit in `/etc/nginx/ipfs-gateway.conf.d/ipfs-gateway.conf` 1. Add the bandwidth limit to `/etc/nginx/sites-available/default` 1. To verify bandwidth limiting, use `curl` to download a file with user `_alice_` and then `_bob_` Note the difference in the Average Dload column (the average download speed). ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252Feb1dIp0qDwSOtqpqLxn8%252Fdownload_speed.jpg%3Falt%3Dmedia%26token%3D5da681e7-c11a-4ebb-8b38-20041a325ef2&width=768&dpr=3&quality=100&sign=b202d7aa&sv=2) Bandwidth limiting result ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#compression-1) Compression We can configure NGINX's compression settings in its main `http` block, typically in the `/etc/nginx/nginx.conf` file. See the [NGINX gzip module documentationarrow-up-right](https://nginx.org/en/docs/http/ngx_http_gzip_module.html) for more information. ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#https) HTTPS NGINX can be configured to serve HTTPS traffic. This is recommended for production deployments. See [NGINX HTTPS configurationarrow-up-right](https://nginx.org/en/docs/http/configuring_https_servers.html) for more information. ### [hashtag](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#caching) Caching NGINX can be configured to cache responses from `booster-http`. Since `booster-http` serves content addressed data that does not change, this is particularly well suited to caching in cases where certain content is frequently requested. `booster-http` sets a long `Cache-Control` header by default, so NGINX will cache responses for a long time by default. See the [NGINX proxy modulearrow-up-right](https://nginx.org/en/docs/http/ngx_http_proxy_module.html) documentation for more information on how to configure caching. [PreviousHTTP retrievalchevron-left](https://boost.filecoin.io/retrieving-data-from-filecoin/http-retrieval) [NextBitswap retrievalchevron-right](https://boost.filecoin.io/retrieving-data-from-filecoin/bitswap-retrieval) Last updated 2 years ago * [Configuration options](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#configuration-options) * [booster-http, boostd and lotus](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#booster-http-boostd-and-lotus) * [Address and port](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#address-and-port) * [Pieces and CARs](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#pieces-and-cars) * [BadBits denylist](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#badbits-denylist) * [Compression](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#compression) * [Logging](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#logging) * [Trusted HTTP Gateway Setup](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#trusted-http-gateway-setup) * [bifrost-gateway](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#bifrost-gateway) * [Reverse Proxy Setup](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#reverse-proxy-setup) * [Setup server block](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#setup-server-block) * [Limiting Access](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#limiting-access) * [Rate Limiting](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#rate-limiting) * [Bandwidth Limiting](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#bandwidth-limiting) * [Compression](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#compression-1) * [HTTPS](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#https) * [Caching](https://boost.filecoin.io/retrieving-data-from-filecoin/advanced-http-configuration#caching) sun-brightdesktopmoon Copy %s %s %s "%s" %d %d %d %s "%s" "%s" Copy PROXY_GATEWAY_URL=http://localhost:7777 GRAPH_BACKEND=true bifrost-gateway Copy curl --output /tmp/museum.jpg "http://localhost:8081/ipfs/bafybeidqindpi4ucx7kmrtnw3woc6jtl7bqvyiokrkpbbuy6gs6trn57tm/vincent/Vincent%20van%20Gogh_files/Caf%C3%A9tafel_met_absint_-_s0186V1962_-_Van_Gogh_Museum.jpg" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 11830 100 11830 0 0 140k 0 --:--:-- --:--:-- --:--:-- 175k open /tmp/museum.jpg Copy # ipfs gateway config server { listen 7575 default_server; listen [::]:7575 default_server; location ~* ^/(ipfs|piece)/ { proxy_pass http://127.0.0.1:7777; } } Copy location /ipfs/ { proxy_pass http://127.0.0.1:7777; } Copy mkdir /etc/nginx/ipfs-gateway.conf.d htpasswd -c /etc/nginx/ipfs-gateway.conf.d/.htpasswd alice New password: Re-type new password: Adding password for user alice Copy # ipfs gateway config server { listen 7575 default_server; listen [::]:7575 default_server; location ~* ^/(ipfs|piece)/ { # htaccess authentication auth_basic "Restricted Server"; auth_basic_user_file /etc/nginx/ipfs-gateway.conf.d/.htpasswd; proxy_pass http://127.0.0.1:7777; } } Copy limit_req_zone $binary_remote_addr zone=client_ip_10rs:1m rate=1r/s; Copy include /etc/nginx/ipfs-gateway.conf.d/ipfs-gateway.conf; server { listen 7575 default_server; listen [::]:7575 default_server; location ~* ^/(ipfs|piece)/ { # htaccess authentication auth_basic "Restricted Server"; auth_basic_user_file /etc/nginx/ipfs-gateway.conf.d/.htpasswd; limit_req zone=client_ip_10rs; limit_req_status 429; proxy_pass http://127.0.0.1:7777; } } Copy htpasswd /etc/nginx/ipfs-gateway.conf.d/.htpasswd bob Copy map $remote_user $bandwidth_limit { default 1k; "alice" 10k; "bob" 512k; } Copy include /etc/nginx/ipfs-gateway.conf.d/ipfs-gateway.conf; server { listen 7575 default_server; listen [::]:7575 default_server; location ~* ^/(ipfs|piece)/ { # htaccess authentication auth_basic "Restricted Server"; auth_basic_user_file /etc/nginx/ipfs-gateway.conf.d/.htpasswd; limit_rate $bandwidth_limit; limit_req zone=client_ip_10rs; limit_req_status 429; proxy_pass http://127.0.0.1:7777; } } Copy http { ... gzip on; gzip_vary on; # Add Vary: Accept-Encoding header gzip_proxied any; # Compress data for all proxied requests ... } sun-brightdesktopmoon --- # HTTP indexer announcement | v1.x (Deprecated) | Boost [hashtag](https://boost.filecoin.io/v1/configuration/http-indexer-announcement#announce-over-http) Announce over HTTP -------------------------------------------------------------------------------------------------------------------------- `IndexProvider.HttpPublisher.AnnounceOverHttp` must be set to `true` to enable the http announcements. Once HTTP announcements are enabled, the local-index provider will continue to announce over libp2p gossipsub along with HTTP for the specific indexers. The advertisements are send to the indexer nodes defined in `DirectAnnounceURLs`. You can specify more than 1 URL to announce to multiple indexer nodes. Once an IPNI node starts processing the advertisements, it will reach out to the Boost node to fetch the data. Thus, Boost node needs to specify a public IP and port which can be used by the indexer node to query for data. Copy [IndexProvider] # Enable set whether to enable indexing announcement to the network and expose endpoints that # allow indexer nodes to process announcements. Enabled by default. # # type: bool # env var: LOTUS_INDEXPROVIDER_ENABLE #Enable = true # EntriesCacheCapacity sets the maximum capacity to use for caching the indexing advertisement # entries. Defaults to 1024 if not specified. The cache is evicted using LRU policy. The # maximum storage used by the cache is a factor of EntriesCacheCapacity, EntriesChunkSize and # the length of multihashes being advertised. For example, advertising 128-bit long multihashes # with the default EntriesCacheCapacity, and EntriesChunkSize means the cache size can grow to # 256MiB when full. # # type: int # env var: LOTUS_INDEXPROVIDER_ENTRIESCACHECAPACITY #EntriesCacheCapacity = 1024 # EntriesChunkSize sets the maximum number of multihashes to include in a single entries chunk. # Defaults to 16384 if not specified. Note that chunks are chained together for indexing # advertisements that include more multihashes than the configured EntriesChunkSize. # # type: int # env var: LOTUS_INDEXPROVIDER_ENTRIESCHUNKSIZE #EntriesChunkSize = 16384 # TopicName sets the topic name on which the changes to the advertised content are announced. # If not explicitly specified, the topic name is automatically inferred from the network name # in following format: '/indexer/ingest/' # Defaults to empty, which implies the topic name is inferred from network name. # # type: string # env var: LOTUS_INDEXPROVIDER_TOPICNAME #TopicName = "" # PurgeCacheOnStart sets whether to clear any cached entries chunks when the provider engine # starts. By default, the cache is rehydrated from previously cached entries stored in # datastore if any is present. # # type: bool # env var: LOTUS_INDEXPROVIDER_PURGECACHEONSTART #PurgeCacheOnStart = true [IndexProvider.Announce] # Make a direct announcement to a list of indexing nodes over http. # Note that announcements are already made over pubsub regardless # of this setting. # # type: bool # env var: LOTUS_INDEXPROVIDER_ANNOUNCE_ANNOUNCEOVERHTTP #AnnounceOverHttp = true # The list of URLs of indexing nodes to announce to. # # type: []string # env var: LOTUS_INDEXPROVIDER_ANNOUNCE_DIRECTANNOUNCEURLS #DirectAnnounceURLs = ["https://cid.contact/ingest/announce"] [IndexProvider.HttpPublisher] # If not enabled, requests are served over graphsync instead. # # type: bool # env var: LOTUS_INDEXPROVIDER_HTTPPUBLISHER_ENABLED #Enabled = true # Set the public hostname / IP for the index provider listener. # eg "82.129.73.111" # This is usually the same as the for the boost node. # # type: string # env var: LOTUS_INDEXPROVIDER_HTTPPUBLISHER_PUBLICHOSTNAME #PublicHostname = "82.129.73.111" # Set the port on which to listen for index provider requests over HTTP. # Note that this port must be open on the firewall. # # type: int # env var: LOTUS_INDEXPROVIDER_HTTPPUBLISHER_PORT #Port = 3401 [PreviousLegacy Deal configurationchevron-left](https://boost.filecoin.io/v1/configuration/legacy-deal-configuration) [NextBackup and Restorechevron-right](https://boost.filecoin.io/v1/backup-and-restore) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Setting up a monitoring stack for Boost | v1.x (Deprecated) | Boost [hashtag](https://boost.filecoin.io/v1/tutorials/setting-up-a-monitoring-stack-for-boost#docs-internal-guid-09f21b17-7fff-fabd-2034-91b1f0465aa1) Background ----------------------------------------------------------------------------------------------------------------------------------------------------------------- The monitoring stack we will use includes: * Prometheus - collects metrics and powers dashboards in Grafana * Tempo - collects traces and powers traces search in Grafana with Jaeger * Grafana - provides visualization tools and dashboards for all metrics and traces Lotus and Boost are already instrumented to produce traces and stats for Prometheus to collect. The Boost team also packages a set of Grafana dashboards that are automatically provisioned as part of this setup. [hashtag](https://boost.filecoin.io/v1/tutorials/setting-up-a-monitoring-stack-for-boost#prerequisites) Prerequisites -------------------------------------------------------------------------------------------------------------------------- This setup has been tested on macOS and on Linux. We haven’t tested it on Windows, so YMMV. All the monitoring stack containers run in Docker. [hashtag](https://boost.filecoin.io/v1/tutorials/setting-up-a-monitoring-stack-for-boost#steps) Steps ---------------------------------------------------------------------------------------------------------- 1. ### [hashtag](https://boost.filecoin.io/v1/tutorials/setting-up-a-monitoring-stack-for-boost#install-docker) Install Docker We have tested this setup with Docker 20.10.23 on macOS and Ubuntu. [https://docs.docker.com/engine/install/arrow-up-right](https://docs.docker.com/engine/install/) 1. ### [hashtag](https://boost.filecoin.io/v1/tutorials/setting-up-a-monitoring-stack-for-boost#dns-resolution-for-prometheus) DNS resolution for Prometheus Update `extra_hosts` in `docker-compose.yaml` for `prometheus`, so that the Prometheus container can reach all its targets - `boostd`, `lotus-miner`, `booster-bitswap`, `booster-http`, etc. [https://github.com/filecoin-project/boost/blob/main/docker/monitoring/docker-compose.yaml#L47-L55arrow-up-right](https://github.com/filecoin-project/boost/blob/main/docker/monitoring/docker-compose.yaml#L47-L55) Depending on where your Filecoin processes (`boostd`, `lotus`, `lotus-miner`, `booster-bitswap`, etc.) are running, you need to confirm that they are reachable from Prometheus so that it can scrape their metrics. By default the setup expects to find them within the same Docker network, so if you are running them elsewhere (i.e. on the \`host\` network), add the following arguments: 1. ### [hashtag](https://boost.filecoin.io/v1/tutorials/setting-up-a-monitoring-stack-for-boost#prometheus-targets) Prometheus targets Confirm that Prometheus targets are scraped at [http://localhost:9090arrow-up-right](http://localhost:9090/) / Targets ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2Flh4.googleusercontent.com%2FsUGTMb0S5lp2sog0DdG9ecSRmCPDtMSmyAIwQ-k-91Pw1IUahhCiudKXkmKUJTHajXrPMR6RGLMmFEU9szwRlvcf9B4DgAIBuAaLTPB7-XaGPvtCrg2A9XGdhDIaTxZYi5wdbpTLdpsC98YcrorNVA2R1K7NLlK2xlk6RJRZ0jJ6qZ28QW543N0Imsrbzg&width=300&dpr=3&quality=100&sign=529b6595&sv=2) If you are running software firewall like \`ufw\`, you might need to modify your iptables and allow access from the Prometheus container / network to the Filecoin stack network, for example: `sudo docker network inspect monitoring # note the Subnet for the network sudo ufw allow from 172.18.0.0/16` 1. ### [hashtag](https://boost.filecoin.io/v1/tutorials/setting-up-a-monitoring-stack-for-boost#grafana-dashboards) Grafana dashboards Go to Grafana at [http://localhost:3333arrow-up-right](http://localhost:3333/) and inspect the dashboards: ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2Flh4.googleusercontent.com%2FUat6-bsTHiMvzwm854YT2W9M7fN5uiqdvQ8oFs1WsWqn98716AO56VCWD8Pok98W019mrjZVRM6fFv2qZ_hY18OR0GmCMa5eVR45mh0OT8ZbX47HT3_lLhuEvNt3CRy9eJ-t2O0De2hf3UDOsYERCgSaDR_atH9czc1gkdUbo6dDC5ZYP75xaLJugAo8VQ&width=300&dpr=3&quality=100&sign=3c2be738&sv=2) [PreviousUsing filters for storage and retrieval dealschevron-left](https://boost.filecoin.io/v1/tutorials/using-filters-for-storage-and-retrieval-deals) [NextBitswap Retrievalchevron-right](https://boost.filecoin.io/v1/bitswap-retrieval) Last updated 2 years ago * [Background](https://boost.filecoin.io/v1/tutorials/setting-up-a-monitoring-stack-for-boost#docs-internal-guid-09f21b17-7fff-fabd-2034-91b1f0465aa1) * [Prerequisites](https://boost.filecoin.io/v1/tutorials/setting-up-a-monitoring-stack-for-boost#prerequisites) * [Steps](https://boost.filecoin.io/v1/tutorials/setting-up-a-monitoring-stack-for-boost#steps) sun-brightdesktopmoon sun-brightdesktopmoon --- # Troubleshooting | v1.x (Deprecated) | Boost [hashtag](https://boost.filecoin.io/v1/troubleshooting#inspect) Inspect ---------------------------------------------------------------------------- The new inspect page in the Boost UI helps with debugging retrieval problems. It allows the user to check the following using a payload CID or piece CID: * Verify if the piece has been correctly added to the Piece Store * Validate if the piece is indexed in the DAG store * Check for an unsealed copy of the piece * Verify that the payload CID -> piece CID index has been created correctly [hashtag](https://boost.filecoin.io/v1/troubleshooting#failed-to-connect-to-peer) Failed to connect to peer ---------------------------------------------------------------------------------------------------------------- If the client cannot connect to Boost running on a Storage provider, with an error similar to the following: The problem is that: * The SP registered their peer id and address on chain. eg "Register the peer id `123abcd` at address `ip4/123.456.12.345/tcp/1234`" * The SP changed their peer id locally but didn't update the peer id on chain. * The client wants to make a storage deal with peer `123abcd`. The client looks on chain for the address of peer `123abcd` and sees peer `123abcd` has registered an address `ip4/123.456.12.345/tcp/1234`. * The client sends a deal proposal for peer `123abcd` to the SP at address `ip4/123.456.12.345/tcp/1234`. * The SP has changed their peer ID, so the SP responds to the deal proposal request with an error: `peer id mismatch` To fix the problem, the SP should register the new peer id on chain: [hashtag](https://boost.filecoin.io/v1/troubleshooting#update-storage-providers-on-chain-address) Update storage provider's on chain address ------------------------------------------------------------------------------------------------------------------------------------------------- Clients would not be able to connect to Boost running on a Storage provider after an IP change. This happens as clients lookup the registered peer id and address on chain for a SP. When a SP changes their IP or address locally, they must update the same on chain. The SP should register the new peer id on chain using the following lotus-miner command circle-exclamation Please make sure to use the public IP and port of the Boost node and not `lotus-miner` node if your miner and `boostd` runs on a separate machine. The on chain address change requires access to the worker key and thus the command lives in `lotus-miner` instead of Boost. [hashtag](https://boost.filecoin.io/v1/troubleshooting#error-in-lotus-miner-info-output) Error in _lotus-miner info_ output -------------------------------------------------------------------------------------------------------------------------------- After migrating to Boost, following error is seen when running `lotus-miner info` : #### [hashtag](https://boost.filecoin.io/v1/troubleshooting#problem) Problem: `lotus-miner` is making a call on `lotus-market` process which has been replaced by Boost, but `lotus-miner` is not aware of the new market process. #### [hashtag](https://boost.filecoin.io/v1/troubleshooting#solution) Solution: Export the MARKETS\_API\_INFO variable on your lotus-miner node. [hashtag](https://boost.filecoin.io/v1/troubleshooting#fix-retrievals-with-error-failed-to-lookup-index-for-mh) Fix retrievals with error "failed to lookup index for mh" ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ The following error shows up when trying to retrieve the data from a storage provider. The error indicates that dagstore does not have a corresponding index shard for the piece containing the requested data. When a retrieval is requested, the dagstore on storage provider side is queried and a reverse look up is used to determine the key(piece CID). This key is then used to query the piece store to find the sector containing the data and byte offset. If for any reason the shard is not registered with the dagstore then reverse look up to find the piece CID fails and the above error is seen. The most widely know reason for not having the shard registered with dagstore is the below error. To fix the deals where retrievals are impacted by above error, user will need to register the shards manually with dagstore: If you have multiple deals in such state then you will need to generate a list of registered pieces with piece store and then compare with the shards available in the dagstore to create a list of missing shards. triangle-exclamation Please stop accepting any deals and ensure all current deals are handed off to the lotus-miner (sealer) subsystem before proceeding from here. 1\. Create a list of all sectors on `lotus-miner` and redirect the output to a file. Copy the output file to boost node to be used by the below command. 2\. Generate a list of shards to be registered 3\. Register the shards with dagstore in an automated fashion. Please note that each shard may take upto 3-5 minutes to get registered. So, the above command might take hours or days to complete depending upon the number of missing shards. [PreviousServing files with booster-httpchevron-left](https://boost.filecoin.io/v1/http-retrieval/serving-files-with-booster-http) [NextExperimental Featureschevron-right](https://boost.filecoin.io/v1/experimental-features) Last updated 2 years ago * [Inspect](https://boost.filecoin.io/v1/troubleshooting#inspect) * [Failed to connect to peer](https://boost.filecoin.io/v1/troubleshooting#failed-to-connect-to-peer) * [Update storage provider's on chain address](https://boost.filecoin.io/v1/troubleshooting#update-storage-providers-on-chain-address) * [Error in lotus-miner info output](https://boost.filecoin.io/v1/troubleshooting#error-in-lotus-miner-info-output) * [Fix retrievals with error "failed to lookup index for mh"](https://boost.filecoin.io/v1/troubleshooting#fix-retrievals-with-error-failed-to-lookup-index-for-mh) sun-brightdesktopmoon Copy failed to connect to peer : failed to dial : * failed to negotiate security protocol: peer id mismatch: expected , but remote key matches Copy lotus-miner actor set-peer-id 123abcd lotus-miner actor set-addrs ip4/123.456.12.345/tcp/1234 Copy lotus-miner actor set-peer-id Copy lotus-miner actor set-addrs /ip4//tcp/ Copy ERROR: fatal error calling 'Filecoin.MarketListIncompleteDeals': panic in rpc method 'Filecoin.MarketListIncompleteDeals': runtime error: invalid memory address or nil pointer dereference Copy export MARKETS_API_INFO= Copy ERROR: offer error: retrieval query offer errored: failed to fetch piece to retrieve from: getting pieces for cid Qmf1ykhUo63qB5dJ8KRyeths9MZfyxpVdT5xwnmoLKefz7: getting pieces containing block Qmf1ykhUo63qB5dJ8KRyeths92mfyxpVdT5xi1moLKefz7: failed to lookup index for mh 1220f7ce2d20772b959c1071868e9495712f12785b1710ee88752af120dd49338190, err: datastore: key not found Copy 2022-02-21T20:06:03.950+1100 INFO markets loggers/loggers.go:20 storage provider event {"name": "ProviderEventFailed", "proposal CID": "bafyreihr743zllr2eckgfiweouiap7pgcjqa3mg3t75jjt7sfcpu", "state": "StorageDealError", "message": "error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1570875): failed to look up deal on chain: deal 3964985 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed"} Copy boostd dagstore register-shard Copy lotus-miner sectors list | awk '{print $1 " " $2}' | grep -v ID > aclist.txt Copy comm -13 <(for i in $(boostd pieces list-pieces); do sector_list=`boostd pieces piece-info $i | awk '{print $2}'| sed -ne '/SectorID/,$p' | grep -v SectorID`; for j in $sector_list; do grep -w $j aclist.txt > /dev/null; if [ $? -eq 0 ]; then break; else echo "$i"; fi; done; done) <(comm -13 <(boostd dagstore list-shards | awk '{print $1}' | sed 1d | sort) <(boostd pieces list-pieces | sort)) Copy for i in `cat ` ; do boostd dagstore register-shard $i; done sun-brightdesktopmoon --- # JSON-RPC API | Boost [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#go-json-rpc-client) Go JSON-RPC client -------------------------------------------------------------------------------------------------------------------------- To use the Boost Go client, the Go RPC-API library can be used to interact with the Boost API node. 1. Import the necessary Go module: Copy go get github.com/filecoin-project/go-jsonrpc 1. Create the following script: Copy package main import ( "context" "fmt" "log" "net/http" jsonrpc "github.com/filecoin-project/go-jsonrpc" boostapi "github.com/filecoin-project/boost/api" ) func main() { authToken := "" headers := http.Header{"Authorization": []string{"Bearer " + authToken}} addr := "127.0.0.1:1288" var api boostapi.BoostStruct closer, err := jsonrpc.NewMergeClient(context.Background(), "ws://"+addr+"/rpc/v0", "Filecoin", []interface{}{&api.Internal, &api.CommonStruct.Internal}, headers) if err != nil { log.Fatalf("connecting with boost failed: %s", err) } defer closer() // Now you can call any API you're interested in. netAddrs, err := api.NetAddrsListen(context.Background()) if err != nil { log.Fatalf("calling netAddrsListen: %s", err) } fmt.Printf("Boost is listening on: %s", netAddrs.Addrs[0]) } 1. Run `go mod init` to set up your `go.mod` file 2. You should now be able to interact with the Boost API. [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#python-json-rpc-client) Python JSON-RPC client ---------------------------------------------------------------------------------------------------------------------------------- The JSON-RPC API can also be communicated with programmatically from other languages. Here is an example written in Python. Note that the `method` must be prefixed with `Filecoin.` [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#groups) Groups -------------------------------------------------------------------------------------------------- * [Uncategorized](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#uncategorized) * [Discover](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#discover) * [Auth](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#auth) * [AuthNew](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#authnew) * [AuthVerify](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#authverify) * [Blockstore](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#blockstore) * [BlockstoreGet](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#blockstoreget) * [BlockstoreGetSize](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#blockstoregetsize) * [BlockstoreHas](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#blockstorehas) * [Boost](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boost) * [BoostDeal](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostdeal) * [BoostDealBySignedProposalCid](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostdealbysignedproposalcid) * [BoostDirectDeal](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostdirectdeal) * [BoostDummyDeal](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostdummydeal) * [BoostIndexerAnnounceAllDeals](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostindexerannouncealldeals) * [BoostIndexerAnnounceDeal](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostindexerannouncedeal) * [BoostIndexerAnnounceDealRemoved](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostindexerannouncedealremoved) * [BoostIndexerAnnounceLatest](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostindexerannouncelatest) * [BoostIndexerAnnounceLatestHttp](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostindexerannouncelatesthttp) * [BoostIndexerAnnounceLegacyDeal](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostindexerannouncelegacydeal) * [BoostIndexerListMultihashes](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostindexerlistmultihashes) * [BoostLegacyDealByProposalCid](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostlegacydealbyproposalcid) * [BoostOfflineDealWithData](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostofflinedealwithdata) * [Log](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#log) * [LogList](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#loglist) * [LogSetLevel](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#logsetlevel) * [Market](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#market) * [MarketGetAsk](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#marketgetask) * [Net](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#net) * [ID](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#id) * [NetAddrsListen](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netaddrslisten) * [NetAgentVersion](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netagentversion) * [NetAutoNatStatus](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netautonatstatus) * [NetBandwidthStats](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netbandwidthstats) * [NetBandwidthStatsByPeer](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netbandwidthstatsbypeer) * [NetBandwidthStatsByProtocol](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netbandwidthstatsbyprotocol) * [NetBlockAdd](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netblockadd) * [NetBlockList](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netblocklist) * [NetBlockRemove](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netblockremove) * [NetConnect](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netconnect) * [NetConnectedness](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netconnectedness) * [NetDisconnect](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netdisconnect) * [NetFindPeer](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netfindpeer) * [NetLimit](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netlimit) * [NetPeerInfo](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netpeerinfo) * [NetPeers](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netpeers) * [NetPing](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netping) * [NetProtectAdd](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netprotectadd) * [NetProtectList](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netprotectlist) * [NetProtectRemove](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netprotectremove) * [NetPubsubScores](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netpubsubscores) * [NetSetLimit](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netsetlimit) * [NetStat](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netstat) * [Backup](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#backup) * [OnlineBackup](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#onlinebackup) * [PieceDirectory](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#piecedirectory) * [PdBuildIndexForPieceCid](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#pdbuildindexforpiececid) * [PdCleanup](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#pdcleanup) * [PdRemoveDealForPiece](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#pdremovedealforpiece) [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#undefined) --------------------------------------------------------------------------------------------- [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#uncategorized) Uncategorized ---------------------------------------------------------------------------------------------------------------- ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#discover) Discover Perms: read Inputs: `null` Response: [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#auth) Auth ---------------------------------------------------------------------------------------------- ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#authnew) AuthNew Perms: admin Inputs: Response: `"Ynl0ZSBhcnJheQ=="` ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#authverify) AuthVerify Perms: read Inputs: Response: [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#blockstore) Blockstore ---------------------------------------------------------------------------------------------------------- ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#blockstoreget) BlockstoreGet There are not yet any comments for this method. Perms: read Inputs: Response: `"Ynl0ZSBhcnJheQ=="` ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#blockstoregetsize) BlockstoreGetSize Perms: read Inputs: Response: `123` ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#blockstorehas) BlockstoreHas Perms: read Inputs: Response: `true` [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boost) Boost ------------------------------------------------------------------------------------------------ ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostdeal) BoostDeal Perms: admin Inputs: Response: ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostdealbysignedproposalcid) BoostDealBySignedProposalCid Perms: admin Inputs: Response: ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostdirectdeal) BoostDirectDeal Perms: admin Inputs: Response: ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostdummydeal) BoostDummyDeal Perms: admin Inputs: Response: ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostindexerannouncealldeals) BoostIndexerAnnounceAllDeals There are not yet any comments for this method. Perms: admin Inputs: `null` Response: `{}` ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostindexerannouncedeal) BoostIndexerAnnounceDeal Perms: admin Inputs: Response: `null` ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostindexerannouncedealremoved) BoostIndexerAnnounceDealRemoved Perms: admin Inputs: Response: `null` ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostindexerannouncelatest) BoostIndexerAnnounceLatest Perms: admin Inputs: `null` Response: `null` ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostindexerannouncelatesthttp) BoostIndexerAnnounceLatestHttp Perms: admin Inputs: Response: `null` ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostindexerannouncelegacydeal) BoostIndexerAnnounceLegacyDeal Perms: admin Inputs: Response: `null` ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostindexerlistmultihashes) BoostIndexerListMultihashes Perms: admin Inputs: Response: ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostlegacydealbyproposalcid) BoostLegacyDealByProposalCid Perms: admin Inputs: Response: ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostofflinedealwithdata) BoostOfflineDealWithData Perms: admin Inputs: Response: [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#log) Log -------------------------------------------------------------------------------------------- ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#loglist) LogList Perms: write Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#logsetlevel) LogSetLevel Perms: write Inputs: Response: `{}` [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#market) Market -------------------------------------------------------------------------------------------------- ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#marketgetask) MarketGetAsk Perms: read Inputs: `null` Response: [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#net) Net -------------------------------------------------------------------------------------------- ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#id) ID Perms: read Inputs: `null` Response: `"12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"` ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netaddrslisten) NetAddrsListen Perms: read Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netagentversion) NetAgentVersion Perms: read Inputs: Response: `"string value"` ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netautonatstatus) NetAutoNatStatus Perms: read Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netbandwidthstats) NetBandwidthStats Perms: read Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netbandwidthstatsbypeer) NetBandwidthStatsByPeer Perms: read Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netbandwidthstatsbyprotocol) NetBandwidthStatsByProtocol Perms: read Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netblockadd) NetBlockAdd Perms: admin Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netblocklist) NetBlockList Perms: read Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netblockremove) NetBlockRemove Perms: admin Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netconnect) NetConnect Perms: write Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netconnectedness) NetConnectedness Perms: read Inputs: Response: `1` ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netdisconnect) NetDisconnect Perms: write Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netfindpeer) NetFindPeer Perms: read Inputs: Response: ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netlimit) NetLimit Perms: read Inputs: Response: ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netpeerinfo) NetPeerInfo Perms: read Inputs: Response: ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netpeers) NetPeers Perms: read Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netping) NetPing Perms: read Inputs: Response: `60000000000` ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netprotectadd) NetProtectAdd Perms: admin Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netprotectlist) NetProtectList Perms: read Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netprotectremove) NetProtectRemove Perms: admin Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netpubsubscores) NetPubsubScores Perms: read Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netsetlimit) NetSetLimit Perms: admin Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netstat) NetStat Perms: read Inputs: Response: [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#backup) Backup -------------------------------------------------------------------------------------------------- ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#onlinebackup) OnlineBackup There are not yet any comments for this method. Perms: admin Inputs: Response: `{}` [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#piecedirectory) PieceDirectory ------------------------------------------------------------------------------------------------------------------ ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#pdbuildindexforpiececid) PdBuildIndexForPieceCid There are not yet any comments for this method. Perms: admin Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#pdcleanup) PdCleanup Perms: admin Inputs: `null` Response: `{}` ### [hashtag](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#pdremovedealforpiece) PdRemoveDealForPiece Perms: admin Inputs: Response: `{}` [PreviousGraphQL APIchevron-left](https://boost.filecoin.io/deployment/boost-architecture/graphql-api) [NextSQLite metadata databasechevron-right](https://boost.filecoin.io/deployment/boost-architecture/getting-started-1) Last updated 1 year ago * [Go JSON-RPC client](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#go-json-rpc-client) * [Python JSON-RPC client](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#python-json-rpc-client) * [Groups](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#groups) * [](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#undefined) * [Uncategorized](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#uncategorized) * [Discover](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#discover) * [Auth](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#auth) * [AuthNew](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#authnew) * [AuthVerify](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#authverify) * [Blockstore](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#blockstore) * [BlockstoreGet](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#blockstoreget) * [BlockstoreGetSize](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#blockstoregetsize) * [BlockstoreHas](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#blockstorehas) * [Boost](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boost) * [BoostDeal](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostdeal) * [BoostDealBySignedProposalCid](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostdealbysignedproposalcid) * [BoostDirectDeal](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostdirectdeal) * [BoostDummyDeal](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostdummydeal) * [BoostIndexerAnnounceAllDeals](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostindexerannouncealldeals) * [BoostIndexerAnnounceDeal](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostindexerannouncedeal) * [BoostIndexerAnnounceDealRemoved](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostindexerannouncedealremoved) * [BoostIndexerAnnounceLatest](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostindexerannouncelatest) * [BoostIndexerAnnounceLatestHttp](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostindexerannouncelatesthttp) * [BoostIndexerAnnounceLegacyDeal](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostindexerannouncelegacydeal) * [BoostIndexerListMultihashes](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostindexerlistmultihashes) * [BoostLegacyDealByProposalCid](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostlegacydealbyproposalcid) * [BoostOfflineDealWithData](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#boostofflinedealwithdata) * [Log](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#log) * [LogList](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#loglist) * [LogSetLevel](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#logsetlevel) * [Market](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#market) * [MarketGetAsk](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#marketgetask) * [Net](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#net) * [ID](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#id) * [NetAddrsListen](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netaddrslisten) * [NetAgentVersion](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netagentversion) * [NetAutoNatStatus](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netautonatstatus) * [NetBandwidthStats](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netbandwidthstats) * [NetBandwidthStatsByPeer](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netbandwidthstatsbypeer) * [NetBandwidthStatsByProtocol](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netbandwidthstatsbyprotocol) * [NetBlockAdd](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netblockadd) * [NetBlockList](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netblocklist) * [NetBlockRemove](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netblockremove) * [NetConnect](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netconnect) * [NetConnectedness](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netconnectedness) * [NetDisconnect](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netdisconnect) * [NetFindPeer](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netfindpeer) * [NetLimit](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netlimit) * [NetPeerInfo](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netpeerinfo) * [NetPeers](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netpeers) * [NetPing](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netping) * [NetProtectAdd](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netprotectadd) * [NetProtectList](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netprotectlist) * [NetProtectRemove](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netprotectremove) * [NetPubsubScores](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netpubsubscores) * [NetSetLimit](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netsetlimit) * [NetStat](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#netstat) * [Backup](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#backup) * [OnlineBackup](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#onlinebackup) * [PieceDirectory](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#piecedirectory) * [PdBuildIndexForPieceCid](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#pdbuildindexforpiececid) * [PdCleanup](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#pdcleanup) * [PdRemoveDealForPiece](https://boost.filecoin.io/deployment/boost-architecture/json-rpc-api#pdremovedealforpiece) sun-brightdesktopmoon Copy import requests import json def main(): url = "http://localhost:3051/rpc/v0" headers = {'content-type': 'application/json', "Authorization": "Bearer "} payload = { "method": "Filecoin.BoostOfflineDealWithData", "params": [\ "",\ "",\ True\ ], "jsonrpc": "2.0", "id": 1, } response = requests.post(url, data=json.dumps(payload), headers=headers) print(response.text) if __name__ == "__main__": main() Copy { "info": { "title": "Boost RPC API", "version": "1.2.1/generated=2020-11-22T08:22:42-06:00" }, "methods": [], "openrpc": "1.2.6" } Copy [\ [\ "write"\ ]\ ] Copy [\ "string value"\ ] Copy [\ "write"\ ] Copy [\ null\ ] Copy [\ null\ ] Copy [\ null\ ] Copy [\ "07070707-0707-0707-0707-070707070707"\ ] Copy { "DealUuid": "07070707-0707-0707-0707-070707070707", "CreatedAt": "0001-01-01T00:00:00Z", "ClientDealProposal": { "Proposal": { "PieceCID": null, "PieceSize": 1032, "VerifiedDeal": true, "Client": "f01234", "Provider": "f01234", "Label": "", "StartEpoch": 10101, "EndEpoch": 10101, "StoragePricePerEpoch": "0", "ProviderCollateral": "0", "ClientCollateral": "0" }, "ClientSignature": { "Type": 2, "Data": "Ynl0ZSBhcnJheQ==" } }, "IsOffline": true, "CleanupData": true, "ClientPeerID": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf", "DealDataRoot": null, "InboundFilePath": "string value", "Transfer": { "Type": "string value", "ClientID": "string value", "Params": "Ynl0ZSBhcnJheQ==", "Size": 42 }, "ChainDealID": 5432, "PublishCID": null, "SectorID": 9, "Offset": 1032, "Length": 1032, "Checkpoint": 1, "CheckpointAt": "0001-01-01T00:00:00Z", "Err": "string value", "Retry": "auto", "NBytesReceived": 9, "FastRetrieval": true, "AnnounceToIPNI": true } Copy [\ null\ ] Copy { "DealUuid": "07070707-0707-0707-0707-070707070707", "CreatedAt": "0001-01-01T00:00:00Z", "ClientDealProposal": { "Proposal": { "PieceCID": null, "PieceSize": 1032, "VerifiedDeal": true, "Client": "f01234", "Provider": "f01234", "Label": "", "StartEpoch": 10101, "EndEpoch": 10101, "StoragePricePerEpoch": "0", "ProviderCollateral": "0", "ClientCollateral": "0" }, "ClientSignature": { "Type": 2, "Data": "Ynl0ZSBhcnJheQ==" } }, "IsOffline": true, "CleanupData": true, "ClientPeerID": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf", "DealDataRoot": null, "InboundFilePath": "string value", "Transfer": { "Type": "string value", "ClientID": "string value", "Params": "Ynl0ZSBhcnJheQ==", "Size": 42 }, "ChainDealID": 5432, "PublishCID": null, "SectorID": 9, "Offset": 1032, "Length": 1032, "Checkpoint": 1, "CheckpointAt": "0001-01-01T00:00:00Z", "Err": "string value", "Retry": "auto", "NBytesReceived": 9, "FastRetrieval": true, "AnnounceToIPNI": true } Copy [\ {\ "DealUUID": "07070707-0707-0707-0707-070707070707",\ "AllocationID": 0,\ "PieceCid": null,\ "ClientAddr": "f01234",\ "StartEpoch": 10101,\ "EndEpoch": 10101,\ "FilePath": "string value",\ "DeleteAfterImport": true,\ "RemoveUnsealedCopy": true,\ "SkipIPNIAnnounce": true\ }\ ] Copy { "Accepted": true, "Reason": "string value" } Copy [\ {\ "DealUUID": "07070707-0707-0707-0707-070707070707",\ "IsOffline": true,\ "ClientDealProposal": {\ "Proposal": {\ "PieceCID": null,\ "PieceSize": 1032,\ "VerifiedDeal": true,\ "Client": "f01234",\ "Provider": "f01234",\ "Label": "",\ "StartEpoch": 10101,\ "EndEpoch": 10101,\ "StoragePricePerEpoch": "0",\ "ProviderCollateral": "0",\ "ClientCollateral": "0"\ },\ "ClientSignature": {\ "Type": 2,\ "Data": "Ynl0ZSBhcnJheQ=="\ }\ },\ "DealDataRoot": null,\ "Transfer": {\ "Type": "string value",\ "ClientID": "string value",\ "Params": "Ynl0ZSBhcnJheQ==",\ "Size": 42\ },\ "RemoveUnsealedCopy": true,\ "SkipIPNIAnnounce": true\ }\ ] Copy { "Accepted": true, "Reason": "string value" } Copy [\ {\ "DealUuid": "07070707-0707-0707-0707-070707070707",\ "CreatedAt": "0001-01-01T00:00:00Z",\ "ClientDealProposal": {\ "Proposal": {\ "PieceCID": null,\ "PieceSize": 1032,\ "VerifiedDeal": true,\ "Client": "f01234",\ "Provider": "f01234",\ "Label": "",\ "StartEpoch": 10101,\ "EndEpoch": 10101,\ "StoragePricePerEpoch": "0",\ "ProviderCollateral": "0",\ "ClientCollateral": "0"\ },\ "ClientSignature": {\ "Type": 2,\ "Data": "Ynl0ZSBhcnJheQ=="\ }\ },\ "IsOffline": true,\ "CleanupData": true,\ "ClientPeerID": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf",\ "DealDataRoot": null,\ "InboundFilePath": "string value",\ "Transfer": {\ "Type": "string value",\ "ClientID": "string value",\ "Params": "Ynl0ZSBhcnJheQ==",\ "Size": 42\ },\ "ChainDealID": 5432,\ "PublishCID": null,\ "SectorID": 9,\ "Offset": 1032,\ "Length": 1032,\ "Checkpoint": 1,\ "CheckpointAt": "0001-01-01T00:00:00Z",\ "Err": "string value",\ "Retry": "auto",\ "NBytesReceived": 9,\ "FastRetrieval": true,\ "AnnounceToIPNI": true\ }\ ] Copy [\ null\ ] Copy [\ [\ "string value"\ ]\ ] Copy [\ null\ ] Copy [\ "Ynl0ZSBhcnJheQ=="\ ] Copy [\ "Bw=="\ ] Copy [\ null\ ] Copy { "Proposal": { "PieceCID": null, "PieceSize": 1032, "VerifiedDeal": true, "Client": "f01234", "Provider": "f01234", "Label": "", "StartEpoch": 10101, "EndEpoch": 10101, "StoragePricePerEpoch": "0", "ProviderCollateral": "0", "ClientCollateral": "0" }, "ClientSignature": { "Type": 2, "Data": "Ynl0ZSBhcnJheQ==" }, "ProposalCid": null, "AddFundsCid": null, "PublishCid": null, "Miner": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf", "Client": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf", "State": 42, "PiecePath": "", "MetadataPath": "", "SlashEpoch": 10101, "FastRetrieval": true, "Message": "string value", "FundsReserved": "0", "Ref": { "TransferType": "string value", "Root": null, "PieceCid": null, "PieceSize": 1024, "RawBlockSize": 42 }, "AvailableForRetrieval": true, "DealID": 5432, "CreationTime": "0001-01-01T00:00:00Z", "TransferChannelId": { "Initiator": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf", "Responder": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf", "ID": 3 }, "SectorNumber": 9, "InboundCAR": "string value" } Copy [\ "07070707-0707-0707-0707-070707070707",\ "string value",\ true\ ] Copy { "Accepted": true, "Reason": "string value" } Copy [\ "string value"\ ] Copy [\ "string value",\ "string value"\ ] Copy { "Ask": { "Price": "0", "VerifiedPrice": "0", "MinPieceSize": 1032, "MaxPieceSize": 1032, "Miner": "f01234", "Timestamp": 10101, "Expiry": 10101, "SeqNo": 42 }, "Signature": { "Type": 2, "Data": "Ynl0ZSBhcnJheQ==" } } Copy { "ID": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf", "Addrs": [\ "/ip4/52.36.61.156/tcp/1347/p2p/12D3KooWFETiESTf1v4PGUvtnxMAcEFMzLZbJGg4tjWfGEimYior"\ ] } Copy [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ] Copy { "Reachability": 1, "PublicAddrs": [\ "string value"\ ] } Copy { "TotalIn": 9, "TotalOut": 9, "RateIn": 12.3, "RateOut": 12.3 } Copy { "12D3KooWSXmXLJmBR1M7i9RW9GQPNUhZSzXKzxDHWtAgNuJAbyEJ": { "TotalIn": 174000, "TotalOut": 12500, "RateIn": 100, "RateOut": 50 } } Copy { "/fil/hello/1.0.0": { "TotalIn": 174000, "TotalOut": 12500, "RateIn": 100, "RateOut": 50 } } Copy [\ {\ "Peers": [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ],\ "IPAddrs": [\ "string value"\ ],\ "IPSubnets": [\ "string value"\ ]\ }\ ] Copy { "Peers": [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ], "IPAddrs": [\ "string value"\ ], "IPSubnets": [\ "string value"\ ] } Copy [\ {\ "Peers": [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ],\ "IPAddrs": [\ "string value"\ ],\ "IPSubnets": [\ "string value"\ ]\ }\ ] Copy [\ {\ "ID": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf",\ "Addrs": [\ "/ip4/52.36.61.156/tcp/1347/p2p/12D3KooWFETiESTf1v4PGUvtnxMAcEFMzLZbJGg4tjWfGEimYior"\ ]\ }\ ] Copy [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ] Copy [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ] Copy [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ] Copy { "ID": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf", "Addrs": [\ "/ip4/52.36.61.156/tcp/1347/p2p/12D3KooWFETiESTf1v4PGUvtnxMAcEFMzLZbJGg4tjWfGEimYior"\ ] } Copy [\ "string value"\ ] Copy { "Memory": 9, "Streams": 123, "StreamsInbound": 123, "StreamsOutbound": 123, "Conns": 123, "ConnsInbound": 123, "ConnsOutbound": 123, "FD": 123 } Copy [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ] Copy { "ID": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf", "Agent": "string value", "Addrs": [\ "string value"\ ], "Protocols": [\ "string value"\ ], "ConnMgrMeta": { "FirstSeen": "0001-01-01T00:00:00Z", "Value": 123, "Tags": { "name": 42 }, "Conns": { "name": "2021-03-08T22:52:18Z" } } } Copy [\ {\ "ID": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf",\ "Addrs": [\ "/ip4/52.36.61.156/tcp/1347/p2p/12D3KooWFETiESTf1v4PGUvtnxMAcEFMzLZbJGg4tjWfGEimYior"\ ]\ }\ ] Copy [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ] Copy [\ [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ]\ ] Copy [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ] Copy [\ [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ]\ ] Copy [\ {\ "ID": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf",\ "Score": {\ "Score": 12.3,\ "Topics": {\ "/blocks": {\ "TimeInMesh": 60000000000,\ "FirstMessageDeliveries": 122,\ "MeshMessageDeliveries": 1234,\ "InvalidMessageDeliveries": 3\ }\ },\ "AppSpecificScore": 12.3,\ "IPColocationFactor": 12.3,\ "BehaviourPenalty": 12.3\ }\ }\ ] Copy [\ "string value",\ {\ "Memory": 9,\ "Streams": 123,\ "StreamsInbound": 123,\ "StreamsOutbound": 123,\ "Conns": 123,\ "ConnsInbound": 123,\ "ConnsOutbound": 123,\ "FD": 123\ }\ ] Copy [\ "string value"\ ] Copy { "System": { "NumStreamsInbound": 123, "NumStreamsOutbound": 123, "NumConnsInbound": 123, "NumConnsOutbound": 123, "NumFD": 123, "Memory": 9 }, "Transient": { "NumStreamsInbound": 123, "NumStreamsOutbound": 123, "NumConnsInbound": 123, "NumConnsOutbound": 123, "NumFD": 123, "Memory": 9 }, "Services": { "abc": { "NumStreamsInbound": 1, "NumStreamsOutbound": 2, "NumConnsInbound": 3, "NumConnsOutbound": 4, "NumFD": 5, "Memory": 123 } }, "Protocols": { "abc": { "NumStreamsInbound": 1, "NumStreamsOutbound": 2, "NumConnsInbound": 3, "NumConnsOutbound": 4, "NumFD": 5, "Memory": 123 } }, "Peers": { "abc": { "NumStreamsInbound": 1, "NumStreamsOutbound": 2, "NumConnsInbound": 3, "NumConnsOutbound": 4, "NumFD": 5, "Memory": 123 } } } Copy [\ "string value"\ ] Copy [\ null\ ] Copy [\ null,\ "string value"\ ] sun-brightdesktopmoon --- # Migrate from Lotus to Boost | Boost The migrating from a monolith Lotus node or Lotus markets is a 2 step process. 1. [Migrate from Lotus to Boost v1arrow-up-right](https://boost.filecoin.io/v/v1/upgrade-from-lotus-to-boost) (Please use Boost v1.7.5 to migrate) 2. [Migrate from Boost v1 to Boost v2](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2) [PreviousUsing filters for storage and retrieval dealschevron-left](https://boost.filecoin.io/tutorials/using-filters-for-storage-and-retrieval-deals) [NextHow to onboard data using DDO dealschevron-right](https://boost.filecoin.io/tutorials/how-to-onboard-data-using-ddo-deals) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Using filters for storage and retrieval deals | Boost Storage providers might demand very precise and dynamic control over a combination of deal parameters. Boost, similarly to Lotus, provides two IPC hooks allowing you to name a command to execute for every deal before the storage provider accepts it: * `Filter` for storage deals. * `RetrievalFilter` for retrieval deals. The executed command receives a JSON representation of the deal parameters, as well as the current state of the sealing pipeline, on standard input, and upon completion, its exit code is interpreted as: * `0`: success, proceed with the deal. * `non-0`: failure, reject the deal. The most trivial filter rejecting any retrieval deal would be something like: `RetrievalFilter = "/bin/false"`. `/bin/false` is binary that immediately exits with a code of `1`. [This Perl scriptarrow-up-right](https://gist.github.com/ribasushi/53b7383aeb6e6f9b030210f4d64351d5/9bd6e898f94d20b50e7c7586dc8b8f3a45dab07c#file-dealfilter-pl) lets the miner deny specific clients and only accept deals that are set to start relatively soon. You can also use a third party content policy framework like `bitscreen` by Murmuration Labs, or [CID gravityarrow-up-right](https://cidgravity.com/) : Copy # grab filter program go get -u -v github.com/Murmuration-Labs/bitscreen # add it to both filters Filter = "/path/to/go/bin/bitscreen" RetrievalFilter = "/path/to/go/bin/bitscreen" Here is a sample JSON representation of the input sent to the deal filter: Copy { "DealParams": { "DealUUID": "48c31c8c-dcc8-4372-a0ac-b5468eea555b", "IsOffline": false, "ClientDealProposal": { "Proposal": { "PieceCID": { "/": "baga6ea4seaqh5prrl6ykov4t64k6m6giijsc44dcxtdnzsp4izjakqhs7twauiq" }, "PieceSize": 2147483648, "VerifiedDeal": false, "Client": "f1sw5zjcyo4mff5cbvgsgmm7uoko6gcr4tptvtkhy", "Provider": "f0127896", "Label": "bafyaa7qsgafcmalqudsaeidrunclaep6mdbipm2gjfvuosjfd6cbqd6th7bshy5hi5npxe727yjaagelucbyabasgafcmalqudsaeieapsxspo2i36no36n7yitswsxdazvziwvgj4vbp2scuxasrc6n4ejaage3r7m3saykcqeaegeavdllsbzaqcaibaaeecakrvvzam", "StartEpoch": 1717840, "EndEpoch": 2236240, "StoragePricePerEpoch": "1", "ProviderCollateral": "363196619502649", "ClientCollateral": "0" }, "ClientSignature": { "Type": 1, "Data": "SmgcBnQE+0ZIb4zAXw7TpxLliSaliShEvX9P4+uwvxBhRDlJD+F6N3NFoNrA2y5bTeWF5aWWuL93w+SSmXFkoAA=" } }, "DealDataRoot": { "/": "bafyaa7qsgafcmalqudsaeidrunclaep6mdbipm2gjfvuosjfd6cbqd6th7bshy5hi5npxe727yjaagelucbyabasgafcmalqudsaeieapsxspo2i36no36n7yitswsxdazvziwvgj4vbp2scuxasrc6n4ejaage3r7m3saykcqeaegeavdllsbzaqcaibaaeecakrvvzam" }, "Transfer": { "Type": "http", "ClientID": "", "Params": "eyJVUkwiOiJodHRwczovL2FudG8uLXB1YmxpYy1idWNrZXQtYm9vc3QuczMuZXUtY2VudHJhbC0xLmFtYXpvbmF3cy5jb20vcmFuZGZpbGVfMkdCXzAuY2FyIiwiSGVhZGVycyI6bnVsbH0=", "Size": 2000177948 } }, "SealingPipelineState": { "SectorStates": { "Available": 684, "Proving": 307, "Removed": 82, "TerminateFailed": 1, "TerminateWait": 5 }, "Workers": null } } [PreviousHow to store files with Boost on Filecoinchevron-left](https://boost.filecoin.io/tutorials/how-to-store-files-with-boost-on-filecoin) [NextMigrate from Lotus to Boostchevron-right](https://boost.filecoin.io/tutorials/upgrade-from-lotus-to-boost) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Architecture | v1.x (Deprecated) | Boost When designing the Local Index Directory we considered the needs of various Storage Providers (SPs) and the operational overhead LID would have on their systems. We built a solution for: - small- SPs - holding up to 1PiB), and - mid- and large- size SPs - holding anywhere from 1PiB, up to 100PiB data Depending on underlying block size and data format, index size can vary in size. Typically block sizes are between 16KiB and 1MiB. At the moment there are two implementations of LID: - a simple LevelDB implementation, for small SPs who want to keep all information in a single process database. - a scalable YugabyteDB implementation, for medium and large size SPs with tens of thousands of deals. [hashtag](https://boost.filecoin.io/v1/experimental-features/local-index-directory/architecture#index-types) Index types ----------------------------------------------------------------------------------------------------------------------------- In order to support the described retrieval use cases, LID maintains the following indexes: #### [hashtag](https://boost.filecoin.io/v1/experimental-features/local-index-directory/architecture#multihash-piece-cid) multihash → \[\]piece cid To look up which pieces contain a block #### [hashtag](https://boost.filecoin.io/v1/experimental-features/local-index-directory/architecture#piece-cid-sector-information-sector-id-offset-size) piece cid → sector information {sector ID, offset, size} To look up which sector a piece is in #### [hashtag](https://boost.filecoin.io/v1/experimental-features/local-index-directory/architecture#piece-cid-map-less-than-mulithash-block-offset-size-greater-than) piece cid → map To look up where in the piece a block is and the block’s size ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F968384846-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F01F3nbbmdopOdEHfj4fF%252Fuploads%252FDRYzHVC2t89Y35VcbiYP%252FScreenshot%25202023-05-18%2520at%252014.01.13.png%3Falt%3Dmedia%26token%3D668ab27e-33df-4dd0-b687-8894ce267da3&width=768&dpr=3&quality=100&sign=13caf171&sv=2) ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F968384846-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F01F3nbbmdopOdEHfj4fF%252Fuploads%252FqiR3OYrUgwRJqX8GTFtM%252FScreenshot%25202023-05-18%2520at%252014.01.23.png%3Falt%3Dmedia%26token%3Dc0459136-b3d5-48ca-9ba3-bab5f7161737&width=768&dpr=3&quality=100&sign=5f41d0e0&sv=2) [PreviousLocal Index Directorychevron-left](https://boost.filecoin.io/v1/experimental-features/local-index-directory) [NextRequirementschevron-right](https://boost.filecoin.io/v1/experimental-features/local-index-directory/requirements) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Experimental Features | v1.x (Deprecated) | Boost Boost is developing new market features on a regular basis as part of the overall market development. This section covers the experimental features released by boost along with details on how to use them. It is **not** recommended to run experimental features in production environments. The features should be tested as per your requirements, and any issues or requests should be reported to the team via Github or Slack. Once the new features have been tested and vetted, they will be released as part of a stable Boost release and all documentation concerning those features will be moved to an appropriate section of this site. Current experimental features are listed below. [FVM Contract Dealschevron-right](https://boost.filecoin.io/v1/experimental-features/fvm-contract-deals) [PreviousTroubleshootingchevron-left](https://boost.filecoin.io/v1/troubleshooting) [NextFVM Contract Dealschevron-right](https://boost.filecoin.io/v1/experimental-features/fvm-contract-deals) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Remote CommP | v1.x (Deprecated) | Boost Boost is introducing a new feature that allows computing `commP` during the deal on a lotus-worker node. This should reduce the overall resource utilisation on the Boost node. In order to enable remote commP on a Boost node, update your `config.toml`: Copy [Dealmaking] RemoteCommp = true Then restart the Boost node [PreviousDeal Filterschevron-left](https://boost.filecoin.io/v1/configuration/deal-filters) [NextLegacy Deal configurationchevron-right](https://boost.filecoin.io/v1/configuration/legacy-deal-configuration) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # FAQ | v1.x (Deprecated) | Boost **Is there a way to stop** `**boostd**` **daemon?** You can use the regular Unix OS signals **Is Boost compatible with the Lotus client? Can a client use** `**lotus client deal**` **to send a deal to Boost storage providers or do they have to use the boost client?** Yes, boost should work with any client given it supports the storage market protocol / default standard of Filecoin network today. **Does Boost provide retrieval functionality?** Yes, Boost provides 3 protocols for retrievals as of now. By default, Boost has Graphsync retrieval enabled. SPs can run Bitswap and HTTP retrievals by running [`booster-bitswap`](https://boost.filecoin.io/v1/bitswap-retrieval) and [`booster-http`](https://boost.filecoin.io/v1/http-retrieval) respectively. **Does Boost client have retreival functionality?** Yes, Boost client supports retrieval over graphsync protocol. But we highly recommend, using [`Lassie`arrow-up-right](https://docs.filecoin.io/basics/how-retrieval-works/basic-retrieval/#lassie) client for Filecoin/IPFS retrievals. **Can Boost make verified deals?** Yes, payments for deals can be made either from a regular wallet, or from DataCap. Deals that are paid for with DataCap are called `verified` deals. **Can I run both Boost and markets at the same time?** No, Boost replaces the legacy markets process. See [Migrate a Lotus markets service process to Boost](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-lotus-markets-service-process-to-boost) [PreviousJSON-RPC APIchevron-left](https://boost.filecoin.io/v1/json-rpc-api) [NextNeed help?chevron-right](https://boost.filecoin.io/v1/need-help) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # As a storage provider | v1.x (Deprecated) | Boost circle-exclamation If you are already running a standalone markets process, follow the guide at [Migrate a Lotus markets service process to Boost](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-lotus-markets-service-process-to-boost) If you are already running a monolith lotus-miner instance, follow the guide at [Migrate a monolith lotus-miner to Boost](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-monolith-lotus-miner-to-boost) ### [hashtag](https://boost.filecoin.io/v1/getting-started/getting-started#initialization-and-running) Initialization and Running 1\. Make sure you have a Lotus node and miner running 2\. Create and send funds to two new wallets on the lotus node to be used for Boost Boost currently uses two wallets for storage deals: * The publish storage deals wallet - This wallet pays the gas cost when Boost sends the `PublishStorageDeals` message. * The deal collateral wallet - When the Storage Provider accepts a deal, they must put collateral for the deal into escrow. Boost moves funds from this wallet into escrow with the `StorageMarketActor`. Copy export PUBLISH_STORAGE_DEALS_WALLET=`lotus wallet new bls` export COLLAT_WALLET=`lotus wallet new bls` lotus send --from mywallet $PUBLISH_STORAGE_DEALS_WALLET 10 lotus send --from mywallet $COLLAT_WALLET 10 3\. Set the publish storage deals wallet as a control wallet. Copy lotus-miner actor control set --really-do-it $PUBLISH_STORAGE_DEALS_WALLET 4\. Create and initialize the Boost repository circle-exclamation If you are already running a Lotus markets service process, you should run `boostd migrate` instead of `boostd init` See section [Migrate a Lotus markets service process to Boost](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-lotus-markets-service-process-to-boost) for more details. Boost keeps all data in a directory called the repository. By default the repository is at `~/.boost`. To use a different location pass the `--boost-repo` parameter (must precede any particular command verb, e.g. `boostd --boost-repo=/path init`). Export the environment variables needed for `boostd init` to connect to the lotus daemon and lotus miner. Export environment variables that point to the API endpoints for the sealing and mining processes. They will be used by the `boost` node to make JSON-RPC calls to the `mining/sealing/proving` node. Run `boostd init` to create and initialize the repository: * `--api-sealer` is the API info for the lotus-miner instance that does sealing * `--api-sector-index` is the API info for the lotus-miner instance that provides storage * `--max-staging-deals-bytes` is the maximum amount of storage to be used for downloaded files (once the limit is reached Boost will reject subsequent incoming deals) 5\. Update `ulimit` file descriptor limit if necessary. Boost deals will fail if the file descriptor limit for the process is not set high enough. This limit can be raised temporarily before starting the Boost process by running the command `ulimit -n 1048576`. We recommend setting it permanently by following the [Permanently Setting Your ULIMIT System Valuearrow-up-right](https://lotus.filecoin.io/kb/soft-fd-limit/) guide. 6\. Make sure that the correct and for your SP is set on chain, given that `boost init` generates a new identity. Use the following commands to update the values on chain: circle-info should be the same as the `ListenAddresses` you set in the `Libp2p` section of the config.toml of Boost can be found in the output of `boostd net id` command 7\. Run the `boostd` service, which will start: * libp2p listeners for storage and retrieval * the JSON RPC API * the graphql interface (used by the react front-end) * the web server for the react front-end circle-info In your firewall you will need to open the ports that libp2p listens on, so that Boost can receive storage and retrieval deals. See the `Libp2p` section of `config.toml` in the [Repository](https://boost.filecoin.io/v1/boost-architecture/architecture) ### [hashtag](https://boost.filecoin.io/v1/getting-started/getting-started#web-ui) Web UI circle-info When you build `boostd` using `make build` the react app is also part of the process. You can skip this section. Following steps are to be used only in case you are building binary and react app separately. 1. Build the React frontend 1. Open the Web UI Open http://localhost:8080 in your browser. circle-info To access a web UI running on a remote server, you can open an SSH tunnel from your local machine: ### [hashtag](https://boost.filecoin.io/v1/getting-started/getting-started#api-access) API Access Boost API can be accessed by setting the environment variable `BOOST_API_INFO` same as `LOTUS_MARKET_INFO`. You can also directly evaluate the `boostd auth` command with: [PreviousGetting startedchevron-left](https://boost.filecoin.io/v1/getting-started) [NextAs a clientchevron-right](https://boost.filecoin.io/v1/getting-started/boost-client) Last updated 2 years ago * [Initialization and Running](https://boost.filecoin.io/v1/getting-started/getting-started#initialization-and-running) * [Web UI](https://boost.filecoin.io/v1/getting-started/getting-started#web-ui) * [API Access](https://boost.filecoin.io/v1/getting-started/getting-started#api-access) sun-brightdesktopmoon Copy export $(lotus auth api-info --perm=admin) export $(lotus-miner auth api-info --perm=admin) Copy export APISEALER=`echo $MINER_API_INFO` export APISECTORINDEX=`echo $MINER_API_INFO` Copy boostd --vv init \ --api-sealer=$APISEALER \ --api-sector-index=$APISECTORINDEX \ --wallet-publish-storage-deals=$PUBLISH_STORAGE_DEALS_WALLET \ --wallet-deal-collateral=$COLLAT_WALLET \ --max-staging-deals-bytes=50000000000 Copy lotus-miner actor set-addrs lotus-miner actor set-peer-id Copy boostd --vv run Copy cd react # Download and install npm packages needed by the React frontend npm install --legacy-peer-deps # Build the optimized JavaScript and CSS in boost/react/build npm run build Copy ssh -L 8080:localhost:8080 myserver Copy boostd auth api-info --perm=admin export BOOST_API_INFO=: Copy export $(boostd auth api-info --perm=admin) sun-brightdesktopmoon --- # Serving files with booster-http | v1.x (Deprecated) | Boost With the release v1.7.0-rc1 of booster-http, Storage Providers can now serve blocks and files directly over the HTTP protocol. `booster-http` now implements a [IPFS HTTP gatewayarrow-up-right](https://docs.ipfs.tech/concepts/ipfs-gateway/#overview) with a [path resolution stylearrow-up-right](https://specs.ipfs.tech/http-gateways/path-gateway/) . This will allow the clients to download individual [IPFS blocksarrow-up-right](https://docs.ipfs.tech/how-to/work-with-blocks/) , car files and request uploaded files directly from their browser. SPs can take advantage of the ecosystem of tools to manage HTTP traffic, like load balancers and reverse proxies. Browsing deal data served via booster-http from web browser triangle-exclamation Before proceeding any further, we request you to read [basics of HTTP retrieval configuration](https://boost.filecoin.io/v1/http-retrieval) . This section is an extension of HTTP retrievals and deals with configuration specific to serving files and raw blocks. [hashtag](https://boost.filecoin.io/v1/http-retrieval/serving-files-with-booster-http#configuring-what-to-serve) Configuring what to serve ----------------------------------------------------------------------------------------------------------------------------------------------- The `booster-http` service can be started with specific type of content on IPFS gateway API This allows SPs to run multiple `booster-http` instances, each serving specific type of content like car files only or raw blocks only. [hashtag](https://boost.filecoin.io/v1/http-retrieval/serving-files-with-booster-http#enable-serving-files) Enable serving files ------------------------------------------------------------------------------------------------------------------------------------- In the curl request below we appended the query parameter format=raw to the URL to get the raw block data for the file. But, if we try to open the file directly in a web browser, with no extra query parameters, we get an error message: ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2Flh3.googleusercontent.com%2FZtp7SEG5XUL-1j2hhgg_QBeJDWqcz_hFjYLE174f1T3HxrI3_O6yXci8cSsxkLQicszSqQs6pZbDbPXVCtHu-rHiAMEX9ObAUMLygp-8rDF97Fc_fxBbBpOw1upFvoB43kq6AdKqrJzFbJ0_JbZ_Rtk&width=768&dpr=3&quality=100&sign=d28b5d4b&sv=2) Error when accessing IPLD block from web browser By default `booster-http` does not serve files in a format that can be read by a web browser. This is to protect Storage Providers from serving content that may be [flagged as illicit contentarrow-up-right](https://blog.nft.storage/posts/2022-04-29-gateways-and-gatekeepers) . To enable serving files to web browsers, we must pass `--serve-files=true` to `booster-http` on startup. Once, `booster-http` is restarted with `--serve-files=true,` we can open the file directly from a web browser: ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2Flh5.googleusercontent.com%2F4kDeP4XYol4NtcIDMYM_LSQaLBMI-_okjSS9cD4NbwfzN9BVBi7SG-R3TV5kbrc-oD4z0Q9M4lmlQDovrTzctgQ6hwL1-ax0g5rFsCfrnOj-Xj3aDZwPn8vIjUpEJngy7DFAoc83v29gp5o63WQREeI&width=768&dpr=3&quality=100&sign=3474f4ac&sv=2) Accessing files from Filecoin SP via web browser triangle-exclamation `booster-http` (and `booster-bitswap`) automatically filter out known flagged content using the denylist maintained at [https://badbits.dwebops.pub/denylist.jsonarrow-up-right](https://badbits.dwebops.pub/denylist.json) We can also browse all files in the CAR archive. ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2Flh4.googleusercontent.com%2FrsuZ86VU0fD_zWqvHYflkoa6hqEkFe3zJ1SHlB_X_iIsXRONFoldkH9cbOayAv2KFSY-6tw7MQO6ENC6lWIKVldBQYsPKWweQw845xXSvrqB-j-YWKFRHBlX9ak3CVjp5Is4MRHuNcva0LLnAg0UH6Q&width=768&dpr=3&quality=100&sign=4d70a0f1&sv=2) Browsing files in a deal with a web browser [hashtag](https://boost.filecoin.io/v1/http-retrieval/serving-files-with-booster-http#protecting-booster-http-with-nginx) Protecting booster-http with NGINX ----------------------------------------------------------------------------------------------------------------------------------------------------------------- SPs must secure their `booster-http` before exposing it to the public. SPs can feel free to use any tool available to limit who can download files, the number of requests per second, and the download bandwidth each client can use per second. Users can follow this example to use [NGNIXarrow-up-right](https://nginx.org/en/docs/) reverse proxy to secure their `booster-http` instance. In this section we’ve just scratched the surface of the ways in which nginx can set access limits, rate limits and bandwidth limits. In particular it’s possible to add limits by request token, or using JWT tokens. The examples in this section are adapted from [Deploying NGINX as an API Gatewayarrow-up-right](http://nginx.com/blog/deploying-nginx-plus-as-an-api-gateway-part-1) which goes into more detail. By default nginx puts configuration files into `/etc/nginx` The default configuration file is `/etc/nginx/sites-available/default` ### [hashtag](https://boost.filecoin.io/v1/http-retrieval/serving-files-with-booster-http#setup-server-block) Setup server block Setup nginx server listen on port `7575` and forward requests to booster-http on port `7777` The IPFS gateway serves files from `/ipfs`. So, we will add a server block for location /ipfs/ ### [hashtag](https://boost.filecoin.io/v1/http-retrieval/serving-files-with-booster-http#limiting-access) Limiting Access Let’s limit access to the IPFS gateway using the standard `.htaccess` file. We need to set up an `.htaccess` file with a username and password. Create a user named `_alice_` Include the `.htaccess` file in the `/etc/nginx/sites-available/default` Now when we open any URL under the path `/ipfs` we will be presented with a Sign in dialog. ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F968384846-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F01F3nbbmdopOdEHfj4fF%252Fuploads%252FhB7TAGEW64Bxbp8Zrh6f%252Fsecure_login.jpg%3Falt%3Dmedia%26token%3D09957b9e-b5f5-4a45-9573-99523c5b6353&width=768&dpr=3&quality=100&sign=cee35be0&sv=2) Login prompt when accessing booster-http url ### [hashtag](https://boost.filecoin.io/v1/http-retrieval/serving-files-with-booster-http#rate-limiting) Rate Limiting To prevent users from making too many requests per second, we should add rate limits. 1. Create a file with the rate limiting configuration at `/etc/nginx/ipfs-gateway.conf.d/ipfs-gateway.conf` 2. Add a request zone limit to the file of 1 request per second, per client IP 1. Include `ipfs-gateway.conf` in `/etc/nginx/sites-available/default` and set the response for too many requests to _HTTP response code 429_ 1. Click the refresh button in your browser on any path under /ipfs more than once per second you will see a 429 error page ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2Flh6.googleusercontent.com%2Fd-wTXF49wBDA8q-RiTRmZ7tAW4ZnH8nNRvpMxilJwr9hgyhtRemO5cceBfveUqsDUYEA1fINOyEOrHKVX0eL4WS8U9FjoKzYNNjef6OZl_FlrlQsg-slRftPCClEJLq9MxvF7EIwQKgfhJRVKx2Fz1M&width=768&dpr=3&quality=100&sign=aafb5c7&sv=2) HTTP error 429 ### [hashtag](https://boost.filecoin.io/v1/http-retrieval/serving-files-with-booster-http#bandwidth-limiting) Bandwidth Limiting It is also recommended to limit the amount of bandwidth that clients can take up when downloading data from `booster-http`. This ensures a fair bandwidth distribution to each client and prevents situations where one client ends up choking the `booster-http` instance. 1. Create a new .htaccess user called `_bob_` 1. Add a mapping from `.htaccess` username to bandwidth limit in `/etc/nginx/ipfs-gateway.conf.d/ipfs-gateway.conf` 1. Add the bandwidth limit to `/etc/nginx/sites-available/default` 1. To verify bandwidth limiting, use `curl` to download a file with user `_alice_` and then `_bob_` Note the difference in the Average Dload column (the average download speed). ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F968384846-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F01F3nbbmdopOdEHfj4fF%252Fuploads%252Fvr9JBxpo87EW7lBgWkR5%252Fdownload_speed.jpg%3Falt%3Dmedia%26token%3D5da681e7-c11a-4ebb-8b38-20041a325ef2&width=768&dpr=3&quality=100&sign=88fb102b&sv=2) Bandwidth limiting result [PreviousHTTP Retrievalchevron-left](https://boost.filecoin.io/v1/http-retrieval) [NextTroubleshootingchevron-right](https://boost.filecoin.io/v1/troubleshooting) Last updated 2 years ago * [Configuring what to serve](https://boost.filecoin.io/v1/http-retrieval/serving-files-with-booster-http#configuring-what-to-serve) * [Enable serving files](https://boost.filecoin.io/v1/http-retrieval/serving-files-with-booster-http#enable-serving-files) * [Protecting booster-http with NGINX](https://boost.filecoin.io/v1/http-retrieval/serving-files-with-booster-http#protecting-booster-http-with-nginx) * [Setup server block](https://boost.filecoin.io/v1/http-retrieval/serving-files-with-booster-http#setup-server-block) * [Limiting Access](https://boost.filecoin.io/v1/http-retrieval/serving-files-with-booster-http#limiting-access) * [Rate Limiting](https://boost.filecoin.io/v1/http-retrieval/serving-files-with-booster-http#rate-limiting) * [Bandwidth Limiting](https://boost.filecoin.io/v1/http-retrieval/serving-files-with-booster-http#bandwidth-limiting) sun-brightdesktopmoon Copy --serve-pieces enables serving raw pieces (default: true) --serve-blocks serve blocks with the ipfs gateway API (default: true) --serve-cars serve CAR files with the ipfs gateway API (default: true) --serve-files serve original files (eg jpg, mov) with the ipfs gateway API (default: false) --api-filter-endpoint value the endpoint to use for fetching a remote retrieval configuration for bitswap requests --api-filter-auth value value to pass in the authorization header when sending a request to the API filter endpoint (e.g. 'Basic ~base64 encoded user/pass~' --badbits-denylists value [ --badbits-denylists value ] the endpoints for fetching one or more custom BadBits list instead of the default one at https://badbits.dwebops.pub/denylist.json (default: "https://badbits.dwebops.pub/denylist.json") --help, -h show help Copy $ curl --output /tmp/museum.jpg "http://localhost:7777/ipfs/bafybeidqindpi4ucx7kmrtnw3woc6jtl7bqvyiokrkpbbuy6gs6trn57tm/vincent/Vincent%20van%20Gogh_files/Caf%C3%A9tafel_met_absint_-_s0186V1962_-_Van_Gogh_Museum.jpg?format=raw" % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 11830 100 11830 0 0 140k 0 --:--:-- --:--:-- --:--:-- 175k $ open /tmp/museum.jpg Copy # ipfs gateway config server { listen 7575 default_server; listen [::]:7575 default_server; location /ipfs/ { proxy_pass http://127.0.0.1:7777; } } Copy $ mkdir /etc/nginx/ipfs-gateway.conf.d $ htpasswd -c /etc/nginx/ipfs-gateway.conf.d/.htpasswd alice New password: Re-type new password: Adding password for user alice Copy # ipfs gateway config server { listen 7575 default_server; listen [::]:7575 default_server; location /ipfs/ { # htaccess authentication auth_basic "Restricted Server"; auth_basic_user_file /etc/nginx/ipfs-gateway.conf.d/.htpasswd; proxy_pass http://127.0.0.1:7777; } } Copy limit_req_zone $binary_remote_addr zone=client_ip_10rs:1m rate=1r/s; Copy include /etc/nginx/ipfs-gateway.conf.d/ipfs-gateway.conf; server { listen 7575 default_server; listen [::]:7575 default_server; location /ipfs/ { # htaccess authentication auth_basic "Restricted Server"; auth_basic_user_file /etc/nginx/ipfs-gateway.conf.d/.htpasswd; limit_req zone=client_ip_10rs; limit_req_status 429; proxy_pass http://127.0.0.1:7878; } } Copy $ htpasswd /etc/nginx/ipfs-gateway.conf.d/.htpasswd bob Copy map $remote_user $bandwidth_limit { default 1k; "alice" 10k; "bob" 512k; } Copy include /etc/nginx/ipfs-gateway.conf.d/ipfs-gateway.conf; server { listen 7575 default_server; listen [::]:7575 default_server; location /ipfs/ { # htaccess authentication auth_basic "Restricted Server"; auth_basic_user_file /etc/nginx/ipfs-gateway.conf.d/.htpasswd; limit_rate $bandwidth_limit; limit_req zone=client_ip_10rs; limit_req_status 429; proxy_pass http://127.0.0.1:7878; } } sun-brightdesktopmoon --- # HTTP Transfer limit | v1.x (Deprecated) | Boost Boost provides a capability to limit the number of simultaneous http transfer in progress to download the deal data from the clients. This new configuration has been introduced in the `ConfigVersion = 3` of the boost configuration file. ### [hashtag](https://boost.filecoin.io/v1/configuration/http-transfer-limit#configuration-variables) Configuration Variables #### [hashtag](https://boost.filecoin.io/v1/configuration/http-transfer-limit#http-variables) HTTP variables Copy # The maximum number of concurrent storage deal HTTP downloads. # Note that this is a soft maximum; if some downloads stall, # more downloads are allowed to start. # # type: uint64 # env var: LOTUS_DEALMAKING_HTTPTRANSFERMAXCONCURRENTDOWNLOADS #HttpTransferMaxConcurrentDownloads = 20 # The period between checking if downloads have stalled. # # type: Duration # env var: LOTUS_DEALMAKING_HTTPTRANSFERSTALLCHECKPERIOD #HttpTransferStallCheckPeriod = "30s" # The time that can elapse before a download is considered stalled (and # another concurrent download is allowed to start). # # type: Duration # env var: LOTUS_DEALMAKING_HTTPTRANSFERSTALLTIMEOUT #HttpTransferStallTimeout = "5m0s" #### [hashtag](https://boost.filecoin.io/v1/configuration/http-transfer-limit#storage-variables) Storage variables ### [hashtag](https://boost.filecoin.io/v1/configuration/http-transfer-limit#how-transferlimiter-works) How TransferLimiter works The `transferLimiter` maintains a queue of transfers with a soft upper limit on the number of concurrent transfers. To prevent slow or stalled transfers from blocking up the queue there are a couple of mitigations: The queue is ordered such that we * start transferring data for the oldest deal first * prefer to start transfers with peers that don't have any ongoing transfer * once the soft limit is reached, don't allow any new transfers with peers that have existing stalled transfers ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F968384846-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F01F3nbbmdopOdEHfj4fF%252Fuploads%252FuXeroXouYe0xkPvffo5Y%252Fimage.png%3Falt%3Dmedia%26token%3D5b244c3e-1cae-4163-b9aa-acbf60db6b33&width=768&dpr=3&quality=100&sign=49a4d748&sv=2) How TransferLimiter works Note that peers are distinguished by their host (eg foo.bar:8080) not by libp2p peer ID. For example, if there is * one active transfer with peer A * one pending transfer (peer A) * one pending transfer (peer B) The algorithm will prefer to start a transfer with peer B than peer A. This helps to ensure that slow peers don't block the transfer queue. The limit on the number of concurrent transfers is soft. Example: if there is a limit of 5 concurrent transfers and there are * three active transfers * two stalled transfers then two more transfers are permitted to start (as long as they're not with one of the stalled peers) [PreviousUI Settingschevron-left](https://boost.filecoin.io/v1/configuration/ui-settings) [NextDeal Filterschevron-right](https://boost.filecoin.io/v1/configuration/deal-filters) Last updated 2 years ago * [Configuration Variables](https://boost.filecoin.io/v1/configuration/http-transfer-limit#configuration-variables) * [How TransferLimiter works](https://boost.filecoin.io/v1/configuration/http-transfer-limit#how-transferlimiter-works) sun-brightdesktopmoon Copy # The maximum allowed disk usage size in bytes of downloaded deal data # that has not yet been passed to the sealing node by boost. # When the client makes a new deal proposal to download data from a host, # boost checks this config value against the sum of: # - the amount of data downloaded in the staging area # - the amount of data that is queued for download # - the amount of data in the proposed deal # If the total amount would exceed the limit, boost rejects the deal. # Set this value to 0 to indicate there is no limit. # # type: int64 # env var: LOTUS_DEALMAKING_MAXSTAGINGDEALSBYTES MaxStagingDealsBytes = 50000000000 # The percentage of MaxStagingDealsBytes that is allocated to each host. # When the client makes a new deal proposal to download data from a host, # boost checks this config value against the sum of: # - the amount of data downloaded from the host in the staging area # - the amount of data that is queued for download from the host # - the amount of data in the proposed deal # If the total amount would exceed the limit, boost rejects the deal. # Set this value to 0 to indicate there is no limit per host. # # type: uint64 # env var: LOTUS_DEALMAKING_MAXSTAGINGDEALSPERCENTPERHOST #MaxStagingDealsPercentPerHost = 0 sun-brightdesktopmoon --- # Requirements | v1.x (Deprecated) | Boost [hashtag](https://boost.filecoin.io/v1/experimental-features/local-index-directory/requirements#dependencies) Dependencies ------------------------------------------------------------------------------------------------------------------------------- Local Index Directory depends on a backend database to store various indices. Currently we support two implementations - YugabyteDB or LevelDB - depending on the size of deal data and indices a storage provider holds. **LevelDB** is an open source on-disk key-value store, and can be used when indices fit on a single host. **YugabyteDB** is an open source modern distributed database designed to run in any public, private, hybrid or multi-cloud environment. circle-info Storage providers who hold more than 1PiB data are encouraged to use YugabyteDB as it is horizontally scalable, provides better monitoring and management utilities and could support future growth. [hashtag](https://boost.filecoin.io/v1/experimental-features/local-index-directory/requirements#hardware-requirements) Hardware requirements ------------------------------------------------------------------------------------------------------------------------------------------------- circle-info For detailed instructions, playbooks and hardware recommendations, see the YugabyteDB website - [https://docs.yugabyte.comarrow-up-right](https://docs.yugabyte.com/) YugabyteDB is designed to run on bare-metal machines, virtual machines (VMs), and containers. CPU and RAM You should allocate adequate CPU and RAM. YugabyteDB has adequate defaults for running on a wide range of machines, and has been tested from 2 core to 64 core machines, and up to 200GB RAM. #### [hashtag](https://boost.filecoin.io/v1/experimental-features/local-index-directory/requirements#minimum-requirement) Minimum requirement Copy 2 cores 2GB RAM #### [hashtag](https://boost.filecoin.io/v1/experimental-features/local-index-directory/requirements#production-requirement) Production requirement Copy 16+ cores 32GB+ RAM Add more CPU (compared to adding more RAM) to improve performance. #### [hashtag](https://boost.filecoin.io/v1/experimental-features/local-index-directory/requirements#verify-support-for-sse2-and-sse4.2) Verify support for SSE2 and SSE4.2 YugabyteDB requires the SSE2 instruction set support, which was introduced into Intel chips with the Pentium 4 in 2001 and AMD processors in 2003. Most systems produced in the last several years are equipped with SSE2. In addition, YugabyteDB requires SSE4.2. To verify that your system supports SSE2, run the following command: `cat /proc/cpuinfo | grep sse2` To verify that your system supports SSE4.2, run the following command: `cat /proc/cpuinfo | grep sse4.2` #### [hashtag](https://boost.filecoin.io/v1/experimental-features/local-index-directory/requirements#disks) Disks We recommend a minimum of 1TiB or more allocated for YugabyteDB, depending on the amount of deal data you store and its average block size. circle-exclamation Assuming you've kept unsealed copies of all your data and have consistently indexed deal data, the size of your DAG store directory should be comparable with the requirements for YugabyteDB [PreviousArchitecturechevron-left](https://boost.filecoin.io/v1/experimental-features/local-index-directory/architecture) [NextInitialisationchevron-right](https://boost.filecoin.io/v1/experimental-features/local-index-directory/initialisation) Last updated 2 years ago * [Dependencies](https://boost.filecoin.io/v1/experimental-features/local-index-directory/requirements#dependencies) * [Hardware requirements](https://boost.filecoin.io/v1/experimental-features/local-index-directory/requirements#hardware-requirements) sun-brightdesktopmoon Copy SSDs (solid state disks) are required. sun-brightdesktopmoon --- # Local Index Directory | v1.x (Deprecated) | Boost circle-exclamation Local Index Directory is not yet released. This is a placeholder page for its documentation. [hashtag](https://boost.filecoin.io/v1/experimental-features/local-index-directory#background) Background -------------------------------------------------------------------------------------------------------------- The Local Index Directory (_LID_) manages and stores indices of deal data so that it can be retrieved by a content identifier (_cid_). Currently this task is performed by the _DAG store_ component. The DAG store keeps its indexes on disk on a single machine. LID replaces the DAG store and introduces a horizontally scalable backend database for storing the data - YugabyteDB. LID is designed to provide a more intuitive experience for the user, by surfacing problems and providing various repair tools. To summarize, LID is the component which keeps fine-grained metadata about all the deals on Filecoin that a given Storage Provider stores, and without it client would only be able to retrieve full pieces, which generally are between 8GiB and 32GiB in size. [hashtag](https://boost.filecoin.io/v1/experimental-features/local-index-directory#storing-data-on-filecoin) Storing data on Filecoin ------------------------------------------------------------------------------------------------------------------------------------------ When a client uploads deal data to Boost, LID records the sector that the deal data is stored in and scans the deal data to create an index of all its blocks indexed by block cid. This way cilents can later retrieve subsets of the original deal data, without retrieving the full deal data. ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F968384846-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F01F3nbbmdopOdEHfj4fF%252Fuploads%252FNOL2DPW8qvtSCQfFMLAz%252FScreenshot%25202023-05-18%2520at%252013.39.53.png%3Falt%3Dmedia%26token%3D512c15e6-b35d-405d-b83b-d29805cff772&width=768&dpr=3&quality=100&sign=4e7dd631&sv=2) How Boost stores deal data from clients [hashtag](https://boost.filecoin.io/v1/experimental-features/local-index-directory#retrieving-data) Retrieving data ------------------------------------------------------------------------------------------------------------------------ When a client makes a request for data by cid, LID: - checks which piece the cid is in, and where in the piece the data is - checks which sector the piece is in, and where in the sector the piece is - reads the data from the sector ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F968384846-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F01F3nbbmdopOdEHfj4fF%252Fuploads%252FeBsYo7WVrvtzEY5jrjhv%252FScreenshot%25202023-05-18%2520at%252013.45.14.png%3Falt%3Dmedia%26token%3Dc4dfeb4f-f81f-4d89-a148-2055b132a663&width=768&dpr=3&quality=100&sign=e84818e3&sv=2) How clients retrieve their data from Boost [hashtag](https://boost.filecoin.io/v1/experimental-features/local-index-directory#use-cases) Use cases ------------------------------------------------------------------------------------------------------------ The retrieval use cases that the Local Index Directory supports are: #### [hashtag](https://boost.filecoin.io/v1/experimental-features/local-index-directory#graphsync-retrieval) Graphsync retrieval _Request one root cid with a selector, receive many blocks_ LID is able to: - look up which piece contains the root cid - look up which sector contains the piece - for each block, get the offset into the piece for the block #### [hashtag](https://boost.filecoin.io/v1/experimental-features/local-index-directory#bitswap-retrieval) Bitswap retrieval _Request one block at a time_ LID is able to: - look up which piece contains the block - get the size of the block (Bitswap asks for the size before getting the block data) - look up which sector contains the piece - get the offset into the piece for the block #### [hashtag](https://boost.filecoin.io/v1/experimental-features/local-index-directory#http-retrieval) HTTP retrieval _Request a whole piece_ LID is able to look up which sector contains the piece. _Request an individual block_ LID is able to: - look up which piece contains the block - look up which sector contains the piece - get the offset into the piece for the block _Request a file by root cid_ LID is able to: - look up which piece contains the block - look up which sector contains the piece - for each block, get the offset into the piece for the block [PreviousFVM Contract Dealschevron-left](https://boost.filecoin.io/v1/experimental-features/fvm-contract-deals) [NextArchitecturechevron-right](https://boost.filecoin.io/v1/experimental-features/local-index-directory/architecture) Last updated 2 years ago * [Background](https://boost.filecoin.io/v1/experimental-features/local-index-directory#background) * [Storing data on Filecoin](https://boost.filecoin.io/v1/experimental-features/local-index-directory#storing-data-on-filecoin) * [Retrieving data](https://boost.filecoin.io/v1/experimental-features/local-index-directory#retrieving-data) * [Use cases](https://boost.filecoin.io/v1/experimental-features/local-index-directory#use-cases) sun-brightdesktopmoon sun-brightdesktopmoon --- # How to upgrade from Boost v1 to Boost v2 | Boost circle-exclamation Make sure you have read the [**Components**](https://boost.filecoin.io/deployment) page before proceeding. Boost v2 introduces a new service called **boostd-data** which requires a database to be installed - YugabyteDB or LevelDB. [hashtag](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2#introduction) Introduction ----------------------------------------------------------------------------------------------------------------- Boost v2 introduces the Local Index Directory as a replacement for the DAG store. It scales horizontally and provides a more intuitive experience for users, by surfacing problems in the UI and providing repair functionality. ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252FAwagrPbzXPS9sskaJwwi%252FScreenshot%25202023-06-14%2520at%252013.13.54.png%3Falt%3Dmedia%26token%3D225d016b-340a-48c7-a5de-4e7b70aed0e2&width=768&dpr=3&quality=100&sign=f7aedf2c&sv=2) ### [hashtag](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2#architecture) Architecture When boost receives a storage deal, it creates an index of all the block locations in the deal data, and stores the index in LID. When boostd / booster-http etc gets a request for a block it: * gets the block sector and offset from the LID index * requests the data at that sector and offset from the miner ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252Fgit-blob-44a62fd2c7451b2cd8c34053132e798d0851d4a9%252FLID-flow.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=b56b3be9&sv=2) LID Flow A large miner with many incoming retrieval requests needs many boostd / booster-http / booster-bitswap processes to serve those requests. These processes need to look up block locations in a centralized index. We tested several databases and found that YugabyteDB is best suited to the indexing workload because * it performs well on off-the-shelf hardware * it's easy to scale up by adding more machines * it has great documentation * once set up, it can be managed through a web UI ### [hashtag](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2#connecting-multiple-boost-instances-to-a-single-lid) Connecting multiple boost instances to a single LID It is possible to connect multiple boostd instances to a single LID instance. In this scenario, each boostd instance still stores data to a single miner. eg boostd A stores data to miner A, boostd B stores data to miner B. However each boostd instance saves retrieval indexes in a single, shared LID instance. ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252Fgit-blob-ae37dd9a92cbe1afbfd5219eee59465b0c30f8f6%252FLID-multi-store.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=b4d47035&sv=2) LID Flow For retrieval, each boostd instance can query the shared LID instance (to find out which miner has the data) and retrieve data from any miner in the cluster. ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252Fgit-blob-835e904abe1ac1e3c96c19d29ce530becb69a59a%252FLID-multi-fetch.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=b4896775&sv=2) LID Flow booster-bitswap and booster-http can also be configured to query the shared LID instance, and retrieve data from any miner in the cluster. ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252Fgit-blob-5c6ae86bc540cbc8b19d7dd9e9fb8087096d60e4%252FLID-multi.png%3Falt%3Dmedia&width=768&dpr=3&quality=100&sign=43780480&sv=2) LID Flow triangle-exclamation If you are deploying multiple `boostd` instances with a single LID instance you will need to set up the networking so that each `boostd`, `booster-bitswap` and `booster-http` instance can query all miners and workers in the cluster. We recommend assigning all of your miner instances and boostd instances to the same subnet. Note also that the Yugabyte DB instance will need enough space for retrieval indexes for all of the miners. [hashtag](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2#prerequisites) Prerequisites ------------------------------------------------------------------------------------------------------------------- ### [hashtag](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2#install-yugabytedb) Install YugabyteDB The Local Index Directory stores retrieval indices in a YugabyteDB database. Retrieval indices store the size and location of each block in the deal data. We recommend running YugabyteDB on a dedicated machine with SSD drives. Depending on how many blocks there are in the user data, the retrieval indices may require up to 2% of the size of the unsealed data. e.g. 1 TiB of unsealed user data may require a 20 GiB index. YugabyteDB should require about the same amount of space as your DAG store requires today. You can find more information about YugabyteDB in the `Components` section: [YugabyteDBchevron-right](https://boost.filecoin.io/deployment/yugabytedb) [hashtag](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2#instructions) Instructions ----------------------------------------------------------------------------------------------------------------- Follow these instructions in order to migrate your existing DAG store into the new Local Index Directory and upgrade from Boost v1 to Boost v2: **1\. Clone the Boost repository to a temporary directory** Note: Don’t overwrite your existing boost instance at this stage **2\. Check out the Boost v2 release** **3\. Build from source** **4\. Migrate dagstore indices** Depending on the amount of data your SP is storing, this step could take anywhere from a few minutes to a few hours. You can run it even while Boost v1 continues to run. The command can be stopped and restarted. It will continue from where it left off. Run the migration with parameters to connect to YugabyteDB on its Cassandra and PostgreSQL interfaces: circle-info The PGX driver from Yugabyte supports cluster aware Postgres connection out of the box. If you are deploying a multi-node YugabyteDB cluster, then please update your connect-string to use a cluster aware connection. With Cluster Mode: "postgresql://postgres:[\[email protected\]](https://boost.filecoin.io/cdn-cgi/l/email-protection) :5433?load\_balance=true" With Cluster Mode + No SSL: "postgresql://postgres:[\[email protected\]](https://boost.filecoin.io/cdn-cgi/l/email-protection) :5433?sslmode=disable&load\_balance=true" It will output a progress bar, and also a log file with detailed migration information at `migrate-yugabyte.log` circle-info If you are deploying a single LID instance with multiple boost instances, you will need to repeat this step for each boost instance in the cluster. **5\. Run the** `**boostd-data**` **service** `boostd-data` is a data proxy service which abstracts the access to LID through an established interface. It makes it easier to secure the underlying database and not expose it. `boostd-data` listens to a websocket interface, which is the entrypoint which should be exposed to `boostd`, and`booster-http` Start the `boostd-data` service with parameters to connect to YugabyteDB on its Cassandra and PostgreSQL interfaces: circle-info The PGX driver from Yugabyte supports cluster aware Postgres connection out of the box. If you are deploying a multi-node YugabyteDB cluster, then please update your connect-string to use a cluster aware connection. With Cluster Mode: "postgresql://postgres:[\[email protected\]](https://boost.filecoin.io/cdn-cgi/l/email-protection) :5433?load\_balance=true" With Cluster Mode + No SSL: "postgresql://postgres:[\[email protected\]](https://boost.filecoin.io/cdn-cgi/l/email-protection) :5433?sslmode=disable&load\_balance=true" circle-info \--hosts takes the IP addresses of the YugabyteDB YT-Servers separated by "," Example: \-- hosts 10.0.0.1,10.0.0.2,10.0.0.3 \--addr is the : where `boostd-data` service should be listening on. The IP here can be a private one (recommended) and should reachable by all boost related processes. Please ensure to update your firewall configuration accordingly. circle-exclamation If you are deploying a single LID instance with multiple `boostd` instances, you should run a single `boostd-data` process on one of the hosts where YugabyteDB is installed. All `boostd`, `booster-bitswap` and `booster-http` instances should be able to reach this single `boostd-data` process. **6\. Update** `**boostd**` **repository config** Configure `boostd` repository config (located at `/config.toml`) to point to the exposed `boostd-data` service endpoint. Note that the connection must be configured to go over a websocket. For example: **6.1 Add miners to** `**boostd**` **repository config** If you are deploying a single LID instance with multiple boost instances, you will also need to add to config the RPC endpoint for each miner in the cluster. This allows boostd to serve data for each miner over Graphsync. Make sure to test that this boostd instance can reach each miner by running **7\. Install Boost v2** Note that in v2 `booster-http` and `booster-bitswap` take slightly different parameters (see below). **8\. Stop** `**boostd**`**,** `**booster-http**` **and** `**booster-bitswap**` You need to stop `boostd` before migrating `piece info` data. **9\. Migrate piece info data (information about which sector each deal is stored in)** This should take no more than a few minutes. circle-info If you are deploying a single LID instance with multiple `boostd` instances, you will need to repeat this step for each `boostd` instance in the cluster. **10\. Start the upgraded versions of** `**boostd**`**,** `**booster-http**` **and** `**booster-bitswap**` Note that `booster-http` and `booster-bitswap` take slightly different parameters: * `--api-boost` is removed * There is a new parameter `--api-lid` that points to the `boostd-data` service (which hosts LID), e.g. `--api-lid="ws://:8044"` * If you are deploying a single LID instance with multiple `booster-bitswap` and `booster-http` instances, you should supply a `--api-storage` flag for each one * eg `--api-storage=MINER_API_INFO_1 --api-storage=MINER_API_INFO_2` * Make sure to test that this booster-http / booster-bitswap instance can reach each miner by running **11\. Clean up the dagstore directory from** `**boostd**` **repo and the temporary boost github repo** triangle-exclamation Be careful when running the below command to ensure that you do not remove incorrect directory $ rm -rf /dagstore $ rm -rf /tmp/boostv2 [hashtag](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2#verify-the-setup) Verify the setup ------------------------------------------------------------------------------------------------------------------------- **1\. Test how long it takes to reindex a piece** **2\. Perform a retrieval using Graphsync, Bitswap and HTTP** [hashtag](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2#conclusion) Conclusion ------------------------------------------------------------------------------------------------------------- At this stage you should have the latest version of Boost running with the Local Index Directory. Go to the **Local Index Directory** page and review the number sections: #### [hashtag](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2#pieces) Pieces ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252F3NgBMaEMAkyGYbLRLQpd%252FScreenshot%25202023-06-19%2520at%252013.43.34.png%3Falt%3Dmedia%26token%3D1985d12c-8b93-4164-9acf-e80b33148fa9&width=768&dpr=3&quality=100&sign=da7f1c23&sv=2) Pieces section shows counters for total pieces of user data that your SP is storing as well as whether you are keeping _unsealed_ and _indexed_ copies of them. #### [hashtag](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2#flagged-pieces) Flagged pieces ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252F3o0hRSr6XEXmnrkpZ5rm%252FScreenshot%25202023-06-19%2520at%252016.31.18.png%3Falt%3Dmedia%26token%3D6bd101e1-dd11-480a-8f22-08d185231e48&width=768&dpr=3&quality=100&sign=587e99c4&sv=2) _Flagged pieces_ are pieces that either lack an unsealed copy or are missing an index. For the sealed-only user data, you should make sure that you unseal individual sectors if you want this data to be retrievable. _Sealed only_ copies of data are not retrievable and are only being proven on-chain within the corresponding _deadline_ / _window_. Typically sealed only data is considered as archival as it is not immediately retrievable. If the client requests it, the SP sealing pipeline must first unseal it, which typically takes 1-2 hours, and only then the data becomes available. _Flagged (unsealed)_ pieces is user data that your SP is hosting, which is not indexed. circle-info We recommend that you trigger re-indexing for these pieces, so that data becomes retrievable. Check the [tutorial on re-indexing flagged unsealed pieces](https://boost.filecoin.io/tutorials/how-to-re-index-unsealed-pieces-that-are-flagged-by-lid-in-boost-v2) for more information. #### [hashtag](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2#deal-sector-copies) Deal Sector Copies ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252FwHBAINXXZm5VFWvtL3Ge%252FScreenshot%25202023-06-19%2520at%252016.34.41.png%3Falt%3Dmedia%26token%3Dc03d3e55-1461-477d-87f4-cf5ac3a23f60&width=768&dpr=3&quality=100&sign=28135436&sv=2) Deal Sector Copies section displays counters of your sectors state - whether you keep unsealed copies for all sectors or not. Ideally the SP should keep unsealed copies for all data that should be immediately retrievable. #### [hashtag](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2#sector-proving-state) Sector Proving State ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F437316145-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FfZMqMjrWeFXMBES7ehiI%252Fuploads%252FjiB1nOKeKaaLyBt2usHK%252FScreenshot%25202023-06-19%2520at%252016.34.46.png%3Falt%3Dmedia%26token%3D495b803e-8ff5-436e-b6f1-b670aa4b95b9&width=768&dpr=3&quality=100&sign=f52b4e6c&sv=2) Sector Proving State section displays counters of your active and inactive sectors - active sectors are those that are actively proven on-chain, inactive sectors are those that you might have failed to publish a _WindowPoSt_ for, or are expired or removed. [PreviousTutorialschevron-left](https://boost.filecoin.io/tutorials) [NextHow to re-index unsealed pieces that are flagged by LID in Boost v2chevron-right](https://boost.filecoin.io/tutorials/how-to-re-index-unsealed-pieces-that-are-flagged-by-lid-in-boost-v2) Last updated 2 years ago * [Introduction](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2#introduction) * [Architecture](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2#architecture) * [Connecting multiple boost instances to a single LID](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2#connecting-multiple-boost-instances-to-a-single-lid) * [Prerequisites](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2#prerequisites) * [Install YugabyteDB](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2#install-yugabytedb) * [Instructions](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2#instructions) * [Verify the setup](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2#verify-the-setup) * [Conclusion](https://boost.filecoin.io/tutorials/how-to-migrate-boost-v1-to-boost-v2#conclusion) sun-brightdesktopmoon Copy cd /tmp git clone https://github.com/filecoin-project/boost.git boostv2 cd boostv2 Copy git checkout v2.x.x Copy make Copy ./migrate-lid yugabyte \ --hosts \ --connect-string="postgresql://:@:5433" \ dagstore Copy ./boostd-data run yugabyte \ --hosts \ --connect-string="postgresql://:@:5433" \ --addr 0.0.0.0:8044 Copy [LocalIndexDirectory] ServiceApiInfo = "ws://:8044" Copy [DealMaking] GraphsyncStorageAccessApiInfo = [\ # Make sure to include the miner that this boostd instance\ # stores data to, as well as the other miners.\ # Use `lotus-miner auth api-info` to get the RPC API connect string.\ ":/ip4//tcp/2345/http",\ ":/ip4//tcp/2345/http"\ ] Copy $ MINER_API_INFO=:/ip4//tcp/2345/http lotus-miner info Copy make install Copy ./migrate-lid yugabyte \ --hosts \ --connect-string="postgresql://:@:5433" \ pieceinfo Copy $ MINER_API_INFO=MINER_API_INFO_1 lotus-miner info Copy time boostd lid gen-index Copy # boost retrieve --provider= -o output.dat # booster-bitswap fetch /ip4/127.0.0.1/tcp/8888/p2p/{peerID} {rootCID} outfile.car # curl -H "Accept:application/vnd.ipld.car;" http://{SP's http retrieval URL}/ipfs/bafySomePayloadCID -o bafySomePayloadCID.car sun-brightdesktopmoon --- # Initialisation | v1.x (Deprecated) | Boost Considering that the Local Index Directory is a new feature, Storage Providers should initialise it after upgrading their Boost deployments. There are two ways a Storage Provider can do that: 1. **Migrate existing indices from the DAG store into LID**: this solution assumes that the Storage Provider has been keeping an _unsealed_ copy for every sector they prove on-chain, and has already indexed all their deal data into the DAG store. Typically index sizes for a given sector range between 100KiB up to 1GiB, depending on deal data and its blocks sizes. The DAG store keeps these indices in the repository directory of Boost under the `./dagstore/index` and `./dagstore/datastore` directories. This data should be migrated to LID with the `migrate-lid` utility. 2. **Recreate indices for deal data based on unsealed copies of sectors**: this solution assumes that the Storage Provider has _unsealed copies_ for every sector they prove on-chain. If this is not the case, then the SP should first trigger an _unseal (UNS)_ job on their system for every sector that contains user data and produce an unseal copy. SPs can use the `boostd recover lid` utility to produce an index for all deal data within an unsealed sector and store it in LID so that they enable retrievals for the data. Depending on SPs deployment and where unsealed copies are hosted (NFS, Ceph, external disks, etc.) and the performance of the hosting system, producing an index for a 32GiB sector can take anywhere from a few seconds up to a few minutes, as the unsealed copy needs to be processed by the utility. [hashtag](https://boost.filecoin.io/v1/experimental-features/local-index-directory/initialisation#migrate-existing-indices-from-the-dag-store-into-lid) Migrate existing indices from the DAG store into LID ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- TODO [hashtag](https://boost.filecoin.io/v1/experimental-features/local-index-directory/initialisation#recreate-indices-for-deal-data-based-on-unsealed-copies-of-sectors) Recreate indices for deal data based on unsealed copies of sectors --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- TODO [PreviousRequirementschevron-left](https://boost.filecoin.io/v1/experimental-features/local-index-directory/requirements) [NextGraphQL APIchevron-right](https://boost.filecoin.io/v1/graphql-api) Last updated 2 years ago * [Migrate existing indices from the DAG store into LID](https://boost.filecoin.io/v1/experimental-features/local-index-directory/initialisation#migrate-existing-indices-from-the-dag-store-into-lid) * [Recreate indices for deal data based on unsealed copies of sectors](https://boost.filecoin.io/v1/experimental-features/local-index-directory/initialisation#recreate-indices-for-deal-data-based-on-unsealed-copies-of-sectors) sun-brightdesktopmoon sun-brightdesktopmoon --- # Deal Filters | v1.x (Deprecated) | Boost ### [hashtag](https://boost.filecoin.io/v1/configuration/deal-filters#using-filters-for-fine-grained-storage-and-retrieval-deal-acceptance) Using filters for fine-grained storage and retrieval deal acceptance Your use case might demand very precise and dynamic control over a combination of deal parameters. Lotus provides two IPC hooks allowing you to name a command to execute for every deal before the miner accepts it: * `Filter` for storage deals. * `RetrievalFilter` for retrieval deals. The executed command receives a JSON representation of the deal parameters on standard input, and upon completion, its exit code is interpreted as: * `0`: success, proceed with the deal. * `non-0`: failure, reject the deal. The most trivial filter rejecting any retrieval deal would be something like: `RetrievalFilter = "/bin/false"`. `/bin/false` is binary that immediately exits with a code of `1`. [This Perl scriptarrow-up-right](https://gist.github.com/ribasushi/53b7383aeb6e6f9b030210f4d64351d5/9bd6e898f94d20b50e7c7586dc8b8f3a45dab07c#file-dealfilter-pl) lets the miner deny specific clients and only accept deals that are set to start relatively soon. You can also use a third party content policy framework like [CIDgravityarrow-up-right](https://www.cidgravity.com/) or `bitscreen` by Murmuration Labs: Copy # grab filter program go get -u -v github.com/Murmuration-Labs/bitscreen # add it to both filters Filter = "/path/to/go/bin/bitscreen" RetrievalFilter = "/path/to/go/bin/bitscreen" [PreviousHTTP Transfer limitchevron-left](https://boost.filecoin.io/v1/configuration/http-transfer-limit) [NextRemote CommPchevron-right](https://boost.filecoin.io/v1/configuration/remote-commp) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Bitswap Retrieval | v1.x (Deprecated) | Boost `booster-bitswap` is a binary that runs alongside the `boostd` process, to serve retrievals over the Bitswap protocol. This feature of boost provides a number of tools for managing a production grade Bitswap retrieval service for a Storage Provider's content. circle-info There is currently no payment method in booster-bitswap. This endpoint is intended to serve free content. [hashtag](https://boost.filecoin.io/v1/bitswap-retrieval#why-enable-retrievals-via-bitswap) Why enable retrievals via bitswap? ----------------------------------------------------------------------------------------------------------------------------------- Bitswap retrieval introduces interoperability between IPFS and Filecoin, as it enables clients to retrieve Filecoin data over IPFS. This expands the reach of the Filecoin network considerably, increasing the value proposition for users to store data on the Filecoin network. This benefits the whole community, including SPs. Users will be able to access data directly via IPFS, as well as benefit from retrieval markets (e.g. Saturn) and compute over data projects (e.g. Bacalhau). [hashtag](https://boost.filecoin.io/v1/bitswap-retrieval#booster-bitswap-modes) `Booster-bitswap` modes ------------------------------------------------------------------------------------------------------------ There are two primary "modes" for exposing `booster-bitswap` to the internet. 1. In `private mode` the `booster-bitswap` peer ID is not publicly accessible to the internet. Instead, public Bitswap traffic goes to `boostd` itself, which then acts as a reverse proxy, forwarding that traffic on to `booster-bitswap`. This is similar to the way one might configure Nginx as a reverse proxy for an otherwise private web server. `private mode` is simpler to setup but may produce greater load on `boostd` as a protocol proxy. 2. In `public mode` the public internet firewall must be configured to forward traffic directly to the `booster-bitswap` instance. `boostd` is configured to announce the public address of `booster-bitswap` to the network indexer (the network indexer is the service that clients can query to discover where to retrieve content). This mode offers greater flexibility and performance. You can even setup `booster-bitswap` to run over a separate internet connection from `boostd`. However, it might require additional configuration and changes to your overall network infrastructure. ![](https://boost.filecoin.io/~gitbook/image?url=https%3A%2F%2F968384846-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252F01F3nbbmdopOdEHfj4fF%252Fuploads%252F3R8javrDBN6i4gt9ttt5%252Fbitswap.png%3Falt%3Dmedia%26token%3D22ca8f57-c37a-4d4e-a383-0ddf6070a22d&width=768&dpr=3&quality=100&sign=abd520f7&sv=2) Bitswap architecture and possible configurations ### [hashtag](https://boost.filecoin.io/v1/bitswap-retrieval#demo-configuration) Demo configuration You can configure booster-bitswap in the demo mode and familiarise yourself with the configuration. Once you are confident and familiar with the options, please go ahead and configure `booster-bitswap` for [production use](https://boost.filecoin.io/v1/bitswap-retrieval#setup-booster-bitswap-to-serve-retrievals-publicly) . 1\. Clone the the boost repo and checkout the latest stable release 2\. Build the `booster-bitswap` binary: 3\. Initialize `booster-bitswap`: 4\. Record the peer ID output by `booster-bitswap init` -- we will need this peer id later 5\. Collect the boost API Info 6\. Run booster-bitswap 7\. By default, booster-bitswap runs on port 8888. You can use `--port` to override this behaviour 8\. Fetching over bitswap by running Where `peerID` is the peer id recorded when you ran `booster-bitswap init` and `rootCID` is the CID of a data CID known to be stored on your SP. [hashtag](https://boost.filecoin.io/v1/bitswap-retrieval#setup-booster-bitswap-to-serve-retrievals) Setup `booster-bitswap` To Serve Retrievals ---------------------------------------------------------------------------------------------------------------------------------------------------- As described above, `booster-bitswap` can be configured to serve the retrievals in 2 modes. We recommend using `public mode` to avoid greater load on `boostd` as a protocol proxy. ### [hashtag](https://boost.filecoin.io/v1/bitswap-retrieval#private-mode) Private Mode 1\. Clone the `main` branch from the boost repo 2\. Build the `booster-bitswap` binary: 3\. Initialize `booster-bitswap`: 4\. Record the peer ID output by `booster-bitswap init` -- we will need this peer id later 5\. Stop `boostd` and edit ~/.boost/config.toml to set the peer ID for bitswap 6\. Start `boostd` service again 7\. Collect the boost API Info 8\. Run booster-bitswap circle-info You can get a `boostd` multiaddress by running `boostd net listen` and using any of the returned addresses 9\. By default, booster-bitswap runs on port 8888. You can use `--port` to override this behaviour 10\. Try to fetch a payload CID over bitswap to verify your configuration ### [hashtag](https://boost.filecoin.io/v1/bitswap-retrieval#public-mode) Public Mode 1\. Clone the `release/booster-bitswap` branch from the boost repo 2\. Build the `booster-bitswap` binary: 3\. Initialize `booster-bitswap`: 4\. Record the peer ID output by `booster-bitswap init` -- we will need this peer id later 5\. Stop `boostd` and edit ~/.boost/config.toml to set the peer ID for bitswap circle-info The libp2p private key file for booster-bitswap can generally be found at /libp2p.key The reason boost needs to know the public multiaddresses and libp2p private key for `booster-bitswap` is so it can properly announce these records to the network indexer. 6\. Start `boostd` service again 7\. Collect the boost API Info 8\. Run booster-bitswap 9\. By default, booster-bitswap runs on port 8888. You can use `--port` to override this behaviour 10\. Try to fetch a payload CID over bitswap to verify your configuration [hashtag](https://boost.filecoin.io/v1/bitswap-retrieval#booster-bitswap-configuration) `Booster-bitswap` configuration ---------------------------------------------------------------------------------------------------------------------------- `booster-bitswap` provides a number of performance and safety tools for managing a production grade bitswap server without overloading your infrastructure. ### [hashtag](https://boost.filecoin.io/v1/bitswap-retrieval#bitswap-server-performance) Bitswap Server Performance Depending on your hardware you may wish to increase or decrease the default parameters for the bitswap server internals. In the following example we are increasing the worker count for various components up to 600. This will utilize more CPU and I/O, but improve the performance of retrievals. See the command line help docs for details on each parameter. ### [hashtag](https://boost.filecoin.io/v1/bitswap-retrieval#badbits-filtering) BadBits filtering Booster-bitswap is automatically setup to deny all requests for CIDs that are on the BadBits Denylist. The default badbits list can be override or addition badbits list can be provided to the `booster-bitswap` instance. #### [hashtag](https://boost.filecoin.io/v1/bitswap-retrieval#to-override-the-default-badbits-list) To override the default badbits list #### [hashtag](https://boost.filecoin.io/v1/bitswap-retrieval#to-provide-additional-badbits-list) To provide additional badbits list ### [hashtag](https://boost.filecoin.io/v1/bitswap-retrieval#request-filtering) Request Filtering `booster-bitswap` provides a number of controls for filtering requests and limiting resource usage. These are expressed in a JSON configuration file `/retrievalconfig.json` circle-info You can create a new `retrievalconfig.json` file if one does not exists To make changes to the current configuration, you need to edit the `retrievalconfig.json` file and restart `booster-bitswap` for the changes to take affect. All configs are optional and absent parameters generally default to no filtering at all for the given parameter. You can also configure `booster-bitswap` to fetch your retrieval config from a remote HTTP API, possibly one provided by a third party configuration tool like [CIDGravityarrow-up-right](https://cidgravity.com/) . To do this, start `booster-bitswap` with the --api-filter-endpoint {url} option where URL is the HTTP URL for an API serving the above JSON format. Optionally, add --api-filter-auth {authheader}, if you need to pass a value for the HTTP Authorization header with your API When you setup with an API endpoint, `booster-bitswap` will update its local configuration from the API every five minutes, so you won't have to restart `booster-bitswap` to make a change. Please, be aware that the remote config will overwrite, rather than merge, with the local config. ### [hashtag](https://boost.filecoin.io/v1/bitswap-retrieval#bandwidth-limiting) Bandwidth Limiting Limiting bandwidth within booster-bitswap will not provide the optimal user experience. Dependent on individual setup, setting up limitations within the software could have a larger impact on the storage provider operations. Therefore, we recommend storage providers to set up their own bandwidth limitations using existing tools. There are multiple options to setup bandwidth limitating. 1. At the ISP level - dedicated bandwidth is provided to the node running booster-bitswap. 2. At the router level - we recommend configuring the bandwidth at the router level as it provides more flexibility and can be updated as needed. To configure the bandwidth on your router, please check with your manufacturer. 3. Limit the bandwidth using different tools available in Linux. Here are some of the examples of such tools. Please feel free to use any other tools not listed here and open a Github issue to add your example to this page. #### [hashtag](https://boost.filecoin.io/v1/bitswap-retrieval#tc) TC [TCarrow-up-right](https://man7.org/linux/man-pages/man8/tc.8.html) is used to configure Traffic Control in the Linux kernel. There are examples available online detailing how to configure rate limiting using TC. You can use the below commands to run a very basic configuration. #### [hashtag](https://boost.filecoin.io/v1/bitswap-retrieval#trickle) Trickle [Tricklearrow-up-right](https://wiki.archlinux.org/title/trickle) is a portable lightweight user space bandwidth shaper, that either runs in collaborative mode (together with trickled) or in standalone mode. You can read more about rate limiting with trickle [herearrow-up-right](https://www.tecmint.com/limit-linux-network-bandwidth-usage-with-trickle/) . Here's a starting point for configuration in trickle to rate limit the booster-bitswap service. #### [hashtag](https://boost.filecoin.io/v1/bitswap-retrieval#wondershaper) Wondershaper Another way of controlling network traffic is to limit bandwidth on individual network interface cards (NICs). [Wondershaperarrow-up-right](https://github.com/magnific0/wondershaper) is a small Bash script that uses the tc command-line utility in the background to let you regulate the amount of data flowing through a particular NIC. As you can imagine, while you can use wondershaper on a machine with a single NIC, its real advantage is on a machine with multiple NICs. Just like trickle, wondershaper is available in the official repositories of mainstream distributions. To limit network traffic with wondershaper, specify the NIC on which you wish to restrict traffic with the download and upload speed in kilobits per second. For example, [PreviousSetting up a monitoring stack for Boostchevron-left](https://boost.filecoin.io/v1/tutorials/setting-up-a-monitoring-stack-for-boost) [NextHTTP Retrievalchevron-right](https://boost.filecoin.io/v1/http-retrieval) Last updated 2 years ago * [Why enable retrievals via bitswap?](https://boost.filecoin.io/v1/bitswap-retrieval#why-enable-retrievals-via-bitswap) * [Booster-bitswap modes](https://boost.filecoin.io/v1/bitswap-retrieval#booster-bitswap-modes) * [Demo configuration](https://boost.filecoin.io/v1/bitswap-retrieval#demo-configuration) * [Setup booster-bitswap To Serve Retrievals](https://boost.filecoin.io/v1/bitswap-retrieval#setup-booster-bitswap-to-serve-retrievals) * [Private Mode](https://boost.filecoin.io/v1/bitswap-retrieval#private-mode) * [Public Mode](https://boost.filecoin.io/v1/bitswap-retrieval#public-mode) * [Booster-bitswap configuration](https://boost.filecoin.io/v1/bitswap-retrieval#booster-bitswap-configuration) * [Bitswap Server Performance](https://boost.filecoin.io/v1/bitswap-retrieval#bitswap-server-performance) * [BadBits filtering](https://boost.filecoin.io/v1/bitswap-retrieval#badbits-filtering) * [Request Filtering](https://boost.filecoin.io/v1/bitswap-retrieval#request-filtering) * [Bandwidth Limiting](https://boost.filecoin.io/v1/bitswap-retrieval#bandwidth-limiting) sun-brightdesktopmoon Copy git clone https://github.com/filecoin-project/boost.git cd boost git checkout Copy make booster-bitswap Copy booster-bitswap init Copy export ENV_BOOST_API_INFO=`boostd auth api-info --perm=admin` export BOOST_API_INFO=`echo $ENV_BOOST_API_INFO | awk '{split($0,a,"="); print a[2]}'` Copy booster-bitswap run --api-boost=$BOOST_API_INFO Copy booster-bitswap fetch /ip4/127.0.0.1/tcp/8888/p2p/{peerID} {rootCID} outfile.car Copy git clone https://github.com/filecoin-project/boost.git cd boost git checkout Copy make booster-bitswap Copy booster-bitswap init Copy [DealMaking] BitswapPeerID ="{peer id for booster bitswap you recorded earlier}" Copy export ENV_BOOST_API_INFO=`boostd auth api-info --perm=admin` export BOOST_API_INFO=`echo $ENV_BOOST_API_INFO | awk '{split($0,a,"="); print a[2]}'` Copy booster-bitswap run --api-boost=$BOOST_API_INFO --proxy={boostd multiaddress} Copy git clone https://github.com/filecoin-project/boost.git cd boost git checkout Copy make booster-bitswap Copy booster-bitswap init Copy [DealMaking] BitswapPeerID ="{peer id for bosoter bitswap you recorded earlier}" BitswapPublicAddresses = ["/ip4/{booster-bitswap public IP}/tcp/{booster-bitswap public port}"] BitswapPrivKeyFile = "{path to libp2p private key file for booster bitswap}" Copy export ENV_BOOST_API_INFO=`boostd auth api-info --perm=admin` export BOOST_API_INFO=`echo $ENV_BOOST_API_INFO | awk '{split($0,a,"="); print a[2]}'` Copy booster-bitswap run --api-boost=$BOOST_API_INFO Copy booster-bitswap run --api-boost=$BOOST_API_INFO \ --engine-blockstore-worker-count=600 \ --engine-task-worker-count=600 \ --max-outstanding-bytes-per-peer=33554432 \ --target-message-size=1048576 \ --task-worker-count=600 Copy booster-bitswap run --api-boost=$BOOST_API_INFO --badbits-denylists Copy booster-bitswap run --api-boost=$BOOST_API_INFO --badbits-denylists https://badbits.dwebops.pub/denylist.json Copy { "AllowDenyList": { // list of peers to either deny or allow (denying all others) "Type": "allowlist", // "allowlist" or "denylist" "PeerIDs": [\ "Qma9T5YraSnpRDZqRR4krcSJabThc8nwZuJV3LercPHufi",\ "QmYyQSo1c1Ym7orWxLYvCrM2EmxFTANf8wXmmE7DWjhx5N"\ ] }, "UnderMaintenance": false, // when set to true, denies all requests "StorageProviderLimits": { "Bitswap": { "SimultaneousRequests": 100, // bitswap block requests served at the same time across peers "SimultaneousRequestsPerPeer": 10, // bitswap block requests served at the same time for a single peer "MaxBandwidth": "100mb" // human readable size metric, per second } } } Copy booster-bitswap run --api-boost=$BOOST_API_INFO --api-filter-endpoint --api-filter-auth Copy sudo tc qdisc add dev root handle 1: htb sudo tc class add dev parent 1: classid 1:20 htb rate 100mibit sudo tc qdisc add dev parent 1:20 handle 20: sfq perturb 10 sudo tc filter add dev parent 1: protocol ip prio 1 basic match 'cmp(u16 at 0 layer transport eq 8888)' flowid 1:20 Copy [booster-bitswap] Priority = Time-Smoothing = Length-Smoothing = Copy wondershaper enp5s0 4096 1024 sun-brightdesktopmoon --- # HTTP Retrieval | v1.x (Deprecated) | Boost Boost introduced a new binary, `booster-http`, with release v1.2.0. This binary can be run alongside the `boostd` market process in order to serve retrievals over http. Currently, there is no payment method or built-in security integrated in the new binary. It can be run with any stable release of `boostd` and can also be run on a separate machine from the `boostd` process. Release v1.7.0-rc1 introduced support in `booster-http` for running an [IPFS gatewayarrow-up-right](https://docs.ipfs.tech/concepts/ipfs-gateway/#overview) , which enables Storage Providers to serve content to their users in multiple formats as described below and demonstrated using `curl`. ### [hashtag](https://boost.filecoin.io/v1/http-retrieval#retrieving-a-full-piece) Retrieving a full Piece When performing certain actions, such as replicating deals, it can be convenient to retrieve the entire Piece (with padding) to ensure commp integrity. Copy curl http://{SP's http retrieval URL}/piece/bagaSomePieceCID -o bagaSomePieceCID.piece ### [hashtag](https://boost.filecoin.io/v1/http-retrieval#retrieving-a-car-file) Retrieving a CAR file To return the CAR file for a given CID, you can pass an `Accept` header with the `application/vnd.ipld.car;` format. This can be useful for retrieving the raw, unpadded data of a deal. Copy curl -H "Accept:application/vnd.ipld.car;" http://{SP's http retrieval URL}/ipfs/bafySomePayloadCID -o bafySomePayloadCID.car ### [hashtag](https://boost.filecoin.io/v1/http-retrieval#retrieving-specific-files) Retrieving specific files For Storage Providers that have enabled serving raw files (disabled by default), users can retrieve specific files, such as images by their cid and path where applicable. See [Enable serving filesarrow-up-right](https://boost.filecoin.io/http-retrieval/serving-files-with-booster-http#enable-serving-files) for a more in depth example. Copy curl http://{SP's http retrieval URL}/ipfs/{content ID}/{optional path to resource} -o myimage.png ### [hashtag](https://boost.filecoin.io/v1/http-retrieval#retrieving-ipld-blocks) Retrieving IPLD blocks For advanced IPFS and IPLD use cases, you can now retrieve individual blocks by passing an `Accept` header with the `application/vnd.ipld.raw;` format Copy curl -H "Accept:application/vnd.ipld.raw;" http://{SP's http retrieval URL}/ipfs/bafySomeBlockCID -o bafySomeBlockCID [hashtag](https://boost.filecoin.io/v1/http-retrieval#local-setup) Local Setup ----------------------------------------------------------------------------------- SPs should try a local setup and test their HTTP retrievals before proceeding to run `booster-http` in production. To build and run `booster-http` : 1. Clone the boost repo and checkout the latest release 1. Build the new binary 1. Collect the token information for boost, lotus-miner and lotus daemon API 1. Start the `booster-http` server with the above details circle-info You can run multiple `booster-http` processes on the same machine by using a different port for each instance with the `--port` flag. You can also run multiple instances of the `booster-http` on different machines. [hashtag](https://boost.filecoin.io/v1/http-retrieval#running-public-boost-http-retrieval) Running Public Boost HTTP Retrieval ----------------------------------------------------------------------------------------------------------------------------------- The booster-http server listens on localhost. To expose the server publically, SPs should run a reverse proxy such as [NGINXarrow-up-right](https://nginx.org/en/docs/) to handle operational concerns like: * SSL * Authentication * Load balancing While booster-http may get more operational features over time, the intent is that providers who want to scale their HTTP operations will handle most of operational concerns via software in front of booster-http. You can setup a simple NGINX proxy using the [example provided](https://boost.filecoin.io/v1/http-retrieval/serving-files-with-booster-http#protecting-booster-http-with-nginx) in [Serving files with booster-http](https://boost.filecoin.io/v1/http-retrieval/serving-files-with-booster-http) ### [hashtag](https://boost.filecoin.io/v1/http-retrieval#making-http-retrieval-discoverable) Making HTTP Retrieval Discoverable To enable public discovery of the Boost HTTP server, SPs should set the domain root in boostd's `config.toml`. Under the `[DealMaking]` section, set `HTTPRetrievalMultiaddr` to the public domain root in multi-address format. Example `config.toml` section: Clients can determine if an SP offers HTTP retrieval by running: Clients can check the HTTP URL scheme version and supported queries Clients can download a piece using the domain root configured by the SP: [PreviousBitswap Retrievalchevron-left](https://boost.filecoin.io/v1/bitswap-retrieval) [NextServing files with booster-httpchevron-right](https://boost.filecoin.io/v1/http-retrieval/serving-files-with-booster-http) Last updated 2 years ago * [Retrieving a full Piece](https://boost.filecoin.io/v1/http-retrieval#retrieving-a-full-piece) * [Retrieving a CAR file](https://boost.filecoin.io/v1/http-retrieval#retrieving-a-car-file) * [Retrieving specific files](https://boost.filecoin.io/v1/http-retrieval#retrieving-specific-files) * [Retrieving IPLD blocks](https://boost.filecoin.io/v1/http-retrieval#retrieving-ipld-blocks) * [Local Setup](https://boost.filecoin.io/v1/http-retrieval#local-setup) * [Running Public Boost HTTP Retrieval](https://boost.filecoin.io/v1/http-retrieval#running-public-boost-http-retrieval) * [Making HTTP Retrieval Discoverable](https://boost.filecoin.io/v1/http-retrieval#making-http-retrieval-discoverable) sun-brightdesktopmoon Copy git clone https://github.com/filecoin-project/boost.git cd boost git checkout Copy make booster-http Copy export ENV_BOOST_API_INFO=`boostd auth api-info --perm=admin` export BOOST_API_INFO=`echo $ENV_BOOST_API_INFO | awk '{split($0,a,"="); print a[2]}'` Copy export ENV_FULLNODE_API_INFO=`lotus auth api-info --perm=admin` export FULLNODE_API_INFO=`echo $ENV_FULLNODE_API_INFO | awk '{split($0,a,"="); print a[2]}'` Copy export ENV_MINER_API_INFO=`lotus-miner auth api-info --perm=admin` export MINER_API_INFO=`echo $ENV_MINER_API_INFO | awk '{split($0,a,"="); print a[2]}'` Copy booster-http run --api-boost=$BOOST_API_INFO --api-fullnode=$FULLNODE_API_INFO --api-storage=$MINER_API_INFO Copy [DealMaking] HTTPRetrievalMultiaddr = "/dns/foo.com/tcp/443/https" Copy boost provider retrieval-transports Copy // Supported queries curl https://foo.com/index // Version curl https://foo.com/info Copy # Download a piece by its CID curl https://foo.com/piece/bagaSomePieceCID -o download.piece sun-brightdesktopmoon --- # Using filters for storage and retrieval deals | v1.x (Deprecated) | Boost Storage providers might demand very precise and dynamic control over a combination of deal parameters. Boost, similarly to Lotus, provides two IPC hooks allowing you to name a command to execute for every deal before the storage provider accepts it: * `Filter` for storage deals. * `RetrievalFilter` for retrieval deals. The executed command receives a JSON representation of the deal parameters, as well as the current state of the sealing pipeline, on standard input, and upon completion, its exit code is interpreted as: * `0`: success, proceed with the deal. * `non-0`: failure, reject the deal. The most trivial filter rejecting any retrieval deal would be something like: `RetrievalFilter = "/bin/false"`. `/bin/false` is binary that immediately exits with a code of `1`. [This Perl scriptarrow-up-right](https://gist.github.com/ribasushi/53b7383aeb6e6f9b030210f4d64351d5/9bd6e898f94d20b50e7c7586dc8b8f3a45dab07c#file-dealfilter-pl) lets the miner deny specific clients and only accept deals that are set to start relatively soon. You can also use a third party content policy framework like `bitscreen` by Murmuration Labs, or [CID gravityarrow-up-right](https://cidgravity.com/) : Copy # grab filter program go get -u -v github.com/Murmuration-Labs/bitscreen # add it to both filters Filter = "/path/to/go/bin/bitscreen" RetrievalFilter = "/path/to/go/bin/bitscreen" Here is a sample JSON representation of the input sent to the deal filter: Copy { "DealParams": { "DealUUID": "48c31c8c-dcc8-4372-a0ac-b5468eea555b", "IsOffline": false, "ClientDealProposal": { "Proposal": { "PieceCID": { "/": "baga6ea4seaqh5prrl6ykov4t64k6m6giijsc44dcxtdnzsp4izjakqhs7twauiq" }, "PieceSize": 2147483648, "VerifiedDeal": false, "Client": "f1sw5zjcyo4mff5cbvgsgmm7uoko6gcr4tptvtkhy", "Provider": "f0127896", "Label": "bafyaa7qsgafcmalqudsaeidrunclaep6mdbipm2gjfvuosjfd6cbqd6th7bshy5hi5npxe727yjaagelucbyabasgafcmalqudsaeieapsxspo2i36no36n7yitswsxdazvziwvgj4vbp2scuxasrc6n4ejaage3r7m3saykcqeaegeavdllsbzaqcaibaaeecakrvvzam", "StartEpoch": 1717840, "EndEpoch": 2236240, "StoragePricePerEpoch": "1", "ProviderCollateral": "363196619502649", "ClientCollateral": "0" }, "ClientSignature": { "Type": 1, "Data": "SmgcBnQE+0ZIb4zAXw7TpxLliSaliShEvX9P4+uwvxBhRDlJD+F6N3NFoNrA2y5bTeWF5aWWuL93w+SSmXFkoAA=" } }, "DealDataRoot": { "/": "bafyaa7qsgafcmalqudsaeidrunclaep6mdbipm2gjfvuosjfd6cbqd6th7bshy5hi5npxe727yjaagelucbyabasgafcmalqudsaeieapsxspo2i36no36n7yitswsxdazvziwvgj4vbp2scuxasrc6n4ejaage3r7m3saykcqeaegeavdllsbzaqcaibaaeecakrvvzam" }, "Transfer": { "Type": "http", "ClientID": "", "Params": "eyJVUkwiOiJodHRwczovL2FudG8uLXB1YmxpYy1idWNrZXQtYm9vc3QuczMuZXUtY2VudHJhbC0xLmFtYXpvbmF3cy5jb20vcmFuZGZpbGVfMkdCXzAuY2FyIiwiSGVhZGVycyI6bnVsbH0=", "Size": 2000177948 } }, "SealingPipelineState": { "SectorStates": { "Available": 684, "Proving": 307, "Removed": 82, "TerminateFailed": 1, "TerminateWait": 5 }, "Workers": null } } [PreviousHow to store files with Boost on Filecoinchevron-left](https://boost.filecoin.io/v1/tutorials/how-to-store-files-with-boost-on-filecoin) [NextSetting up a monitoring stack for Boostchevron-right](https://boost.filecoin.io/v1/tutorials/setting-up-a-monitoring-stack-for-boost) Last updated 2 years ago sun-brightdesktopmoon sun-brightdesktopmoon --- # Migrate a monolith lotus-miner to Boost | v1.x (Deprecated) | Boost circle-exclamation If you have already split a your lotus-miner into a separate markets process (MRA), follow the steps in [Migrate a Lotus markets service process to Boost](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-lotus-markets-service-process-to-boost) . circle-info Please note that a monolith miner can only split into boost(market)+miner on the same physical machine as it requires access to the miner repo to migrate the deal metadata. ### [hashtag](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-monolith-lotus-miner-to-boost#prepare-to-migrate) Prepare to migrate 1\. Make sure you have a Lotus node and miner running 2\. Create and send funds to two new wallets on the lotus node to be used for Boost Boost currently uses two wallets for storage deals: * The publish storage deals wallet - This wallet pays the gas cost when Boost sends the `PublishStorageDeals` message. circle-info If you already have a PublishStorageDeal control wallet setup then it can be reused in boost as the `PUBLISH_STORAGE_DEALS_WALLET`. * The deal collateral wallet - When the Storage Provider accepts a deal, they must put collateral for the deal into escrow. Boost moves funds from this wallet into escrow with the `StorageMarketActor`. circle-info If you already have a wallet that you want to use as the source of funds for deal collateral, then it can be reused in boost as the `COLLAT_WALLET`. Copy export PUBLISH_STORAGE_DEALS_WALLET=`lotus wallet new bls` export COLLAT_WALLET=`lotus wallet new bls` lotus send --from mywallet $PUBLISH_STORAGE_DEALS_WALLET 10 lotus send --from mywallet $COLLAT_WALLET 10 3\. Set the publish storage deals wallet as a control wallet. circle-exclamation Add the value of`PUBLISH_STORAGE_DEALS_WALLET` to the parameter `DealPublishControl` in the `Address` section of lotus-miner configuration if not present. Restart lotus-miner if configuration has been updated. Copy export OLD_CONTROL_ADDRESS=`lotus-miner actor control list --verbose | grep -v owner | grep -v worker | grep -v beneficiary | awk '{print $3}' | grep -v key | tr -s '\n' ' '` lotus-miner actor control set --really-do-it $PUBLISH_STORAGE_DEALS_WALLET $OLD_CONTROL_ADDRESS 4\. Set up environment variables needed for Boost migration Export the environment variables needed for `boostd migrate-monolith` to connect to the lotus daemon and lotus miner. Export environment variables that point to the API endpoints for the sealing and mining processes. They will be used by the `boost` node to make JSON-RPC calls to the `mining/sealing/proving` node. ### [hashtag](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-monolith-lotus-miner-to-boost#shut-down-lotus-miner) Shut down lotus-miner 1\. Stop accepting incoming deals 2\. Wait for incoming deals to complete 3\. Shutdown the lotus-miner 4\. Backup the lotus-miner repository 5\. Backup the lotus-miner datastore (in case you decide to roll back from Boost to Lotus) with: 6\. Set the environment variable `LOTUS_FULLNODE_API` to allow access to the lotus node API. ### [hashtag](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-monolith-lotus-miner-to-boost#migrate-from-the-lotus-miner-repo-to-the-boost-repo) Migrate from the lotus-miner repo to the Boost repo Run `boostd migrate-monolith` to create and initialize the boost repository: The `migrate-monolith` command * Initializes a Boost repository * Migrates markets datastore keys to Boost * Storage and retrieval deal metadata * Storage and retrieval ask data * Migrates markets libp2p keys to Boost * Migrates markets config to Boost (libp2p endpoints, settings etc) * Migrates the markets DAG store to Boost ### [hashtag](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-monolith-lotus-miner-to-boost#update-the-lotus-miner-config) Update the lotus-miner config 1\. Backup lotus-miner's `config.toml` 2\. Disable the markets subsystem in miner config: Boost replaces the markets subsystem in the lotus-miner, so we need to disable the subsystem in config: Under the `[Subsystems]` section set `EnableMarkets = false` 3\. Change the miner's libp2p port Boost replaces the markets subsystems, and listens on the same libp2p port, so we need to change the libp2p port that the miner is listening on. Under the `[Libp2p]` section change the port in `ListenAddresses` ### [hashtag](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-monolith-lotus-miner-to-boost#restart-lotus-miner) Restart lotus-miner Start lotus-miner up again so that Boost can connect to the miner when it starts. ### [hashtag](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-monolith-lotus-miner-to-boost#run-the-boostd-service) Run the `boostd` service The `boostd` service will start: * libp2p listeners for storage and retrieval * the JSON RPC API * the graphql interface (used by the react front-end) * the web server for the react front-end circle-info In your firewall you will need to ensure that the libp2p ports that Boost listens on are open, so that Boost can receive storage and retrieval deals. See the `Libp2p` section of `config.toml` in the [Repository](https://boost.filecoin.io/v1/boost-architecture/architecture) ### [hashtag](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-monolith-lotus-miner-to-boost#web-ui) Web UI Open http://localhost:8080 in your browser. circle-info To access a web UI running on a remote server, you can open an SSH tunnel from your local machine: ### [hashtag](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-monolith-lotus-miner-to-boost#api-access) API Access Boost API can be accessed by setting the environment variable `BOOST_API_INFO` same as `LOTUS_MARKET_INFO`. ### [hashtag](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-monolith-lotus-miner-to-boost#migrating-boost-from-one-machine-to-another) Migrating Boost from one machine to another Once the Boost has been split from the monolith miner, it can be moved to another physical or virtual machine by following the below steps. 1. Build the boost binary on the new machine by following the [Getting Started](https://boost.filecoin.io/v1/getting-started) step. 2. Copy the boost repo from the original monolith miner machine to the new dedicated boost machine. 3. Set the environment variable `LOTUS_FULLNODE_API` to allow access to the lotus node API. 4. Open the required port on the firewall on the monolith miner machine to allow connection to lotus-miner API. 5. In your firewall you will need to ensure that the libp2p ports that Boost listens on are open, so that Boost can receive storage and retrieval deals. See the `Libp2p` section of `config.toml` in the [Repository](https://boost.filecoin.io/v1/boost-architecture/architecture) 6. Start the `boostd` process. [PreviousMigrate from Lotus to Boostchevron-left](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost) [NextMigrate a Lotus markets service process to Boostchevron-right](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-lotus-markets-service-process-to-boost) Last updated 2 years ago * [Prepare to migrate](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-monolith-lotus-miner-to-boost#prepare-to-migrate) * [Shut down lotus-miner](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-monolith-lotus-miner-to-boost#shut-down-lotus-miner) * [Migrate from the lotus-miner repo to the Boost repo](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-monolith-lotus-miner-to-boost#migrate-from-the-lotus-miner-repo-to-the-boost-repo) * [Update the lotus-miner config](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-monolith-lotus-miner-to-boost#update-the-lotus-miner-config) * [Restart lotus-miner](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-monolith-lotus-miner-to-boost#restart-lotus-miner) * [Run the boostd service](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-monolith-lotus-miner-to-boost#run-the-boostd-service) * [Web UI](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-monolith-lotus-miner-to-boost#web-ui) * [API Access](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-monolith-lotus-miner-to-boost#api-access) * [Migrating Boost from one machine to another](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/migrate-a-monolith-lotus-miner-to-boost#migrating-boost-from-one-machine-to-another) sun-brightdesktopmoon Copy export $(lotus auth api-info --perm=admin) export $(lotus-miner auth api-info --perm=admin) Copy export APISEALER=`echo $MINER_API_INFO` export APISECTORINDEX=`echo $MINER_API_INFO` Copy lotus-shed market export-datastore --repo --backup-dir Copy lotus auth api-info -perm admin Copy boostd --vv migrate-monolith \ --import-miner-repo= \ --api-sealer=$APISEALER \ --api-sector-index=$APISECTORINDEX \ --wallet-publish-storage-deals=$PUBLISH_STORAGE_DEALS_WALLET \ --wallet-deal-collateral=$COLLAT_WALLET \ --max-staging-deals-bytes=50000000000 Copy cp /config.toml /config.toml.backup Copy boostd --vv run Copy ssh -L 8080:localhost:8080 myserver Copy export BOOST_API_INFO=: boostd auth api-info -perm auth Copy boostd --vv run sun-brightdesktopmoon --- # Legacy Deal configuration | v1.x (Deprecated) | Boost ### [hashtag](https://boost.filecoin.io/v1/configuration/legacy-deal-configuration#dealmaking-section) Dealmaking section This section controls parameters for making storage and retrieval deals: Copy [LotusDealmaking] # When enabled, the miner can accept online deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDERONLINESTORAGEDEALS #ConsiderOnlineStorageDeals = true # When enabled, the miner can accept offline deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDEROFFLINESTORAGEDEALS #ConsiderOfflineStorageDeals = true # When enabled, the miner can accept retrieval deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDERONLINERETRIEVALDEALS #ConsiderOnlineRetrievalDeals = true # When enabled, the miner can accept offline retrieval deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDEROFFLINERETRIEVALDEALS #ConsiderOfflineRetrievalDeals = true # When enabled, the miner can accept verified deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDERVERIFIEDSTORAGEDEALS #ConsiderVerifiedStorageDeals = true # When enabled, the miner can accept unverified deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDERUNVERIFIEDSTORAGEDEALS #ConsiderUnverifiedStorageDeals = true # A list of Data CIDs to reject when making deals # # type: []cid.Cid # env var: LOTUS_LOTUSDEALMAKING_PIECECIDBLOCKLIST #PieceCidBlocklist = [] # Maximum expected amount of time getting the deal into a sealed sector will take # This includes the time the deal will need to get transferred and published # before being assigned to a sector # # type: Duration # env var: LOTUS_LOTUSDEALMAKING_EXPECTEDSEALDURATION #ExpectedSealDuration = "24h0m0s" # Maximum amount of time proposed deal StartEpoch can be in future # # type: Duration # env var: LOTUS_LOTUSDEALMAKING_MAXDEALSTARTDELAY #MaxDealStartDelay = "336h0m0s" # When a deal is ready to publish, the amount of time to wait for more # deals to be ready to publish before publishing them all as a batch # # type: Duration # env var: LOTUS_LOTUSDEALMAKING_PUBLISHMSGPERIOD # PublishMsgPeriod = "40m0s" # The maximum number of deals to include in a single PublishStorageDeals # message # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_MAXDEALSPERPUBLISHMSG #MaxDealsPerPublishMsg = 8 # The maximum collateral that the provider will put up against a deal, # as a multiplier of the minimum collateral bound # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_MAXPROVIDERCOLLATERALMULTIPLIER #MaxProviderCollateralMultiplier = 2 # The maximum allowed disk usage size in bytes of staging deals not yet # passed to the sealing node by the markets service. 0 is unlimited. # # type: int64 # env var: LOTUS_LOTUSDEALMAKING_MAXSTAGINGDEALSBYTES # MaxStagingDealsBytes = 100000000000 # The maximum number of parallel online data transfers for storage deals # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_SIMULTANEOUSTRANSFERSFORSTORAGE #SimultaneousTransfersForStorage = 20 # The maximum number of simultaneous data transfers from any single client # for storage deals. # Unset by default (0), and values higher than SimultaneousTransfersForStorage # will have no effect; i.e. the total number of simultaneous data transfers # across all storage clients is bound by SimultaneousTransfersForStorage # regardless of this number. # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_SIMULTANEOUSTRANSFERSFORSTORAGEPERCLIENT #SimultaneousTransfersForStoragePerClient = 0 # The maximum number of parallel online data transfers for retrieval deals # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_SIMULTANEOUSTRANSFERSFORRETRIEVAL #SimultaneousTransfersForRetrieval = 20 # Minimum start epoch buffer to give time for sealing of sector with deal. # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_STARTEPOCHSEALINGBUFFER #StartEpochSealingBuffer = 480 # A command used for fine-grained evaluation of storage deals # see https://lotus.filecoin.io/storage-providers/advanced-configurations/market/#using-filters-for-fine-grained-storage-and-retrieval-deal-acceptance for more details # # type: string # env var: LOTUS_LOTUSDEALMAKING_FILTER #Filter = "" # A command used for fine-grained evaluation of retrieval deals # see https://lotus.filecoin.io/storage-providers/advanced-configurations/market/#using-filters-for-fine-grained-storage-and-retrieval-deal-acceptance for more details # # type: string # env var: LOTUS_LOTUSDEALMAKING_RETRIEVALFILTER #RetrievalFilter = "" [LotusDealmaking.RetrievalPricing] # env var: LOTUS_LOTUSDEALMAKING_RETRIEVALPRICING_STRATEGY #Strategy = "default" [LotusDealmaking.RetrievalPricing.Default] # env var: LOTUS_LOTUSDEALMAKING_RETRIEVALPRICING_DEFAULT_VERIFIEDDEALSFREETRANSFER #VerifiedDealsFreeTransfer = true [LotusDealmaking.RetrievalPricing.External] # env var: LOTUS_LOTUSDEALMAKING_RETRIEVALPRICING_EXTERNAL_PATH #Path = "" `ExpectedSealDuration` is an estimate of how long sealing will take and is used to reject deals whose start epoch might be earlier than the expected completion of sealing. It can be estimated by [benchmarkingarrow-up-right](https://lotus.filecoin.io/storage-providers/operate/benchmarks/) or by [pledging a sectorarrow-up-right](https://lotus.filecoin.io/storage-providers/operate/sector-pledging/) . circle-exclamation The final value of `ExpectedSealDuration` should equal `(TIME_TO_SEAL_A_SECTOR + WaitDealsDelay) * 1.5`. This equation ensures that the miner does not commit to having the sector sealed too soon `StartEpochSealingBuffer` allows `lotus-miner` to seal a sector before a certain epoch. For example: if the current epoch is 1000 and a deal within a sector must start on epoch 1500, then `lotus-miner` must wait until the current epoch is 1500 before it can start sealing that sector. However, if Boost sets `StartEpochSealingBuffer` to 500, the `lotus-miner` can start sealing the sector at epoch 1000. If there are multiple deals in a sector, the deal with a start time closest to the current epoch is what `StartEpochSealingBuffer` will be based off. So, if the sector in our example has three deals that start on epoch 1000, 1200, and 1400, then `lotus-miner` will start sealing the sector at epoch 500. #### [hashtag](https://boost.filecoin.io/v1/configuration/legacy-deal-configuration#publishing-several-deals-in-one-message) Publishing several deals in one message The `PublishStorageDeals` message can publish multiple deals in a single message. When a deal is ready to be published, Boost will wait up to `PublishMsgPeriod` for other deals to be ready before sending the `PublishStorageDeals` message. However, once `MaxDealsPerPublishMsg` is ready, Boost will immediately publish all the deals. For example, if `PublishMsgPeriod` is 1 hour: * At 1:00 pm, deal 1 is ready to publish. Boost will wait until 2:00 pm for other deals to be ready before sending `PublishStorageDeals`. * At 1:30 pm, Deal 2 is ready to publish * At 1:45 pm, Deal 3 is ready to publish * At 2:00pm, Boost publishes Deals 1, 2, and 3 in a single `PublishStorageDeals` message. If `MaxDealsPerPublishMsg` is 2, then in the above example, when deal 2 is ready to be published at 1:30, Boost would immediately publish Deals 1 & 2 in a single `PublishStorageDeals` message. Deal 3 would be published in a subsequent `PublishStorageDeals` message. triangle-exclamation If any of the deals in the `PublishStorageDeals` fails validation upon execution, or if the start epoch has passed, all deals will fail to be published ### [hashtag](https://boost.filecoin.io/v1/configuration/legacy-deal-configuration#using-filters-for-fine-grained-storage-and-retrieval-deal-acceptance) Using filters for fine-grained storage and retrieval deal acceptance Your use case might demand very precise and dynamic control over a combination of deal parameters. Boost provides two IPC hooks allowing you to name a command to execute for every deal before the miner accepts it: * `Filter` for storage deals. * `RetrievalFilter` for retrieval deals. The executed command receives a JSON representation of the deal parameters on standard input, and upon completion, its exit code is interpreted as: * `0`: success, proceed with the deal. * `non-0`: failure, reject the deal. The most trivial filter rejecting any retrieval deal would be something like: `RetrievalFilter = "/bin/false"`. `/bin/false` is binary that immediately exits with a code of `1`. [This Perl scriptarrow-up-right](https://gist.github.com/ribasushi/53b7383aeb6e6f9b030210f4d64351d5/9bd6e898f94d20b50e7c7586dc8b8f3a45dab07c#file-dealfilter-pl) lets the miner deny specific clients and only accept deals that are set to start relatively soon. You can also use a third party content policy framework like [CIDgravityarrow-up-right](https://www.cidgravity.com/) or `bitscreen` by Murmuration Labs: [PreviousRemote CommPchevron-left](https://boost.filecoin.io/v1/configuration/remote-commp) [NextHTTP indexer announcementchevron-right](https://boost.filecoin.io/v1/configuration/http-indexer-announcement) Last updated 2 years ago * [Dealmaking section](https://boost.filecoin.io/v1/configuration/legacy-deal-configuration#dealmaking-section) * [Using filters for fine-grained storage and retrieval deal acceptance](https://boost.filecoin.io/v1/configuration/legacy-deal-configuration#using-filters-for-fine-grained-storage-and-retrieval-deal-acceptance) sun-brightdesktopmoon Copy # grab filter program go get -u -v github.com/Murmuration-Labs/bitscreen # add it to both filters Filter = "/path/to/go/bin/bitscreen" RetrievalFilter = "/path/to/go/bin/bitscreen" sun-brightdesktopmoon --- # Configuration | v1.x (Deprecated) | Boost [hashtag](https://boost.filecoin.io/v1/configuration#sample-config-file) Sample config file ------------------------------------------------------------------------------------------------ Copy # The version of the config file (used for migrations) # # type: int # env var: LOTUS__CONFIGVERSION ConfigVersion = 4 # The connect string for the sealing RPC API (lotus miner) # # type: string # env var: LOTUS__SEALERAPIINFO SealerApiInfo = "" # The connect string for the sector index RPC API (lotus miner) # # type: string # env var: LOTUS__SECTORINDEXAPIINFO SectorIndexApiInfo = "" [API] # Binding address for the Lotus API # # type: string # env var: LOTUS_API_LISTENADDRESS #ListenAddress = "/ip4/127.0.0.1/tcp/2345/http" # type: string # env var: LOTUS_API_REMOTELISTENADDRESS #RemoteListenAddress = "" # type: Duration # env var: LOTUS_API_TIMEOUT #Timeout = "30s" [Backup] # Note that in case of metadata corruption it might be much harder to recover # your node if metadata log is disabled # # type: bool # env var: LOTUS_BACKUP_DISABLEMETADATALOG #DisableMetadataLog = false [Libp2p] # Binding address for the libp2p host - 0 means random port. # Format: multiaddress; see https://multiformats.io/multiaddr/ # # type: []string # env var: LOTUS_LIBP2P_LISTENADDRESSES # ListenAddresses = ["/ip4/0.0.0.0/tcp/24001", "/ip6/::/tcp/24001"] # Addresses to explicitally announce to other peers. If not specified, # all interface addresses are announced # Format: multiaddress # # type: []string # env var: LOTUS_LIBP2P_ANNOUNCEADDRESSES # AnnounceAddresses = [] # Addresses to not announce # Format: multiaddress # # type: []string # env var: LOTUS_LIBP2P_NOANNOUNCEADDRESSES # NoAnnounceAddresses = [] # When not disabled (default), lotus asks NAT devices (e.g., routers), to # open up an external port and forward it to the port lotus is running on. # When this works (i.e., when your router supports NAT port forwarding), # it makes the local lotus node accessible from the public internet # # type: bool # env var: LOTUS_LIBP2P_DISABLENATPORTMAP # DisableNatPortMap = false # ConnMgrLow is the number of connections that the basic connection manager # will trim down to. # # type: uint # env var: LOTUS_LIBP2P_CONNMGRLOW # ConnMgrLow = 350 # ConnMgrHigh is the number of connections that, when exceeded, will trigger # a connection GC operation. Note: protected/recently formed connections don't # count towards this limit. # # type: uint # env var: LOTUS_LIBP2P_CONNMGRHIGH # ConnMgrHigh = 400 # ConnMgrGrace is a time duration that new connections are immune from being # closed by the connection manager. # # type: Duration # env var: LOTUS_LIBP2P_CONNMGRGRACE # ConnMgrGrace = "20s" [Pubsub] # Run the node in bootstrap-node mode # # type: bool # env var: LOTUS_PUBSUB_BOOTSTRAPPER #Bootstrapper = false # type: string # env var: LOTUS_PUBSUB_REMOTETRACER #RemoteTracer = "" [Storage] # The maximum number of concurrent fetch operations to the storage subsystem # # type: int # env var: LOTUS_STORAGE_PARALLELFETCHLIMIT # ParallelFetchLimit = 10 [Dealmaking] # When enabled, the miner can accept online deals # # type: bool # env var: LOTUS_DEALMAKING_CONSIDERONLINESTORAGEDEALS #ConsiderOnlineStorageDeals = true # When enabled, the miner can accept offline deals # # type: bool # env var: LOTUS_DEALMAKING_CONSIDEROFFLINESTORAGEDEALS #ConsiderOfflineStorageDeals = true # When enabled, the miner can accept retrieval deals # # type: bool # env var: LOTUS_DEALMAKING_CONSIDERONLINERETRIEVALDEALS #ConsiderOnlineRetrievalDeals = true # When enabled, the miner can accept offline retrieval deals # # type: bool # env var: LOTUS_DEALMAKING_CONSIDEROFFLINERETRIEVALDEALS #ConsiderOfflineRetrievalDeals = true # When enabled, the miner can accept verified deals # # type: bool # env var: LOTUS_DEALMAKING_CONSIDERVERIFIEDSTORAGEDEALS #ConsiderVerifiedStorageDeals = true # When enabled, the miner can accept unverified deals # # type: bool # env var: LOTUS_DEALMAKING_CONSIDERUNVERIFIEDSTORAGEDEALS #ConsiderUnverifiedStorageDeals = true # A list of Data CIDs to reject when making deals # # type: []cid.Cid # env var: LOTUS_DEALMAKING_PIECECIDBLOCKLIST #PieceCidBlocklist = [] # Maximum expected amount of time getting the deal into a sealed sector will take # This includes the time the deal will need to get transferred and published # before being assigned to a sector # # type: Duration # env var: LOTUS_DEALMAKING_EXPECTEDSEALDURATION #ExpectedSealDuration = "24h0m0s" # Maximum amount of time proposed deal StartEpoch can be in future # # type: Duration # env var: LOTUS_DEALMAKING_MAXDEALSTARTDELAY #MaxDealStartDelay = "336h0m0s" # The maximum collateral that the provider will put up against a deal, # as a multiplier of the minimum collateral bound # # type: uint64 # env var: LOTUS_DEALMAKING_MAXPROVIDERCOLLATERALMULTIPLIER #MaxProviderCollateralMultiplier = 2 # The maximum allowed disk usage size in bytes of downloaded deal data # that has not yet been passed to the sealing node by boost. # When the client makes a new deal proposal to download data from a host, # boost checks this config value against the sum of: # - the amount of data downloaded in the staging area # - the amount of data that is queued for download # - the amount of data in the proposed deal # If the total amount would exceed the limit, boost rejects the deal. # Set this value to 0 to indicate there is no limit. # # type: int64 # env var: LOTUS_DEALMAKING_MAXSTAGINGDEALSBYTES # MaxStagingDealsBytes = 500000000 # The percentage of MaxStagingDealsBytes that is allocated to each host. # When the client makes a new deal proposal to download data from a host, # boost checks this config value against the sum of: # - the amount of data downloaded from the host in the staging area # - the amount of data that is queued for download from the host # - the amount of data in the proposed deal # If the total amount would exceed the limit, boost rejects the deal. # Set this value to 0 to indicate there is no limit per host. # # type: uint64 # env var: LOTUS_DEALMAKING_MAXSTAGINGDEALSPERCENTPERHOST # MaxStagingDealsPercentPerHost = 50 # Minimum start epoch buffer to give time for sealing of sector with deal. # # type: uint64 # env var: LOTUS_DEALMAKING_STARTEPOCHSEALINGBUFFER #StartEpochSealingBuffer = 480 # The amount of time to keep deal proposal logs for before cleaning them up. # # type: Duration # env var: LOTUS_DEALMAKING_DEALPROPOSALLOGDURATION #DealProposalLogDuration = "24h0m0s" # The amount of time to keep retrieval deal logs for before cleaning them up. # # type: Duration # env var: LOTUS_DEALMAKING_RETRIEVALLOGDURATION #RetrievalLogDuration = "24h0m0s" # A command used for fine-grained evaluation of storage deals # see https://boost.filecoin.io/configuration/deal-filters for more details # # type: string # env var: LOTUS_DEALMAKING_FILTER #Filter = "" # A command used for fine-grained evaluation of retrieval deals # see https://boost.filecoin.io/configuration/deal-filters for more details # # type: string # env var: LOTUS_DEALMAKING_RETRIEVALFILTER #RetrievalFilter = "" # The maximum amount of time a transfer can take before it fails # # type: Duration # env var: LOTUS_DEALMAKING_MAXTRANSFERDURATION #MaxTransferDuration = "24h0m0s" # Whether to do commp on the Boost node (local) or on the Sealer (remote) # # type: bool # env var: LOTUS_DEALMAKING_REMOTECOMMP #RemoteCommp = false # The maximum number of commp processes to run in parallel on the local # boost process # # type: uint64 # env var: LOTUS_DEALMAKING_MAXCONCURRENTLOCALCOMMP #MaxConcurrentLocalCommp = 1 # The public multi-address for retrieving deals with booster-http. # Note: Must be in multiaddr format, eg /dns/foo.com/tcp/443/https # # type: string # env var: LOTUS_DEALMAKING_HTTPRETRIEVALMULTIADDR #HTTPRetrievalMultiaddr = "" # The maximum number of concurrent storage deal HTTP downloads. # Note that this is a soft maximum; if some downloads stall, # more downloads are allowed to start. # # type: uint64 # env var: LOTUS_DEALMAKING_HTTPTRANSFERMAXCONCURRENTDOWNLOADS HttpTransferMaxConcurrentDownloads = 5 # The period between checking if downloads have stalled. # # type: Duration # env var: LOTUS_DEALMAKING_HTTPTRANSFERSTALLCHECKPERIOD #HttpTransferStallCheckPeriod = "30s" # The time that can elapse before a download is considered stalled (and # another concurrent download is allowed to start). # # type: Duration # env var: LOTUS_DEALMAKING_HTTPTRANSFERSTALLTIMEOUT #HttpTransferStallTimeout = "5m0s" # The peed id used by booster-bitswap. To set, copy the value # printed by running 'booster-bitswap init'. If this value is set, # Boost will: # - listen on bitswap protocols on its own peer id and forward them # to booster bitswap # - advertise bitswap records to the content indexer # - list bitswap in available transports on the retrieval transport protocol # # type: string # env var: LOTUS_DEALMAKING_BITSWAPPEERID # BitswapPeerID = "" # The deal logs older than DealLogDurationDays are deleted from the logsDB # to keep the size of logsDB in check. Set the value as "0" to disable log cleanup # # type: int # env var: LOTUS_DEALMAKING_DEALLOGDURATIONDAYS #DealLogDurationDays = 30 [Dealmaking.RetrievalPricing] # env var: LOTUS_DEALMAKING_RETRIEVALPRICING_STRATEGY #Strategy = "default" [Dealmaking.RetrievalPricing.Default] # env var: LOTUS_DEALMAKING_RETRIEVALPRICING_DEFAULT_VERIFIEDDEALSFREETRANSFER #VerifiedDealsFreeTransfer = true [Dealmaking.RetrievalPricing.External] # env var: LOTUS_DEALMAKING_RETRIEVALPRICING_EXTERNAL_PATH #Path = "" [Wallets] # The "owner" address of the miner # # type: string # env var: LOTUS_WALLETS_MINER Miner = "" # The wallet used to send PublishStorageDeals messages. # Must be a control or worker address of the miner. # # type: string # env var: LOTUS_WALLETS_PUBLISHSTORAGEDEALS PublishStorageDeals = "" # The wallet used as the source for storage deal collateral # # type: string # env var: LOTUS_WALLETS_DEALCOLLATERAL #DealCollateral = "" # Deprecated: Renamed to DealCollateral # # type: string # env var: LOTUS_WALLETS_PLEDGECOLLATERAL PledgeCollateral = "" [Graphql] # The port that the graphql server listens on # # type: uint64 # env var: LOTUS_GRAPHQL_PORT #Port = 8080 [LotusDealmaking] # When enabled, the miner can accept online deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDERONLINESTORAGEDEALS #ConsiderOnlineStorageDeals = true # When enabled, the miner can accept offline deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDEROFFLINESTORAGEDEALS #ConsiderOfflineStorageDeals = true # When enabled, the miner can accept retrieval deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDERONLINERETRIEVALDEALS #ConsiderOnlineRetrievalDeals = true # When enabled, the miner can accept offline retrieval deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDEROFFLINERETRIEVALDEALS #ConsiderOfflineRetrievalDeals = true # When enabled, the miner can accept verified deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDERVERIFIEDSTORAGEDEALS #ConsiderVerifiedStorageDeals = true # When enabled, the miner can accept unverified deals # # type: bool # env var: LOTUS_LOTUSDEALMAKING_CONSIDERUNVERIFIEDSTORAGEDEALS #ConsiderUnverifiedStorageDeals = true # A list of Data CIDs to reject when making deals # # type: []cid.Cid # env var: LOTUS_LOTUSDEALMAKING_PIECECIDBLOCKLIST #PieceCidBlocklist = [] # Maximum expected amount of time getting the deal into a sealed sector will take # This includes the time the deal will need to get transferred and published # before being assigned to a sector # # type: Duration # env var: LOTUS_LOTUSDEALMAKING_EXPECTEDSEALDURATION #ExpectedSealDuration = "24h0m0s" # Maximum amount of time proposed deal StartEpoch can be in future # # type: Duration # env var: LOTUS_LOTUSDEALMAKING_MAXDEALSTARTDELAY #MaxDealStartDelay = "336h0m0s" # When a deal is ready to publish, the amount of time to wait for more # deals to be ready to publish before publishing them all as a batch # # type: Duration # env var: LOTUS_LOTUSDEALMAKING_PUBLISHMSGPERIOD #PublishMsgPeriod = "40m0s" # The maximum number of deals to include in a single PublishStorageDeals # message # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_MAXDEALSPERPUBLISHMSG #MaxDealsPerPublishMsg = 8 # The maximum collateral that the provider will put up against a deal, # as a multiplier of the minimum collateral bound # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_MAXPROVIDERCOLLATERALMULTIPLIER #MaxProviderCollateralMultiplier = 2 # The maximum allowed disk usage size in bytes of staging deals not yet # passed to the sealing node by the markets service. 0 is unlimited. # # type: int64 # env var: LOTUS_LOTUSDEALMAKING_MAXSTAGINGDEALSBYTES MaxStagingDealsBytes = 100000000000 # The maximum number of parallel online data transfers for storage deals # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_SIMULTANEOUSTRANSFERSFORSTORAGE #SimultaneousTransfersForStorage = 20 # The maximum number of simultaneous data transfers from any single client # for storage deals. # Unset by default (0), and values higher than SimultaneousTransfersForStorage # will have no effect; i.e. the total number of simultaneous data transfers # across all storage clients is bound by SimultaneousTransfersForStorage # regardless of this number. # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_SIMULTANEOUSTRANSFERSFORSTORAGEPERCLIENT #SimultaneousTransfersForStoragePerClient = 0 # The maximum number of parallel online data transfers for retrieval deals # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_SIMULTANEOUSTRANSFERSFORRETRIEVAL #SimultaneousTransfersForRetrieval = 20 # Minimum start epoch buffer to give time for sealing of sector with deal. # # type: uint64 # env var: LOTUS_LOTUSDEALMAKING_STARTEPOCHSEALINGBUFFER #StartEpochSealingBuffer = 480 # A command used for fine-grained evaluation of storage deals # see https://lotus.filecoin.io/storage-providers/advanced-configurations/market/#using-filters-for-fine-grained-storage-and-retrieval-deal-acceptance for more details # # type: string # env var: LOTUS_LOTUSDEALMAKING_FILTER Filter = "" # A command used for fine-grained evaluation of retrieval deals # see https://lotus.filecoin.io/storage-providers/advanced-configurations/market/#using-filters-for-fine-grained-storage-and-retrieval-deal-acceptance for more details # # type: string # env var: LOTUS_LOTUSDEALMAKING_RETRIEVALFILTER #RetrievalFilter = "" [LotusDealmaking.RetrievalPricing] # env var: LOTUS_LOTUSDEALMAKING_RETRIEVALPRICING_STRATEGY #Strategy = "default" [LotusDealmaking.RetrievalPricing.Default] # env var: LOTUS_LOTUSDEALMAKING_RETRIEVALPRICING_DEFAULT_VERIFIEDDEALSFREETRANSFER #VerifiedDealsFreeTransfer = true [LotusDealmaking.RetrievalPricing.External] # env var: LOTUS_LOTUSDEALMAKING_RETRIEVALPRICING_EXTERNAL_PATH #Path = "" [LotusFees] # The maximum fee to pay when sending the PublishStorageDeals message # # type: types.FIL # env var: LOTUS_LOTUSFEES_MAXPUBLISHDEALSFEE MaxPublishDealsFee = "0.5 FIL" # The maximum fee to pay when sending the AddBalance message (used by legacy markets) # # type: types.FIL # env var: LOTUS_LOTUSFEES_MAXMARKETBALANCEADDFEE #MaxMarketBalanceAddFee = "0.007 FIL" [DAGStore] # Path to the dagstore root directory. This directory contains three # subdirectories, which can be symlinked to alternative locations if # need be: # - ./transients: caches unsealed deals that have been fetched from the # storage subsystem for serving retrievals. # - ./indices: stores shard indices. # - ./datastore: holds the KV store tracking the state of every shard # known to the DAG store. # Default value: /dagstore (split deployment) or # /dagstore (monolith deployment) # # type: string # env var: LOTUS_DAGSTORE_ROOTDIR #RootDir = "" # The maximum amount of indexing jobs that can run simultaneously. # 0 means unlimited. # Default value: 5. # # type: int # env var: LOTUS_DAGSTORE_MAXCONCURRENTINDEX #MaxConcurrentIndex = 5 # The maximum amount of unsealed deals that can be fetched simultaneously # from the storage subsystem. 0 means unlimited. # Default value: 0 (unlimited). # # type: int # env var: LOTUS_DAGSTORE_MAXCONCURRENTREADYFETCHES #MaxConcurrentReadyFetches = 0 # The maximum amount of unseals that can be processed simultaneously # from the storage subsystem. 0 means unlimited. # Default value: 0 (unlimited). # # type: int # env var: LOTUS_DAGSTORE_MAXCONCURRENTUNSEALS #MaxConcurrentUnseals = 0 # The maximum number of simultaneous inflight API calls to the storage # subsystem. # Default value: 100. # # type: int # env var: LOTUS_DAGSTORE_MAXCONCURRENCYSTORAGECALLS #MaxConcurrencyStorageCalls = 100 # The time between calls to periodic dagstore GC, in time.Duration string # representation, e.g. 1m, 5m, 1h. # Default value: 1 minute. # # type: Duration # env var: LOTUS_DAGSTORE_GCINTERVAL #GCInterval = "1m0s" [IndexProvider] # Enable set whether to enable indexing announcement to the network and expose endpoints that # allow indexer nodes to process announcements. Enabled by default. # # type: bool # env var: LOTUS_INDEXPROVIDER_ENABLE #Enable = true # EntriesCacheCapacity sets the maximum capacity to use for caching the indexing advertisement # entries. Defaults to 1024 if not specified. The cache is evicted using LRU policy. The # maximum storage used by the cache is a factor of EntriesCacheCapacity, EntriesChunkSize and # the length of multihashes being advertised. For example, advertising 128-bit long multihashes # with the default EntriesCacheCapacity, and EntriesChunkSize means the cache size can grow to # 256MiB when full. # # type: int # env var: LOTUS_INDEXPROVIDER_ENTRIESCACHECAPACITY #EntriesCacheCapacity = 1024 # EntriesChunkSize sets the maximum number of multihashes to include in a single entries chunk. # Defaults to 16384 if not specified. Note that chunks are chained together for indexing # advertisements that include more multihashes than the configured EntriesChunkSize. # # type: int # env var: LOTUS_INDEXPROVIDER_ENTRIESCHUNKSIZE #EntriesChunkSize = 16384 # TopicName sets the topic name on which the changes to the advertised content are announced. # If not explicitly specified, the topic name is automatically inferred from the network name # in following format: '/indexer/ingest/' # Defaults to empty, which implies the topic name is inferred from network name. # # type: string # env var: LOTUS_INDEXPROVIDER_TOPICNAME #TopicName = "" # PurgeCacheOnStart sets whether to clear any cached entries chunks when the provider engine # starts. By default, the cache is rehydrated from previously cached entries stored in # datastore if any is present. # # type: bool # env var: LOTUS_INDEXPROVIDER_PURGECACHEONSTART #PurgeCacheOnStart = false [ContractDeals] Enabled = true circle-info [hashtag](https://boost.filecoin.io/v1/configuration#sealer) Sealer ------------------------------------------------------------------------ Parameter Example Description SealerApiInfo "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBbGxvdyI6WyJyZwdyIiwid3JpdGUiLCJzaWduIiwiYWRtaW4iXX0.nbSvy11-tSUbXqo465hZqzTohGDfSdgh28C4irkmE10:/ip4/0.0.0.0/tcp/2345/http" Miner API info passed during boost init. Requires admin permissions. Connect string for the miner/sealer instance API endpoint SectorIndexApiInfo "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJBbGxvdyI6WyJyZwdyIiwid3JpdGUiLCJzaWduIiwiYWRtaW4iXX0.nbSvy11-tSUbXqo465hZqzTohGDfSdgh28C4irkmE10:/ip4/0.0.0.0/tcp/2345/http" Miner API info passed during boost init. Requires admin permissions. Connect string for the miner/sealer instance API endpoint [hashtag](https://boost.filecoin.io/v1/configuration#api) API ------------------------------------------------------------------ Parameter Example Description ListenAddress "/ip4/127.0.0.1/tcp/1288/http" \# Format: multiaddress Address Boost API will be listening on. No need to update unless you are planning to make API calls from outside the boost node RemoteListenAddress "0.0.0.0:1288" Address boost API can reached at from outside. No need to update unless you are planning to make API calls from outside the boost node Timeout "30s" RPC timeout value [hashtag](https://boost.filecoin.io/v1/configuration#libp2p) Libp2p ------------------------------------------------------------------------ Parameter Example Description ListenAddresses \# Format: multiaddress \["/ip4/209.94.92.3/tcp/24001"\] Binding address for the libp2p host - 0 means random port. AnnounceAddresses \# Format: multiaddress \["/ip4/209.94.92.3/tcp/24001"\] Addresses to explicitly announce to other peers. If not specified, all interface addresses are announced. On chain address need to be updated when this address is changed # lotus-miner actor set-addrs /ip4//tcp/24001 NoAnnounceAddresses \# Format: multiaddress \["/ip4/209.94.92.3/tcp/24001"\] Addresses to not announce. Can be used if you want to announce addresses with exceptions ConnMgrLow 150 ConnMgrLow is the number of connections that the basic connection manager will trim down to. Too low number can cause frequent connectivity issues ConnMgrHigh 200 ConnMgrHigh is the number of connections that, when exceeded, will trigger a connection GC operation Note: protected/recently formed connections don't count towards this limit. A high limit can cause very high resource utilization ConnMgrGrace "20s" ConnMgrGrace is a time duration that new connections are immune from being closed by the connection manager. [hashtag](https://boost.filecoin.io/v1/configuration#storage) Storage -------------------------------------------------------------------------- Parameter Example Description ParallelFetchLimit 10 Upper bound on how many sectors can be fetched in parallel by the storage system at a time [hashtag](https://boost.filecoin.io/v1/configuration#dealmaking) Dealmaking -------------------------------------------------------------------------------- `Dealmaking` section handles deal making configuration explicitly for boost deal that uses the new `/fil/storage/mk/1.2.0` protocol. [hashtag](https://boost.filecoin.io/v1/configuration#wallets) Wallets -------------------------------------------------------------------------- Parameter Example Description Miner f032187 Miner ID PublishStorageDeals f3syzhufifmnbzcznoquhy4mlxo3byetqlamzbeijk62bjpoohrj3wiphkgxe3yjrlh5dmxlca3zqxp3yvd33a #BLS wallet address This value is taken during init with `--wallet-publish-storage-deals.` This wallet is used to send PublishDeal messages. It can be hosted on the remote daemon node and does not require to be present locally. DealCollateral f3syzhufifmnbzcznoquhy4mlxo3byetqlamzbeijk62bjpoohrj3wiphkgxe3yjrlh5dmxlca3zqxp3yvd33a #BLS wallet address This value is taken during init with `--wallet-deal-collateral`. This wallet is used to provide collateral for the deal. Funds from this wallet are moved to market actor and locked during the deal duration. It can be hosted on the remote daemon node and does not require to be present locally. [hashtag](https://boost.filecoin.io/v1/configuration#lotusfees) LotusFees ------------------------------------------------------------------------------ Parameter Example Description MaxPublishDealsFee "0.05 FIL" Maximum fee user is willing to pay for a PublishDeal message MaxMarketBalanceAddFee "0.007 FIL" The maximum fee to pay when sending the AddBalance message (used by legacy markets) [hashtag](https://boost.filecoin.io/v1/configuration#dagstore) DAGStore ---------------------------------------------------------------------------- Parameter Example Description RootDir Empty If a custom value is specified, boost instance will refuse to start. This will be deprecated and removed in the future. MaxConcurrentIndex 5 The maximum amount of indexing jobs that can run simultaneously. 0 means unlimited. MaxConcurrentReadyFetches 0 The maximum amount of unsealed deals that can be fetched simultaneously from the storage subsystem. 0 means unlimited. MaxConcurrentUnseals 0 The maximum amount of unseals that can be processed simultaneously from the storage subsystem. 0 means unlimited. MaxConcurrencyStorageCalls 100 The maximum number of simultaneous inflight API calls to the storage subsystem. GCInterval "1m0s" The time between calls to periodic dagstore GC, in time.Duration string representation, e.g. 1m, 5m, 1h. [hashtag](https://boost.filecoin.io/v1/configuration#indexprovider) IndexProvider -------------------------------------------------------------------------------------- Parameter Example Description Enable True/False Enabled or disable the index-provider subsystem EntriesCacheCapacity 5 EntriesCacheCapacity sets the maximum capacity to use for caching the indexing advertisement entries. Defaults to 1024 if not specified. The cache is evicted using LRU policy. The maximum storage used by the cache is a factor of EntriesCacheCapacity, EntriesChunkSize and the length of multihashes being advertised. EntriesChunkSize 0 EntriesChunkSize sets the maximum number of multihashes to include in a single entries chunk. Defaults to 16384 if not specified. Note that chunks are chained together for indexing advertisements that include more multihashes than the configured EntriesChunkSize. TopicName "" TopicName sets the topic name on which the changes to the advertised content are announced. If not explicitly specified, the topic name is automatically inferred from the network name in following format: '/indexer/ingest/' PurgeCacheOnStart 100 PurgeCacheOnStart sets whether to clear any cached entries chunks when the provider engine starts. By default, the cache is rehydrated from previously cached entries stored in datastore if any is present. GCInterval "1m0s" The time between calls to periodic dagstore GC, in time.Duration string representation, e.g. 1m, 5m, 1h. Advertising 128-bit long multihashes with the default EntriesCacheCapacity, and EntriesChunkSize means the cache size can grow to 256MiB when full. [PreviousRoll back to Lotus markets service processchevron-left](https://boost.filecoin.io/v1/upgrade-from-lotus-to-boost/roll-back-to-legacy-markets) [NextUI Settingschevron-right](https://boost.filecoin.io/v1/configuration/ui-settings) Last updated 2 years ago * [Sample config file](https://boost.filecoin.io/v1/configuration#sample-config-file) * [Sealer](https://boost.filecoin.io/v1/configuration#sealer) * [API](https://boost.filecoin.io/v1/configuration#api) * [Libp2p](https://boost.filecoin.io/v1/configuration#libp2p) * [Storage](https://boost.filecoin.io/v1/configuration#storage) * [Dealmaking](https://boost.filecoin.io/v1/configuration#dealmaking) * [Wallets](https://boost.filecoin.io/v1/configuration#wallets) * [LotusFees](https://boost.filecoin.io/v1/configuration#lotusfees) * [DAGStore](https://boost.filecoin.io/v1/configuration#dagstore) * [IndexProvider](https://boost.filecoin.io/v1/configuration#indexprovider) sun-brightdesktopmoon Copy This guide covers all the configuration in use by boostd process. Some of the configuration parameters found in the config.toml file are not used in boost and thus are not covered here. These configuration parameters can be ignored sun-brightdesktopmoon --- # JSON-RPC API | v1.x (Deprecated) | Boost [hashtag](https://boost.filecoin.io/v1/json-rpc-api#go-json-rpc-client) Go JSON-RPC client ----------------------------------------------------------------------------------------------- To use the Boost Go client, the Go RPC-API library can be used to interact with the Boost API node. 1. Import the necessary Go module: Copy go get github.com/filecoin-project/go-jsonrpc 1. Create the following script: Copy package main import ( "context" "fmt" "log" "net/http" jsonrpc "github.com/filecoin-project/go-jsonrpc" boostapi "github.com/filecoin-project/boost/api" ) func main() { authToken := "" headers := http.Header{"Authorization": []string{"Bearer " + authToken}} addr := "127.0.0.1:1288" var api boostapi.BoostStruct closer, err := jsonrpc.NewMergeClient(context.Background(), "ws://"+addr+"/rpc/v0", "Filecoin", []interface{}{&api.Internal, &api.CommonStruct.Internal}, headers) if err != nil { log.Fatalf("connecting with boost failed: %s", err) } defer closer() // Now you can call any API you're interested in. netAddrs, err := api.NetAddrsListen(context.Background()) if err != nil { log.Fatalf("calling netAddrsListen: %s", err) } fmt.Printf("Boost is listening on: %s", netAddrs.Addrs[0]) } 1. Run `go mod init` to set up your `go.mod` file 2. You should now be able to interact with the Boost API. [hashtag](https://boost.filecoin.io/v1/json-rpc-api#python-json-rpc-client) Python JSON-RPC client ------------------------------------------------------------------------------------------------------- The JSON-RPC API can also be communicated with programmatically from other languages. Here is an example written in Python. Note that the `method` must be prefixed with `Filecoin.` [hashtag](https://boost.filecoin.io/v1/json-rpc-api#groups) Groups ----------------------------------------------------------------------- * [Actor](https://boost.filecoin.io/v1/json-rpc-api#actor) * [ActorSectorSize](https://boost.filecoin.io/v1/json-rpc-api#actorsectorsize) * [Auth](https://boost.filecoin.io/v1/json-rpc-api#auth) * [AuthNew](https://boost.filecoin.io/v1/json-rpc-api#authnew) * [AuthVerify](https://boost.filecoin.io/v1/json-rpc-api#authverify) * [Blockstore](https://boost.filecoin.io/v1/json-rpc-api#blockstore) * [BlockstoreGet](https://boost.filecoin.io/v1/json-rpc-api#blockstoreget) * [BlockstoreGetSize](https://boost.filecoin.io/v1/json-rpc-api#blockstoregetsize) * [BlockstoreHas](https://boost.filecoin.io/v1/json-rpc-api#blockstorehas) * [Boost](https://boost.filecoin.io/v1/json-rpc-api#boost) * [BoostDagstoreDestroyShard](https://boost.filecoin.io/v1/json-rpc-api#boostdagstoredestroyshard) * [BoostDagstoreGC](https://boost.filecoin.io/v1/json-rpc-api#boostdagstoregc) * [BoostDagstoreInitializeAll](https://boost.filecoin.io/v1/json-rpc-api#boostdagstoreinitializeall) * [BoostDagstoreInitializeShard](https://boost.filecoin.io/v1/json-rpc-api#boostdagstoreinitializeshard) * [BoostDagstoreListShards](https://boost.filecoin.io/v1/json-rpc-api#boostdagstorelistshards) * [BoostDagstorePiecesContainingMultihash](https://boost.filecoin.io/v1/json-rpc-api#boostdagstorepiecescontainingmultihash) * [BoostDagstoreRecoverShard](https://boost.filecoin.io/v1/json-rpc-api#boostdagstorerecovershard) * [BoostDagstoreRegisterShard](https://boost.filecoin.io/v1/json-rpc-api#boostdagstoreregistershard) * [BoostDeal](https://boost.filecoin.io/v1/json-rpc-api#boostdeal) * [BoostDealBySignedProposalCid](https://boost.filecoin.io/v1/json-rpc-api#boostdealbysignedproposalcid) * [BoostDummyDeal](https://boost.filecoin.io/v1/json-rpc-api#boostdummydeal) * [BoostIndexerAnnounceAllDeals](https://boost.filecoin.io/v1/json-rpc-api#boostindexerannouncealldeals) * [BoostMakeDeal](https://boost.filecoin.io/v1/json-rpc-api#boostmakedeal) * [BoostOfflineDealWithData](https://boost.filecoin.io/v1/json-rpc-api#boostofflinedealwithdata) * [Deals](https://boost.filecoin.io/v1/json-rpc-api#deals) * [DealsConsiderOfflineRetrievalDeals](https://boost.filecoin.io/v1/json-rpc-api#dealsconsiderofflineretrievaldeals) * [DealsConsiderOfflineStorageDeals](https://boost.filecoin.io/v1/json-rpc-api#dealsconsiderofflinestoragedeals) * [DealsConsiderOnlineRetrievalDeals](https://boost.filecoin.io/v1/json-rpc-api#dealsconsideronlineretrievaldeals) * [DealsConsiderOnlineStorageDeals](https://boost.filecoin.io/v1/json-rpc-api#dealsconsideronlinestoragedeals) * [DealsConsiderUnverifiedStorageDeals](https://boost.filecoin.io/v1/json-rpc-api#dealsconsiderunverifiedstoragedeals) * [DealsConsiderVerifiedStorageDeals](https://boost.filecoin.io/v1/json-rpc-api#dealsconsiderverifiedstoragedeals) * [DealsPieceCidBlocklist](https://boost.filecoin.io/v1/json-rpc-api#dealspiececidblocklist) * [DealsSetConsiderOfflineRetrievalDeals](https://boost.filecoin.io/v1/json-rpc-api#dealssetconsiderofflineretrievaldeals) * [DealsSetConsiderOfflineStorageDeals](https://boost.filecoin.io/v1/json-rpc-api#dealssetconsiderofflinestoragedeals) * [DealsSetConsiderOnlineRetrievalDeals](https://boost.filecoin.io/v1/json-rpc-api#dealssetconsideronlineretrievaldeals) * [DealsSetConsiderOnlineStorageDeals](https://boost.filecoin.io/v1/json-rpc-api#dealssetconsideronlinestoragedeals) * [DealsSetConsiderUnverifiedStorageDeals](https://boost.filecoin.io/v1/json-rpc-api#dealssetconsiderunverifiedstoragedeals) * [DealsSetConsiderVerifiedStorageDeals](https://boost.filecoin.io/v1/json-rpc-api#dealssetconsiderverifiedstoragedeals) * [DealsSetPieceCidBlocklist](https://boost.filecoin.io/v1/json-rpc-api#dealssetpiececidblocklist) * [I](https://boost.filecoin.io/v1/json-rpc-api#i) * [ID](https://boost.filecoin.io/v1/json-rpc-api#id) * [Log](https://boost.filecoin.io/v1/json-rpc-api#log) * [LogList](https://boost.filecoin.io/v1/json-rpc-api#loglist) * [LogSetLevel](https://boost.filecoin.io/v1/json-rpc-api#logsetlevel) * [Market](https://boost.filecoin.io/v1/json-rpc-api#market) * [MarketCancelDataTransfer](https://boost.filecoin.io/v1/json-rpc-api#marketcanceldatatransfer) * [MarketDataTransferUpdates](https://boost.filecoin.io/v1/json-rpc-api#marketdatatransferupdates) * [MarketGetAsk](https://boost.filecoin.io/v1/json-rpc-api#marketgetask) * [MarketGetRetrievalAsk](https://boost.filecoin.io/v1/json-rpc-api#marketgetretrievalask) * [MarketImportDealData](https://boost.filecoin.io/v1/json-rpc-api#marketimportdealdata) * [MarketListDataTransfers](https://boost.filecoin.io/v1/json-rpc-api#marketlistdatatransfers) * [MarketListIncompleteDeals](https://boost.filecoin.io/v1/json-rpc-api#marketlistincompletedeals) * [MarketListRetrievalDeals](https://boost.filecoin.io/v1/json-rpc-api#marketlistretrievaldeals) * [MarketPendingDeals](https://boost.filecoin.io/v1/json-rpc-api#marketpendingdeals) * [MarketRestartDataTransfer](https://boost.filecoin.io/v1/json-rpc-api#marketrestartdatatransfer) * [MarketSetAsk](https://boost.filecoin.io/v1/json-rpc-api#marketsetask) * [MarketSetRetrievalAsk](https://boost.filecoin.io/v1/json-rpc-api#marketsetretrievalask) * [Net](https://boost.filecoin.io/v1/json-rpc-api#net) * [NetAddrsListen](https://boost.filecoin.io/v1/json-rpc-api#netaddrslisten) * [NetAgentVersion](https://boost.filecoin.io/v1/json-rpc-api#netagentversion) * [NetAutoNatStatus](https://boost.filecoin.io/v1/json-rpc-api#netautonatstatus) * [NetBandwidthStats](https://boost.filecoin.io/v1/json-rpc-api#netbandwidthstats) * [NetBandwidthStatsByPeer](https://boost.filecoin.io/v1/json-rpc-api#netbandwidthstatsbypeer) * [NetBandwidthStatsByProtocol](https://boost.filecoin.io/v1/json-rpc-api#netbandwidthstatsbyprotocol) * [NetBlockAdd](https://boost.filecoin.io/v1/json-rpc-api#netblockadd) * [NetBlockList](https://boost.filecoin.io/v1/json-rpc-api#netblocklist) * [NetBlockRemove](https://boost.filecoin.io/v1/json-rpc-api#netblockremove) * [NetConnect](https://boost.filecoin.io/v1/json-rpc-api#netconnect) * [NetConnectedness](https://boost.filecoin.io/v1/json-rpc-api#netconnectedness) * [NetDisconnect](https://boost.filecoin.io/v1/json-rpc-api#netdisconnect) * [NetFindPeer](https://boost.filecoin.io/v1/json-rpc-api#netfindpeer) * [NetLimit](https://boost.filecoin.io/v1/json-rpc-api#netlimit) * [NetPeerInfo](https://boost.filecoin.io/v1/json-rpc-api#netpeerinfo) * [NetPeers](https://boost.filecoin.io/v1/json-rpc-api#netpeers) * [NetPing](https://boost.filecoin.io/v1/json-rpc-api#netping) * [NetProtectAdd](https://boost.filecoin.io/v1/json-rpc-api#netprotectadd) * [NetProtectList](https://boost.filecoin.io/v1/json-rpc-api#netprotectlist) * [NetProtectRemove](https://boost.filecoin.io/v1/json-rpc-api#netprotectremove) * [NetPubsubScores](https://boost.filecoin.io/v1/json-rpc-api#netpubsubscores) * [NetSetLimit](https://boost.filecoin.io/v1/json-rpc-api#netsetlimit) * [NetStat](https://boost.filecoin.io/v1/json-rpc-api#netstat) * [Online](https://boost.filecoin.io/v1/json-rpc-api#online) * [OnlineBackup](https://boost.filecoin.io/v1/json-rpc-api#onlinebackup) * [Pieces](https://boost.filecoin.io/v1/json-rpc-api#pieces) * [PiecesGetCIDInfo](https://boost.filecoin.io/v1/json-rpc-api#piecesgetcidinfo) * [PiecesGetMaxOffset](https://boost.filecoin.io/v1/json-rpc-api#piecesgetmaxoffset) * [PiecesGetPieceInfo](https://boost.filecoin.io/v1/json-rpc-api#piecesgetpieceinfo) * [PiecesListCidInfos](https://boost.filecoin.io/v1/json-rpc-api#pieceslistcidinfos) * [PiecesListPieces](https://boost.filecoin.io/v1/json-rpc-api#pieceslistpieces) * [Runtime](https://boost.filecoin.io/v1/json-rpc-api#runtime) * [RuntimeSubsystems](https://boost.filecoin.io/v1/json-rpc-api#runtimesubsystems) * [Sectors](https://boost.filecoin.io/v1/json-rpc-api#sectors) * [SectorsRefs](https://boost.filecoin.io/v1/json-rpc-api#sectorsrefs) [hashtag](https://boost.filecoin.io/v1/json-rpc-api#actor) Actor --------------------------------------------------------------------- ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#actorsectorsize) ActorSectorSize There are not yet any comments for this method. Perms: read Inputs: Response: `34359738368` [hashtag](https://boost.filecoin.io/v1/json-rpc-api#auth) Auth ------------------------------------------------------------------- ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#authnew) AuthNew Perms: admin Inputs: Response: `"Ynl0ZSBhcnJheQ=="` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#authverify) AuthVerify Perms: read Inputs: Response: [hashtag](https://boost.filecoin.io/v1/json-rpc-api#blockstore) Blockstore ------------------------------------------------------------------------------- ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#blockstoreget) BlockstoreGet There are not yet any comments for this method. Perms: read Inputs: Response: `"Ynl0ZSBhcnJheQ=="` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#blockstoregetsize) BlockstoreGetSize Perms: read Inputs: Response: `123` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#blockstorehas) BlockstoreHas Perms: read Inputs: Response: `true` [hashtag](https://boost.filecoin.io/v1/json-rpc-api#boost) Boost --------------------------------------------------------------------- ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#boostdagstoredestroyshard) BoostDagstoreDestroyShard Perms: admin Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#boostdagstoregc) BoostDagstoreGC Perms: admin Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#boostdagstoreinitializeall) BoostDagstoreInitializeAll Perms: admin Inputs: Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#boostdagstoreinitializeshard) BoostDagstoreInitializeShard Perms: admin Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#boostdagstorelistshards) BoostDagstoreListShards Perms: admin Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#boostdagstorepiecescontainingmultihash) BoostDagstorePiecesContainingMultihash Perms: read Inputs: Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#boostdagstorerecovershard) BoostDagstoreRecoverShard Perms: admin Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#boostdagstoreregistershard) BoostDagstoreRegisterShard Perms: admin Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#boostdeal) BoostDeal Perms: admin Inputs: Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#boostdealbysignedproposalcid) BoostDealBySignedProposalCid Perms: admin Inputs: Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#boostdummydeal) BoostDummyDeal Perms: admin Inputs: Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#boostindexerannouncealldeals) BoostIndexerAnnounceAllDeals There are not yet any comments for this method. Perms: admin Inputs: `null` Response: `{}` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#boostmakedeal) BoostMakeDeal Perms: write Inputs: Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#boostofflinedealwithdata) BoostOfflineDealWithData Perms: admin Inputs: Response: [hashtag](https://boost.filecoin.io/v1/json-rpc-api#deals) Deals --------------------------------------------------------------------- ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#dealsconsiderofflineretrievaldeals) DealsConsiderOfflineRetrievalDeals Perms: admin Inputs: `null` Response: `true` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#dealsconsiderofflinestoragedeals) DealsConsiderOfflineStorageDeals Perms: admin Inputs: `null` Response: `true` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#dealsconsideronlineretrievaldeals) DealsConsiderOnlineRetrievalDeals Perms: admin Inputs: `null` Response: `true` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#dealsconsideronlinestoragedeals) DealsConsiderOnlineStorageDeals There are not yet any comments for this method. Perms: admin Inputs: `null` Response: `true` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#dealsconsiderunverifiedstoragedeals) DealsConsiderUnverifiedStorageDeals Perms: admin Inputs: `null` Response: `true` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#dealsconsiderverifiedstoragedeals) DealsConsiderVerifiedStorageDeals Perms: admin Inputs: `null` Response: `true` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#dealspiececidblocklist) DealsPieceCidBlocklist Perms: admin Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#dealssetconsiderofflineretrievaldeals) DealsSetConsiderOfflineRetrievalDeals Perms: admin Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#dealssetconsiderofflinestoragedeals) DealsSetConsiderOfflineStorageDeals Perms: admin Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#dealssetconsideronlineretrievaldeals) DealsSetConsiderOnlineRetrievalDeals Perms: admin Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#dealssetconsideronlinestoragedeals) DealsSetConsiderOnlineStorageDeals Perms: admin Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#dealssetconsiderunverifiedstoragedeals) DealsSetConsiderUnverifiedStorageDeals Perms: admin Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#dealssetconsiderverifiedstoragedeals) DealsSetConsiderVerifiedStorageDeals Perms: admin Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#dealssetpiececidblocklist) DealsSetPieceCidBlocklist Perms: admin Inputs: Response: `{}` [hashtag](https://boost.filecoin.io/v1/json-rpc-api#i) I ------------------------------------------------------------- ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#id) ID Perms: read Inputs: `null` Response: `"12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"` [hashtag](https://boost.filecoin.io/v1/json-rpc-api#log) Log ----------------------------------------------------------------- ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#loglist) LogList Perms: write Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#logsetlevel) LogSetLevel Perms: write Inputs: Response: `{}` [hashtag](https://boost.filecoin.io/v1/json-rpc-api#market) Market ----------------------------------------------------------------------- ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#marketcanceldatatransfer) MarketCancelDataTransfer Perms: write Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#marketdatatransferupdates) MarketDataTransferUpdates Perms: write Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#marketgetask) MarketGetAsk Perms: read Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#marketgetretrievalask) MarketGetRetrievalAsk Perms: read Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#marketimportdealdata) MarketImportDealData Perms: write Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#marketlistdatatransfers) MarketListDataTransfers Perms: write Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#marketlistincompletedeals) MarketListIncompleteDeals Perms: read Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#marketlistretrievaldeals) MarketListRetrievalDeals There are not yet any comments for this method. Perms: read Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#marketpendingdeals) MarketPendingDeals Perms: write Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#marketrestartdatatransfer) MarketRestartDataTransfer Perms: write Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#marketsetask) MarketSetAsk Perms: admin Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#marketsetretrievalask) MarketSetRetrievalAsk Perms: admin Inputs: Response: `{}` [hashtag](https://boost.filecoin.io/v1/json-rpc-api#net) Net ----------------------------------------------------------------- ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#netaddrslisten) NetAddrsListen Perms: read Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#netagentversion) NetAgentVersion Perms: read Inputs: Response: `"string value"` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#netautonatstatus) NetAutoNatStatus Perms: read Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#netbandwidthstats) NetBandwidthStats Perms: read Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#netbandwidthstatsbypeer) NetBandwidthStatsByPeer Perms: read Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#netbandwidthstatsbyprotocol) NetBandwidthStatsByProtocol Perms: read Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#netblockadd) NetBlockAdd Perms: admin Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#netblocklist) NetBlockList Perms: read Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#netblockremove) NetBlockRemove Perms: admin Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#netconnect) NetConnect Perms: write Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#netconnectedness) NetConnectedness Perms: read Inputs: Response: `1` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#netdisconnect) NetDisconnect Perms: write Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#netfindpeer) NetFindPeer Perms: read Inputs: Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#netlimit) NetLimit Perms: read Inputs: Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#netpeerinfo) NetPeerInfo Perms: read Inputs: Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#netpeers) NetPeers Perms: read Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#netping) NetPing Perms: read Inputs: Response: `60000000000` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#netprotectadd) NetProtectAdd Perms: admin Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#netprotectlist) NetProtectList Perms: read Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#netprotectremove) NetProtectRemove Perms: admin Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#netpubsubscores) NetPubsubScores Perms: read Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#netsetlimit) NetSetLimit Perms: admin Inputs: Response: `{}` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#netstat) NetStat Perms: read Inputs: Response: [hashtag](https://boost.filecoin.io/v1/json-rpc-api#online) Online ----------------------------------------------------------------------- ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#onlinebackup) OnlineBackup There are not yet any comments for this method. Perms: admin Inputs: Response: `{}` [hashtag](https://boost.filecoin.io/v1/json-rpc-api#pieces) Pieces ----------------------------------------------------------------------- ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#piecesgetcidinfo) PiecesGetCIDInfo Perms: read Inputs: Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#piecesgetmaxoffset) PiecesGetMaxOffset Perms: read Inputs: Response: `42` ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#piecesgetpieceinfo) PiecesGetPieceInfo Perms: read Inputs: Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#pieceslistcidinfos) PiecesListCidInfos Perms: read Inputs: `null` Response: ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#pieceslistpieces) PiecesListPieces Perms: read Inputs: `null` Response: [hashtag](https://boost.filecoin.io/v1/json-rpc-api#runtime) Runtime ------------------------------------------------------------------------- ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#runtimesubsystems) RuntimeSubsystems RuntimeSubsystems returns the subsystems that are enabled in this instance. Perms: read Inputs: `null` Response: [hashtag](https://boost.filecoin.io/v1/json-rpc-api#sectors) Sectors ------------------------------------------------------------------------- ### [hashtag](https://boost.filecoin.io/v1/json-rpc-api#sectorsrefs) SectorsRefs Perms: read Inputs: `null` Response: [PreviousGraphQL APIchevron-left](https://boost.filecoin.io/v1/graphql-api) [NextFAQchevron-right](https://boost.filecoin.io/v1/faq) Last updated 2 years ago * [Go JSON-RPC client](https://boost.filecoin.io/v1/json-rpc-api#go-json-rpc-client) * [Python JSON-RPC client](https://boost.filecoin.io/v1/json-rpc-api#python-json-rpc-client) * [Groups](https://boost.filecoin.io/v1/json-rpc-api#groups) * [Actor](https://boost.filecoin.io/v1/json-rpc-api#actor) * [ActorSectorSize](https://boost.filecoin.io/v1/json-rpc-api#actorsectorsize) * [Auth](https://boost.filecoin.io/v1/json-rpc-api#auth) * [AuthNew](https://boost.filecoin.io/v1/json-rpc-api#authnew) * [AuthVerify](https://boost.filecoin.io/v1/json-rpc-api#authverify) * [Blockstore](https://boost.filecoin.io/v1/json-rpc-api#blockstore) * [BlockstoreGet](https://boost.filecoin.io/v1/json-rpc-api#blockstoreget) * [BlockstoreGetSize](https://boost.filecoin.io/v1/json-rpc-api#blockstoregetsize) * [BlockstoreHas](https://boost.filecoin.io/v1/json-rpc-api#blockstorehas) * [Boost](https://boost.filecoin.io/v1/json-rpc-api#boost) * [BoostDagstoreDestroyShard](https://boost.filecoin.io/v1/json-rpc-api#boostdagstoredestroyshard) * [BoostDagstoreGC](https://boost.filecoin.io/v1/json-rpc-api#boostdagstoregc) * [BoostDagstoreInitializeAll](https://boost.filecoin.io/v1/json-rpc-api#boostdagstoreinitializeall) * [BoostDagstoreInitializeShard](https://boost.filecoin.io/v1/json-rpc-api#boostdagstoreinitializeshard) * [BoostDagstoreListShards](https://boost.filecoin.io/v1/json-rpc-api#boostdagstorelistshards) * [BoostDagstorePiecesContainingMultihash](https://boost.filecoin.io/v1/json-rpc-api#boostdagstorepiecescontainingmultihash) * [BoostDagstoreRecoverShard](https://boost.filecoin.io/v1/json-rpc-api#boostdagstorerecovershard) * [BoostDagstoreRegisterShard](https://boost.filecoin.io/v1/json-rpc-api#boostdagstoreregistershard) * [BoostDeal](https://boost.filecoin.io/v1/json-rpc-api#boostdeal) * [BoostDealBySignedProposalCid](https://boost.filecoin.io/v1/json-rpc-api#boostdealbysignedproposalcid) * [BoostDummyDeal](https://boost.filecoin.io/v1/json-rpc-api#boostdummydeal) * [BoostIndexerAnnounceAllDeals](https://boost.filecoin.io/v1/json-rpc-api#boostindexerannouncealldeals) * [BoostMakeDeal](https://boost.filecoin.io/v1/json-rpc-api#boostmakedeal) * [BoostOfflineDealWithData](https://boost.filecoin.io/v1/json-rpc-api#boostofflinedealwithdata) * [Deals](https://boost.filecoin.io/v1/json-rpc-api#deals) * [DealsConsiderOfflineRetrievalDeals](https://boost.filecoin.io/v1/json-rpc-api#dealsconsiderofflineretrievaldeals) * [DealsConsiderOfflineStorageDeals](https://boost.filecoin.io/v1/json-rpc-api#dealsconsiderofflinestoragedeals) * [DealsConsiderOnlineRetrievalDeals](https://boost.filecoin.io/v1/json-rpc-api#dealsconsideronlineretrievaldeals) * [DealsConsiderOnlineStorageDeals](https://boost.filecoin.io/v1/json-rpc-api#dealsconsideronlinestoragedeals) * [DealsConsiderUnverifiedStorageDeals](https://boost.filecoin.io/v1/json-rpc-api#dealsconsiderunverifiedstoragedeals) * [DealsConsiderVerifiedStorageDeals](https://boost.filecoin.io/v1/json-rpc-api#dealsconsiderverifiedstoragedeals) * [DealsPieceCidBlocklist](https://boost.filecoin.io/v1/json-rpc-api#dealspiececidblocklist) * [DealsSetConsiderOfflineRetrievalDeals](https://boost.filecoin.io/v1/json-rpc-api#dealssetconsiderofflineretrievaldeals) * [DealsSetConsiderOfflineStorageDeals](https://boost.filecoin.io/v1/json-rpc-api#dealssetconsiderofflinestoragedeals) * [DealsSetConsiderOnlineRetrievalDeals](https://boost.filecoin.io/v1/json-rpc-api#dealssetconsideronlineretrievaldeals) * [DealsSetConsiderOnlineStorageDeals](https://boost.filecoin.io/v1/json-rpc-api#dealssetconsideronlinestoragedeals) * [DealsSetConsiderUnverifiedStorageDeals](https://boost.filecoin.io/v1/json-rpc-api#dealssetconsiderunverifiedstoragedeals) * [DealsSetConsiderVerifiedStorageDeals](https://boost.filecoin.io/v1/json-rpc-api#dealssetconsiderverifiedstoragedeals) * [DealsSetPieceCidBlocklist](https://boost.filecoin.io/v1/json-rpc-api#dealssetpiececidblocklist) * [I](https://boost.filecoin.io/v1/json-rpc-api#i) * [ID](https://boost.filecoin.io/v1/json-rpc-api#id) * [Log](https://boost.filecoin.io/v1/json-rpc-api#log) * [LogList](https://boost.filecoin.io/v1/json-rpc-api#loglist) * [LogSetLevel](https://boost.filecoin.io/v1/json-rpc-api#logsetlevel) * [Market](https://boost.filecoin.io/v1/json-rpc-api#market) * [MarketCancelDataTransfer](https://boost.filecoin.io/v1/json-rpc-api#marketcanceldatatransfer) * [MarketDataTransferUpdates](https://boost.filecoin.io/v1/json-rpc-api#marketdatatransferupdates) * [MarketGetAsk](https://boost.filecoin.io/v1/json-rpc-api#marketgetask) * [MarketGetRetrievalAsk](https://boost.filecoin.io/v1/json-rpc-api#marketgetretrievalask) * [MarketImportDealData](https://boost.filecoin.io/v1/json-rpc-api#marketimportdealdata) * [MarketListDataTransfers](https://boost.filecoin.io/v1/json-rpc-api#marketlistdatatransfers) * [MarketListIncompleteDeals](https://boost.filecoin.io/v1/json-rpc-api#marketlistincompletedeals) * [MarketListRetrievalDeals](https://boost.filecoin.io/v1/json-rpc-api#marketlistretrievaldeals) * [MarketPendingDeals](https://boost.filecoin.io/v1/json-rpc-api#marketpendingdeals) * [MarketRestartDataTransfer](https://boost.filecoin.io/v1/json-rpc-api#marketrestartdatatransfer) * [MarketSetAsk](https://boost.filecoin.io/v1/json-rpc-api#marketsetask) * [MarketSetRetrievalAsk](https://boost.filecoin.io/v1/json-rpc-api#marketsetretrievalask) * [Net](https://boost.filecoin.io/v1/json-rpc-api#net) * [NetAddrsListen](https://boost.filecoin.io/v1/json-rpc-api#netaddrslisten) * [NetAgentVersion](https://boost.filecoin.io/v1/json-rpc-api#netagentversion) * [NetAutoNatStatus](https://boost.filecoin.io/v1/json-rpc-api#netautonatstatus) * [NetBandwidthStats](https://boost.filecoin.io/v1/json-rpc-api#netbandwidthstats) * [NetBandwidthStatsByPeer](https://boost.filecoin.io/v1/json-rpc-api#netbandwidthstatsbypeer) * [NetBandwidthStatsByProtocol](https://boost.filecoin.io/v1/json-rpc-api#netbandwidthstatsbyprotocol) * [NetBlockAdd](https://boost.filecoin.io/v1/json-rpc-api#netblockadd) * [NetBlockList](https://boost.filecoin.io/v1/json-rpc-api#netblocklist) * [NetBlockRemove](https://boost.filecoin.io/v1/json-rpc-api#netblockremove) * [NetConnect](https://boost.filecoin.io/v1/json-rpc-api#netconnect) * [NetConnectedness](https://boost.filecoin.io/v1/json-rpc-api#netconnectedness) * [NetDisconnect](https://boost.filecoin.io/v1/json-rpc-api#netdisconnect) * [NetFindPeer](https://boost.filecoin.io/v1/json-rpc-api#netfindpeer) * [NetLimit](https://boost.filecoin.io/v1/json-rpc-api#netlimit) * [NetPeerInfo](https://boost.filecoin.io/v1/json-rpc-api#netpeerinfo) * [NetPeers](https://boost.filecoin.io/v1/json-rpc-api#netpeers) * [NetPing](https://boost.filecoin.io/v1/json-rpc-api#netping) * [NetProtectAdd](https://boost.filecoin.io/v1/json-rpc-api#netprotectadd) * [NetProtectList](https://boost.filecoin.io/v1/json-rpc-api#netprotectlist) * [NetProtectRemove](https://boost.filecoin.io/v1/json-rpc-api#netprotectremove) * [NetPubsubScores](https://boost.filecoin.io/v1/json-rpc-api#netpubsubscores) * [NetSetLimit](https://boost.filecoin.io/v1/json-rpc-api#netsetlimit) * [NetStat](https://boost.filecoin.io/v1/json-rpc-api#netstat) * [Online](https://boost.filecoin.io/v1/json-rpc-api#online) * [OnlineBackup](https://boost.filecoin.io/v1/json-rpc-api#onlinebackup) * [Pieces](https://boost.filecoin.io/v1/json-rpc-api#pieces) * [PiecesGetCIDInfo](https://boost.filecoin.io/v1/json-rpc-api#piecesgetcidinfo) * [PiecesGetMaxOffset](https://boost.filecoin.io/v1/json-rpc-api#piecesgetmaxoffset) * [PiecesGetPieceInfo](https://boost.filecoin.io/v1/json-rpc-api#piecesgetpieceinfo) * [PiecesListCidInfos](https://boost.filecoin.io/v1/json-rpc-api#pieceslistcidinfos) * [PiecesListPieces](https://boost.filecoin.io/v1/json-rpc-api#pieceslistpieces) * [Runtime](https://boost.filecoin.io/v1/json-rpc-api#runtime) * [RuntimeSubsystems](https://boost.filecoin.io/v1/json-rpc-api#runtimesubsystems) * [Sectors](https://boost.filecoin.io/v1/json-rpc-api#sectors) * [SectorsRefs](https://boost.filecoin.io/v1/json-rpc-api#sectorsrefs) sun-brightdesktopmoon Copy import requests import json def main(): url = "http://localhost:3051/rpc/v0" headers = {'content-type': 'application/json', "Authorization": "Bearer "} payload = { "method": "Filecoin.BoostOfflineDealWithData", "params": [\ "",\ "",\ True\ ], "jsonrpc": "2.0", "id": 1, } response = requests.post(url, data=json.dumps(payload), headers=headers) print(response.text) if __name__ == "__main__": main() Copy [\ "f01234"\ ] Copy [\ [\ "write"\ ]\ ] Copy [\ "string value"\ ] Copy [\ "write"\ ] Copy [\ {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ }\ ] Copy [\ {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ }\ ] Copy [\ {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ }\ ] Copy [\ "string value"\ ] Copy [\ {\ "Key": "baga6ea4seaqecmtz7iak33dsfshi627abz4i4665dfuzr3qfs4bmad6dx3iigdq",\ "Success": false,\ "Error": "\u003cerror\u003e"\ }\ ] Copy [\ {\ "MaxConcurrency": 123,\ "IncludeSealed": true\ }\ ] Copy { "Key": "string value", "Event": "string value", "Success": true, "Error": "string value", "Total": 123, "Current": 123 } Copy [\ "string value"\ ] Copy [\ {\ "Key": "baga6ea4seaqecmtz7iak33dsfshi627abz4i4665dfuzr3qfs4bmad6dx3iigdq",\ "State": "ShardStateAvailable",\ "Error": "\u003cerror\u003e"\ }\ ] Copy [\ "Bw=="\ ] Copy [\ {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ }\ ] Copy [\ "string value"\ ] Copy [\ "string value"\ ] Copy [\ "07070707-0707-0707-0707-070707070707"\ ] Copy { "DealUuid": "07070707-0707-0707-0707-070707070707", "CreatedAt": "0001-01-01T00:00:00Z", "ClientDealProposal": { "Proposal": { "PieceCID": { "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4" }, "PieceSize": 1032, "VerifiedDeal": true, "Client": "f01234", "Provider": "f01234", "Label": "", "StartEpoch": 10101, "EndEpoch": 10101, "StoragePricePerEpoch": "0", "ProviderCollateral": "0", "ClientCollateral": "0" }, "ClientSignature": { "Type": 2, "Data": "Ynl0ZSBhcnJheQ==" } }, "IsOffline": true, "CleanupData": true, "ClientPeerID": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf", "DealDataRoot": { "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4" }, "InboundFilePath": "string value", "Transfer": { "Type": "string value", "ClientID": "string value", "Params": "Ynl0ZSBhcnJheQ==", "Size": 42 }, "ChainDealID": 5432, "PublishCID": null, "SectorID": 9, "Offset": 1032, "Length": 1032, "Checkpoint": 1, "CheckpointAt": "0001-01-01T00:00:00Z", "Err": "string value", "Retry": "auto", "NBytesReceived": 9, "FastRetrieval": true, "AnnounceToIPNI": true } Copy [\ {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ }\ ] Copy { "DealUuid": "07070707-0707-0707-0707-070707070707", "CreatedAt": "0001-01-01T00:00:00Z", "ClientDealProposal": { "Proposal": { "PieceCID": { "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4" }, "PieceSize": 1032, "VerifiedDeal": true, "Client": "f01234", "Provider": "f01234", "Label": "", "StartEpoch": 10101, "EndEpoch": 10101, "StoragePricePerEpoch": "0", "ProviderCollateral": "0", "ClientCollateral": "0" }, "ClientSignature": { "Type": 2, "Data": "Ynl0ZSBhcnJheQ==" } }, "IsOffline": true, "CleanupData": true, "ClientPeerID": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf", "DealDataRoot": { "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4" }, "InboundFilePath": "string value", "Transfer": { "Type": "string value", "ClientID": "string value", "Params": "Ynl0ZSBhcnJheQ==", "Size": 42 }, "ChainDealID": 5432, "PublishCID": null, "SectorID": 9, "Offset": 1032, "Length": 1032, "Checkpoint": 1, "CheckpointAt": "0001-01-01T00:00:00Z", "Err": "string value", "Retry": "auto", "NBytesReceived": 9, "FastRetrieval": true, "AnnounceToIPNI": true } Copy [\ {\ "DealUUID": "07070707-0707-0707-0707-070707070707",\ "IsOffline": true,\ "ClientDealProposal": {\ "Proposal": {\ "PieceCID": {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ },\ "PieceSize": 1032,\ "VerifiedDeal": true,\ "Client": "f01234",\ "Provider": "f01234",\ "Label": "",\ "StartEpoch": 10101,\ "EndEpoch": 10101,\ "StoragePricePerEpoch": "0",\ "ProviderCollateral": "0",\ "ClientCollateral": "0"\ },\ "ClientSignature": {\ "Type": 2,\ "Data": "Ynl0ZSBhcnJheQ=="\ }\ },\ "DealDataRoot": {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ },\ "Transfer": {\ "Type": "string value",\ "ClientID": "string value",\ "Params": "Ynl0ZSBhcnJheQ==",\ "Size": 42\ },\ "RemoveUnsealedCopy": true,\ "SkipIPNIAnnounce": true\ }\ ] Copy { "Accepted": true, "Reason": "string value" } Copy [\ {\ "DealUUID": "07070707-0707-0707-0707-070707070707",\ "IsOffline": true,\ "ClientDealProposal": {\ "Proposal": {\ "PieceCID": {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ },\ "PieceSize": 1032,\ "VerifiedDeal": true,\ "Client": "f01234",\ "Provider": "f01234",\ "Label": "",\ "StartEpoch": 10101,\ "EndEpoch": 10101,\ "StoragePricePerEpoch": "0",\ "ProviderCollateral": "0",\ "ClientCollateral": "0"\ },\ "ClientSignature": {\ "Type": 2,\ "Data": "Ynl0ZSBhcnJheQ=="\ }\ },\ "DealDataRoot": {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ },\ "Transfer": {\ "Type": "string value",\ "ClientID": "string value",\ "Params": "Ynl0ZSBhcnJheQ==",\ "Size": 42\ },\ "RemoveUnsealedCopy": true,\ "SkipIPNIAnnounce": true\ }\ ] Copy { "Accepted": true, "Reason": "string value" } Copy [\ "07070707-0707-0707-0707-070707070707",\ "string value",\ true\ ] Copy { "Accepted": true, "Reason": "string value" } Copy [\ {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ }\ ] Copy [\ true\ ] Copy [\ true\ ] Copy [\ true\ ] Copy [\ true\ ] Copy [\ true\ ] Copy [\ true\ ] Copy [\ [\ {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ }\ ]\ ] Copy [\ "string value"\ ] Copy [\ "string value",\ "string value"\ ] Copy [\ 3,\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf",\ true\ ] Copy { "TransferID": 3, "Status": 1, "BaseCID": { "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4" }, "IsInitiator": true, "IsSender": true, "Voucher": "string value", "Message": "string value", "OtherPeer": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf", "Transferred": 42, "Stages": { "Stages": [\ {\ "Name": "string value",\ "Description": "string value",\ "CreatedTime": "0001-01-01T00:00:00Z",\ "UpdatedTime": "0001-01-01T00:00:00Z",\ "Logs": [\ {\ "Log": "string value",\ "UpdatedTime": "0001-01-01T00:00:00Z"\ }\ ]\ }\ ] } } Copy { "Ask": { "Price": "0", "VerifiedPrice": "0", "MinPieceSize": 1032, "MaxPieceSize": 1032, "Miner": "f01234", "Timestamp": 10101, "Expiry": 10101, "SeqNo": 42 }, "Signature": { "Type": 2, "Data": "Ynl0ZSBhcnJheQ==" } } Copy { "PricePerByte": "0", "UnsealPrice": "0", "PaymentInterval": 42, "PaymentIntervalIncrease": 42 } Copy [\ {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ },\ "string value"\ ] Copy [\ {\ "TransferID": 3,\ "Status": 1,\ "BaseCID": {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ },\ "IsInitiator": true,\ "IsSender": true,\ "Voucher": "string value",\ "Message": "string value",\ "OtherPeer": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf",\ "Transferred": 42,\ "Stages": {\ "Stages": [\ {\ "Name": "string value",\ "Description": "string value",\ "CreatedTime": "0001-01-01T00:00:00Z",\ "UpdatedTime": "0001-01-01T00:00:00Z",\ "Logs": [\ {\ "Log": "string value",\ "UpdatedTime": "0001-01-01T00:00:00Z"\ }\ ]\ }\ ]\ }\ }\ ] Copy [\ {\ "Proposal": {\ "PieceCID": {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ },\ "PieceSize": 1032,\ "VerifiedDeal": true,\ "Client": "f01234",\ "Provider": "f01234",\ "Label": "",\ "StartEpoch": 10101,\ "EndEpoch": 10101,\ "StoragePricePerEpoch": "0",\ "ProviderCollateral": "0",\ "ClientCollateral": "0"\ },\ "ClientSignature": {\ "Type": 2,\ "Data": "Ynl0ZSBhcnJheQ=="\ },\ "ProposalCid": {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ },\ "AddFundsCid": null,\ "PublishCid": null,\ "Miner": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf",\ "Client": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf",\ "State": 42,\ "PiecePath": ".lotusminer/fstmp123",\ "MetadataPath": ".lotusminer/fstmp123",\ "SlashEpoch": 10101,\ "FastRetrieval": true,\ "Message": "string value",\ "FundsReserved": "0",\ "Ref": {\ "TransferType": "string value",\ "Root": {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ },\ "PieceCid": null,\ "PieceSize": 1024,\ "RawBlockSize": 42\ },\ "AvailableForRetrieval": true,\ "DealID": 5432,\ "CreationTime": "0001-01-01T00:00:00Z",\ "TransferChannelId": {\ "Initiator": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf",\ "Responder": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf",\ "ID": 3\ },\ "SectorNumber": 9,\ "InboundCAR": "string value"\ }\ ] Copy [\ {\ "PayloadCID": {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ },\ "ID": 5,\ "Selector": {\ "Raw": "Ynl0ZSBhcnJheQ=="\ },\ "PieceCID": null,\ "PricePerByte": "0",\ "PaymentInterval": 42,\ "PaymentIntervalIncrease": 42,\ "UnsealPrice": "0",\ "StoreID": 42,\ "ChannelID": {\ "Initiator": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf",\ "Responder": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf",\ "ID": 3\ },\ "PieceInfo": {\ "PieceCID": {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ },\ "Deals": [\ {\ "DealID": 5432,\ "SectorID": 9,\ "Offset": 1032,\ "Length": 1032\ }\ ]\ },\ "Status": 0,\ "Receiver": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf",\ "TotalSent": 42,\ "FundsReceived": "0",\ "Message": "string value",\ "CurrentInterval": 42,\ "LegacyProtocol": true\ }\ ] Copy { "Deals": [\ {\ "Proposal": {\ "PieceCID": {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ },\ "PieceSize": 1032,\ "VerifiedDeal": true,\ "Client": "f01234",\ "Provider": "f01234",\ "Label": "",\ "StartEpoch": 10101,\ "EndEpoch": 10101,\ "StoragePricePerEpoch": "0",\ "ProviderCollateral": "0",\ "ClientCollateral": "0"\ },\ "ClientSignature": {\ "Type": 2,\ "Data": "Ynl0ZSBhcnJheQ=="\ }\ }\ ], "PublishPeriodStart": "0001-01-01T00:00:00Z", "PublishPeriod": 60000000000 } Copy [\ 3,\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf",\ true\ ] Copy [\ "0",\ "0",\ 10101,\ 1032,\ 1032\ ] Copy [\ {\ "PricePerByte": "0",\ "UnsealPrice": "0",\ "PaymentInterval": 42,\ "PaymentIntervalIncrease": 42\ }\ ] Copy { "ID": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf", "Addrs": [\ "/ip4/52.36.61.156/tcp/1347/p2p/12D3KooWFETiESTf1v4PGUvtnxMAcEFMzLZbJGg4tjWfGEimYior"\ ] } Copy [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ] Copy { "Reachability": 1, "PublicAddr": "string value" } Copy { "TotalIn": 9, "TotalOut": 9, "RateIn": 12.3, "RateOut": 12.3 } Copy { "12D3KooWSXmXLJmBR1M7i9RW9GQPNUhZSzXKzxDHWtAgNuJAbyEJ": { "TotalIn": 174000, "TotalOut": 12500, "RateIn": 100, "RateOut": 50 } } Copy { "/fil/hello/1.0.0": { "TotalIn": 174000, "TotalOut": 12500, "RateIn": 100, "RateOut": 50 } } Copy [\ {\ "Peers": [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ],\ "IPAddrs": [\ "string value"\ ],\ "IPSubnets": [\ "string value"\ ]\ }\ ] Copy { "Peers": [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ], "IPAddrs": [\ "string value"\ ], "IPSubnets": [\ "string value"\ ] } Copy [\ {\ "Peers": [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ],\ "IPAddrs": [\ "string value"\ ],\ "IPSubnets": [\ "string value"\ ]\ }\ ] Copy [\ {\ "ID": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf",\ "Addrs": [\ "/ip4/52.36.61.156/tcp/1347/p2p/12D3KooWFETiESTf1v4PGUvtnxMAcEFMzLZbJGg4tjWfGEimYior"\ ]\ }\ ] Copy [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ] Copy [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ] Copy [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ] Copy { "ID": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf", "Addrs": [\ "/ip4/52.36.61.156/tcp/1347/p2p/12D3KooWFETiESTf1v4PGUvtnxMAcEFMzLZbJGg4tjWfGEimYior"\ ] } Copy [\ "string value"\ ] Copy { "Memory": 9, "Streams": 123, "StreamsInbound": 123, "StreamsOutbound": 123, "Conns": 123, "ConnsInbound": 123, "ConnsOutbound": 123, "FD": 123 } Copy [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ] Copy { "ID": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf", "Agent": "string value", "Addrs": [\ "string value"\ ], "Protocols": [\ "string value"\ ], "ConnMgrMeta": { "FirstSeen": "0001-01-01T00:00:00Z", "Value": 123, "Tags": { "name": 42 }, "Conns": { "name": "2021-03-08T22:52:18Z" } } } Copy [\ {\ "ID": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf",\ "Addrs": [\ "/ip4/52.36.61.156/tcp/1347/p2p/12D3KooWFETiESTf1v4PGUvtnxMAcEFMzLZbJGg4tjWfGEimYior"\ ]\ }\ ] Copy [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ] Copy [\ [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ]\ ] Copy [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ] Copy [\ [\ "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf"\ ]\ ] Copy [\ {\ "ID": "12D3KooWGzxzKZYveHXtpG6AsrUJBcWxHBFS2HsEoGTxrMLvKXtf",\ "Score": {\ "Score": 12.3,\ "Topics": {\ "/blocks": {\ "TimeInMesh": 60000000000,\ "FirstMessageDeliveries": 122,\ "MeshMessageDeliveries": 1234,\ "InvalidMessageDeliveries": 3\ }\ },\ "AppSpecificScore": 12.3,\ "IPColocationFactor": 12.3,\ "BehaviourPenalty": 12.3\ }\ }\ ] Copy [\ "string value",\ {\ "Memory": 9,\ "Streams": 123,\ "StreamsInbound": 123,\ "StreamsOutbound": 123,\ "Conns": 123,\ "ConnsInbound": 123,\ "ConnsOutbound": 123,\ "FD": 123\ }\ ] Copy [\ "string value"\ ] Copy { "System": { "NumStreamsInbound": 123, "NumStreamsOutbound": 123, "NumConnsInbound": 123, "NumConnsOutbound": 123, "NumFD": 123, "Memory": 9 }, "Transient": { "NumStreamsInbound": 123, "NumStreamsOutbound": 123, "NumConnsInbound": 123, "NumConnsOutbound": 123, "NumFD": 123, "Memory": 9 }, "Services": { "abc": { "NumStreamsInbound": 1, "NumStreamsOutbound": 2, "NumConnsInbound": 3, "NumConnsOutbound": 4, "NumFD": 5, "Memory": 123 } }, "Protocols": { "abc": { "NumStreamsInbound": 1, "NumStreamsOutbound": 2, "NumConnsInbound": 3, "NumConnsOutbound": 4, "NumFD": 5, "Memory": 123 } }, "Peers": { "abc": { "NumStreamsInbound": 1, "NumStreamsOutbound": 2, "NumConnsInbound": 3, "NumConnsOutbound": 4, "NumFD": 5, "Memory": 123 } } } Copy [\ "string value"\ ] Copy [\ {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ }\ ] Copy { "CID": { "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4" }, "PieceBlockLocations": [\ {\ "RelOffset": 42,\ "BlockSize": 42,\ "PieceCID": {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ }\ }\ ] } Copy [\ {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ }\ ] Copy [\ {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ }\ ] Copy { "PieceCID": { "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4" }, "Deals": [\ {\ "DealID": 5432,\ "SectorID": 9,\ "Offset": 1032,\ "Length": 1032\ }\ ] } Copy [\ {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ }\ ] Copy [\ {\ "/": "bafy2bzacea3wsdh6y3a36tb3skempjoxqpuyompjbmfeyf34fi3uy6uue42v4"\ }\ ] Copy [\ "Markets"\ ] Copy { "98000": [\ {\ "SectorID": 100,\ "Offset": 10485760,\ "Size": 1048576\ }\ ] } sun-brightdesktopmoon ---