# Table of Contents - [Documentation | ElectricSQL](#documentation-electricsql) - [Auth - Guide | ElectricSQL](#auth-guide-electricsql) - [Quickstart | ElectricSQL](#quickstart-electricsql) - [Shapes - Guide | ElectricSQL](#shapes-guide-electricsql) - [Writes - Guide | ElectricSQL](#writes-guide-electricsql) - [HTTP API | ElectricSQL](#http-api-electricsql) - [Installation - Guide | ElectricSQL](#installation-guide-electricsql) - [Deployment - Guide | ElectricSQL](#deployment-guide-electricsql) - [Elixir Client | ElectricSQL](#elixir-client-electricsql) - [Client development - Guide | ElectricSQL](#client-development-guide-electricsql) - [MobX - Integrations | ElectricSQL](#mobx-integrations-electricsql) - [Sync service | ElectricSQL](#sync-service-electricsql) - [Typescript Client | ElectricSQL](#typescript-client-electricsql) - [Troubleshooting - Guide | ElectricSQL](#troubleshooting-guide-electricsql) - [Security - Guide | ElectricSQL](#security-guide-electricsql) - [LiveStore - Integrations | ElectricSQL](#livestore-integrations-electricsql) - [TanStack - Integrations | ElectricSQL](#tanstack-integrations-electricsql) - [Yjs - Integrations | ElectricSQL](#yjs-integrations-electricsql) - [Cloudflare - Integrations | ElectricSQL](#cloudflare-integrations-electricsql) - [React - Integrations | ElectricSQL](#react-integrations-electricsql) - [Redis - Integrations | ElectricSQL](#redis-integrations-electricsql) - [Phoenix - Integrations | ElectricSQL](#phoenix-integrations-electricsql) - [Next.js - Integrations | ElectricSQL](#next-js-integrations-electricsql) - [Amazon Web Services (AWS) - Integrations | ElectricSQL](#amazon-web-services-aws-integrations-electricsql) - [Crunchy Data - Integrations | ElectricSQL](#crunchy-data-integrations-electricsql) - [Neon - Integrations | ElectricSQL](#neon-integrations-electricsql) - [Digital Ocean - Integrations | ElectricSQL](#digital-ocean-integrations-electricsql) - [Expo - Integrations | ElectricSQL](#expo-integrations-electricsql) - [Fly.io - Integrations | ElectricSQL](#fly-io-integrations-electricsql) - [Google Cloud Platform (GCP) - Integrations | ElectricSQL](#google-cloud-platform-gcp-integrations-electricsql) - [Netlify - Integrations | ElectricSQL](#netlify-integrations-electricsql) - [Render - Integrations | ElectricSQL](#render-integrations-electricsql) - [Supabase - Integrations | ElectricSQL](#supabase-integrations-electricsql) - [Alternatives | ElectricSQL](#alternatives-electricsql) - [Benchmarks - Reference | ElectricSQL](#benchmarks-reference-electricsql) - [Telemetry | ElectricSQL](#telemetry-electricsql) - [Literature | ElectricSQL](#literature-electricsql) --- # Documentation | ElectricSQL Return to top ![Electric zap with halo](/img/home/zap-with-halo.svg) Documentation [​](#documentation) ================================== Welcome to the ElectricSQL developer documentation! ElectricSQL is a Postgres sync engine. Use it to sync [little subsets](/docs/guides/shapes) of your Postgres data into [local apps](/use-cases/data-sync) , services and [environments](/use-cases/dev-and-test) . Electric BETA release The Electric sync engine is now in BETA! See the [release post here](/blog/2024/12/10/electric-beta-release) . New to ElectricSQL? [​](#new-to-electricsql) --------------------------------------------- Start with the [Quickstart](/docs/quickstart) to get up-and-running. The guides on [Auth](/docs/guides/auth) , [Shapes](/docs/guides/shapes) and [Writes](/docs/guides/writes) are also good entrypoints and helpful to understand how Electric works. The [HTTP API](/docs/api/http) and [TypeScript Client](/docs/api/clients/typescript) docs show how to sync data. The [React](/docs/integrations/react) page illustrates how to bind these into a reactivity framework. The easiest way to use Electric in production is the [Electric Cloud](/product/cloud) . Alternatively, the [Deployment](/docs/guides/deployment) guide covers how to self host. Looking for PGlite docs? If you're interested in using [PGlite](/product/pglite) , it has it's own docs site at [pglite.dev/docs](https://pglite.dev/docs) Examples [​](#examples) ------------------------ See the [Demos](/demos) section and [`examples`](https://github.com/electric-sql/electric/tree/main/examples) folder on GitHub for demo apps and examples, e.g.: [![](/img/demos/linearlite-demo.png)](/demos/linearlite) [### Linearlite](/demos/linearlite) Local-first project management app built with Electric and PGlite. * [Open demo](https://linearlite.examples.electric-sql.com) * [Source code](https://github.com/electric-sql/electric/tree/main/examples/linearlite) [![](/img/demos/notes-demo.png)](/demos/notes) [### Notes](/demos/notes) Collaborative note-taking app with sync powered by Electric and Yjs. * [Open demo](https://notes.examples.electric-sql.com) * [Source code](https://github.com/KyleAMathews/electric-notes) The integration docs also illustrate common patterns, e.g. using Electric with frameworks like [TanStack](/docs/integrations/tanstack) and [Phoenix](/docs/integrations/phoenix) and platforms like [Supabase](/docs/integrations/supabase) and [Cloudflare](/docs/integrations/cloudflare) . Source code [​](#source-code) ------------------------------ ElectricSQL is an open source project developed at [github.com/electric-sql](https://github.com/electric-sql) . Check out the source code, issues and development in progress there. Support [​](#support) ---------------------- See the [Community page](/about/community) for information on support and events, including our [community Discord](https://discord.electric-sql.com) where you can ask questions and get support. --- # Auth - Guide | ElectricSQL Return to top ![](/img/icons/auth.svg) Auth [​](#auth) ================ How to do authentication and authorization with Electric. Including examples for [proxy](#proxy-auth) and [gatekeeper](#gatekeeper-auth)  auth. How to do auth with Electric. Including examples for [proxy](#proxy-auth) and [gatekeeper](#gatekeeper-auth)  auth. It's all HTTP [​](#it-s-all-http) ---------------------------------- The golden rule with Electric is that it's [all just HTTP](/docs/api/http) . So when it comes to auth, you can use existing primitives, such as your API, middleware and external authorization services. ### Shapes are resources [​](#shapes-are-resources) With Electric, you sync data using [Shapes](/docs/guides/shapes) and shapes are just resources. You access them by making a request to `GET /v1/shape`, with the [shape definition](/docs/guides/shapes#defining-shapes) in the query string (`?table=items`, etc.). You can authorise access to them exactly the same way you would any other web resource. ### Requests can be proxied [​](#requests-can-be-proxied) When you make a request to Electric, you can route it through an HTTP proxy or middleware stack. This allows you to authorise the request before it reaches Electric. [![Illustration of an authorzing proxy](/assets/authorizing-proxy.-X7O9Iw2.png)![Illustration of an authorzing proxy](/assets/authorizing-proxy.sm.BmP4MML0.png)](/assets/authorizing-proxy.Ko3brxY6.jpg) You can proxy the request in your cloud, or at the edge, [in-front of a CDN](#cdn-proxy) . Your auth logic can query your database, or call an external service. It's all completely up-to-you. ### Rules are optional [​](#rules-are-optional) You _don't_ have to codify your auth logic into a database rule system. There's no need to use database rules to [secure data access](/docs/guides/security) when your sync engine runs over standard HTTP. Patterns [​](#patterns) ------------------------ The two patterns we recommend and describe below, with code and examples, are: * [proxy auth](#proxy-auth) — authorising Shape requests using a proxy * [gatekeeper auth](#gatekeeper-auth) — using your API to generate shape-scoped access tokens ### Proxy auth [​](#proxy-auth) GitHub example See the [proxy-auth example](https://github.com/electric-sql/electric/tree/main/examples/proxy-auth) on GitHub for an example that implements this pattern. The simplest pattern is to authorise Shape requests using a reverse-proxy. The proxy can be your API, or a seperate proxy service or edge-function. When you make a request to sync a shape, route it via your API/proxy, validate the user credentials and shape parameters, and then only proxy the data through if authorized. For example: 1. add an `Authorization` header to your [`GET /v1/shape`](/docs/api/http#syncing-shapes) request 2. use the header to check that the client exists and has access to the shape 3. if not, return a `401` or `403` status to tell the client it doesn't have access 4. if the client does have access, proxy the request to Electric and stream the response back to the client #### Example [​](#example) When using the [Typescript client](/docs/api/clients/typescript) , you can pass in a [`headers` option](/docs/api/clients/typescript#options) to add an `Authorization` header. tsx const usersShape = (): ShapeStreamOptions => { const user = loadCurrentUser() return { url: new URL(`/api/shapes/users`, window.location.origin).href, headers: { authorization: `Bearer ${user.token}` } } } export default function ExampleComponent () { const { data: users } = useShape(usersShape()) } Then for the `/api/shapes/users` route: tsx export async function GET( request: Request, ) { const url = new URL(request.url) // Construct the upstream URL const originUrl = new URL(`http://localhost:3000/v1/shape`) // Copy over the relevant query params that the Electric client adds // so that we return the right part of the Shape log. url.searchParams.forEach((value, key) => { if ([`live`, `table`, `handle`, `offset`, `cursor`].includes(key)) { originUrl.searchParams.set(key, value) } }) // // Authentication and authorization // const user = await loadUser(request.headers.get(`authorization`)) // If the user isn't set, return 401 if (!user) { return new Response(`user not found`, { status: 401 }) } // Only query data the user has access to unless they're an admin. if (!user.roles.includes(`admin`)) { originUrl.searchParams.set(`where`, `"org_id" = ${user.org_id}`) } // When proxying long-polling requests, content-encoding & // content-length are added erroneously (saying the body is // gzipped when it's not) so we'll just remove them to avoid // content decoding errors in the browser. // // Similar-ish problem to https://github.com/wintercg/fetch/issues/23 let resp = await fetch(originUrl.toString()) if (resp.headers.get(`content-encoding`)) { const headers = new Headers(resp.headers) headers.delete(`content-encoding`) headers.delete(`content-length`) resp = new Response(resp.body, { status: resp.status, statusText: resp.statusText, headers, }) } return resp } ### Gatekeeper auth [​](#gatekeeper-auth) GitHub example See the [gatekeeper-auth example](https://github.com/electric-sql/electric/tree/main/examples/gatekeeper-auth) on GitHub for an example that implements this pattern. The Gatekeeper pattern works as follows: 1. post to a gatekeeper endpoint in your API to generate a shape-scoped auth token 2. make shape requests to Electric via an authorising proxy that validates the auth token against the request parameters The auth token should include a claim containing the shape definition. This allows the proxy to authorize the shape request by comparing the shape claim signed into the token with the [shape defined in the request parameters](/docs/quickstart#http-api) . This keeps your main auth logic: * in your API (in the gatekeeper endpoint) where it's natural to do things like query the database and call external services * running _once_ when generating a token, rather than on the "hot path" of every shape request in your authorising proxy #### Implementation [​](#implementation) The [GitHub example](https://github.com/electric-sql/electric/tree/main/examples/gatekeeper-auth) provides an [`./api`](https://github.com/electric-sql/electric/tree/main/examples/gatekeeper-auth/api) service for generating auth tokens and three options for validating those auth tokens when proxying requests to Electric: 1. [`./api`](https://github.com/electric-sql/electric/tree/main/examples/gatekeeper-auth/api) the API itself 2. [`./caddy`](https://github.com/electric-sql/electric/tree/main/examples/gatekeeper-auth/caddy) a Caddy web server as a reverse proxy 3. [`./edge`](https://github.com/electric-sql/electric/tree/main/examples/gatekeeper-auth/edge) an edge function that you can run in front of a CDN The API is an [Elixir/Phoenix](/docs/integrations/phoenix) web application that [exposes](https://github.com/electric-sql/electric/blog/main/examples/gatekeeper-auth/api/lib/api_web/router.ex) two endpoints: 1. a gatekeeper endpoint at `POST /gatekeeper/:table` 2. a proxy endpoint at `GET /proxy/v1/shape` [![Illustration of the gatekeeper request flow](/assets/gatekeeper-flow.dark.BpHfoAxB.png)](/assets/gatekeeper-flow.CsUNNIXv.jpg) ##### Gatekeeper endpoint [​](#gatekeeper-endpoint) 1. the user makes a `POST` request to `POST /gatekeeper/:table` with some authentication credentials and a shape definition in the request parameters; the gatekeeper is then responsible for authorising the user's access to the shape 2. if access is granted, the gatekeeper generates a shape-scoped auth token and returns it to the client 3. the client can then use the auth token when connecting to the Electric HTTP API, via the proxy endpoint ##### Proxy endpoint [​](#proxy-endpoint) 4. the proxy validates the JWT and verifies that the shape claim in the token matches the shape being requested; if so it sends the request on to Electric 5. Electric then handles the request as normal 6. sending a response back _through the proxy_ to the client The client can then process the data and make additional requests using the same token (step 3). If the token expires or is rejected, the client starts again (step 1). Interactive walkthrough See [How to run](https://github.com/electric-sql/electric/blob/main/examples/gatekeeper-auth/README.md#how-to-run) on GitHub for an interactive walkthrough of the three different gatekeeper-auth example proxy options. #### Example [​](#example-1) See the [./client](https://github.com/electric-sql/electric/tree/main/examples/gatekeeper-auth/client) for an example using the [Typescript client](/docs/api/clients/typescript) with gatekeeper and proxy endpoints: typescript import { FetchError, Shape, ShapeStream } from '@electric-sql/client' const API_URL = process.env.API_URL || 'http://localhost:4000' /* * Makes a request to the gatekeeper endpoint to fetch a config object * in the format expected by the ShapeStreamOptions including the * proxy `url` to connect to and auth `headers`. */ async function fetchConfig() { const url = `${API_URL}/gatekeeper/items` const resp = await fetch(url, {method: 'POST'}) return await resp.json() } // Stream the shape through the proxy, using the url and auth headers // provided by the gatekeeper. const config = await fetchConfig() const stream = new ShapeStream({ ...config, onError: async (error) => { if (error instanceof FetchError) { const status = error.status console.log('handling fetch error: ', status) // If the auth token is invalid or expires, hit the gatekeeper // again to update the auth headers and thus keep streaming // without interruption. if (status === 401 || status === 403) { return await fetchConfig() } } throw error } }) // Materialize the stream into a `Shape` and subscibe to data changes // so we can see the client working. const shape = new Shape(stream) shape.subscribe(({ rows }) => { console.log('num rows: ', rows ? rows.length : 0) }) ### Dynamic Auth Options [​](#dynamic-auth-options) The TypeScript client supports function-based options for headers and params, making it easy to handle dynamic auth tokens: typescript const stream = new ShapeStream({ url: 'http://localhost:3000/v1/shape', headers: { // Token will be refreshed on each request 'Authorization': async () => `Bearer ${await getAccessToken()}` } }) This pattern is particularly useful when: * Your auth tokens need periodic refreshing * You're using session-based authentication * You need to fetch tokens from a secure storage * You want to handle token rotation automatically The function is called when needed and its value is resolved in parallel with other dynamic options, making it efficient for real-world auth scenarios. Notes [​](#notes) ------------------ ### External services [​](#external-services) Both proxy and gatekeeper patterns work well with external auth services. If you're using an external authentication service, such as [Auth0](https://auth0.com) , to generate user credentials, for example, to generate a JWT, you just need to make sure that you can decode the JWT in your proxy or gatekeeper endpoint. If you're using an external authorization service to authorize a user's access to a shape, then you can call this whereever you run your authorization logic. For proxy auth this is the proxy. For gatekeeper auth this is the gatekeeper endpoint. Note that if you're using a distributed auth service to ensure consistent distributed auth, such as [Authzed](https://authzed.com/) , then this works best with the proxy auth pattern. This is because you explicitly _want_ to authorize the user each shape request, as opposed to the gatekeeper generating a token that can potentially become stale. ### CDN <-> Proxy [​](#cdn-proxy) If you're deploying Electric [behind a CDN](/docs/guides/deployment#caching-proxy) , then it's best to run your authorising proxy at the edge, between your CDN and your user. Both proxy and gatekeeper patterns work well for this. The gatekeeper pattern is ideal because it minimises the logic that your proxy needs to perform at the edge and minimises the network and database access that you need to provide to your edge worker. See the [edge function](https://github.com/electric-sql/electric/tree/main/examples/gatekeeper-auth/edge) proxy option in the gatekeeper example for an example designed to run at the edge on [Supabase Edge Functions](/docs/integrations/supabase) . --- # Quickstart | ElectricSQL Return to top ![Electric zap with halo](/img/home/zap-with-halo.svg) Quickstart [​](#quickstart) ============================ Let's get you up-and-running with Electric and real-time sync of your Postgres data. First we'll setup Electric and show you how to use the low-level [HTTP API](/docs/api/http) directly. Then we'll create a simple React app using our higher-level [React hooks](/docs/integrations/react#useshape) . Setup [​](#setup) ------------------ We're going to run a fresh Postgres and Electric using [Docker Compose](https://docs.docker.com/compose) . First create a new folder to work in: sh mkdir my-first-electric cd my-first-electric Then download and run this [docker-compose.yaml](https://github.com/electric-sql/electric/blob/main/website/public/docker-compose.yaml) file: sh curl -O https://electric-sql.com/docker-compose.yaml docker compose up You can now start using Electric! HTTP API [​](#http-api) ------------------------ First let's try the low-level [HTTP API](/docs/api/http) . In a new terminal, use `curl` to request a [Shape](/docs/guides/shapes) containing all rows in the `foo` table: sh curl -i 'http://localhost:3000/v1/shape?table=foo&offset=-1' A bit of explanation about the URL structure. * `/v1/shape` is a standard prefix with the API version and the shape sync endpoint path * `foo` is the name of the [`table`](/docs/guides/shapes#table) of the shape (and is required); if you wanted to sync data from the `items` table, you would change the path to `/v1/shape?table=items` * `offset=-1` means we're asking for the _entire_ Shape as we don't have any of the data cached locally yet. If we had previously fetched the shape and wanted to see if there were any updates, we'd set the offset to the last offset we'd already seen. You should get a response like this: http HTTP/1.1 400 Bad Request date: Thu, 18 Jul 2024 10:36:01 GMT content-length: 34 vary: accept-encoding cache-control: max-age=0, private, must-revalidate x-request-id: F-NISWIE1CJTnIgAAADQ access-control-allow-origin: * access-control-expose-headers: * access-control-allow-methods: GET, POST, OPTIONS content-type: application/json; charset=utf-8 {"table":["table not found"]} So it didn't work! Which makes sense... as it's an empty database without any tables or data. Let's fix that. ### Create a table and insert some data [​](#create-a-table-and-insert-some-data) Use a Postgres client to connect to Postgres. For example, with [psql](https://www.postgresql.org/docs/current/app-psql.html) you can run: sh psql "postgresql://postgres:password@localhost:54321/electric" Then create a `foo` table sql CREATE TABLE foo ( id SERIAL PRIMARY KEY, name VARCHAR(255), value FLOAT ); And insert some rows: sql INSERT INTO foo (name, value) VALUES ('Alice', 3.14), ('Bob', 2.71), ('Charlie', -1.618), ('David', 1.414), ('Eve', 0); #### Now try the curl command again [​](#now-try-the-curl-command-again) Exit your Postgres client (e.g.: with `psql` enter `\q`) and try the `curl` request again: sh curl -i 'http://localhost:3000/v1/shape?table=foo&offset=-1' Success! You should see the data you just put into Postgres in the shape response: bash HTTP/1.1 200 OK date: Thu, 18 Jul 2024 10:49:12 GMT content-length: 643 vary: accept-encoding cache-control: public, max-age=60, stale-while-revalidate=300 x-request-id: F-NJAXyulHAQP2MAAABN access-control-allow-origin: * access-control-expose-headers: * access-control-allow-methods: GET, POST, OPTIONS content-type: application/json; charset=utf-8 electric-handle: 3833821-1721299734314 electric-offset: 0_0 electric-schema: {"id":{"type":"int4","pk_index":0},"name":{"type":"varchar","max_length":255},"value":{"type":"float8"}} electric-up-to-date: etag: 3833821-1721299734314:-1:0_0 [{"offset":"0_0","value":{"id":"1","name":"Alice","value":"3.14"},"key":"\"public\".\"foo\"/1","headers":{"operation"\ :"insert"}},{"offset":"0_0","value":{"id":"2","name":"Bob","value":"2.71"},"key":"\"public\".\"foo\"/2","headers":\ {"operation":"insert"}},{"offset":"0_0","value":{"id":"3","name":"Charlie","value":"-1.618"},"key":"\"public\".\"foo\\ "/3","headers":{"operation":"insert"}},{"offset":"0_0","value":{"id":"4","name":"David","value":"1.414"},"key":"\"pub\ lic\".\"foo\"/4","headers":{"operation":"insert"}},{"offset":"0_0","value":{"id":"5","name":"Eve","value":"0.0"},"key\ ":"\"public\".\"foo\"/5","headers":{"operation":"insert"}},{"headers":{"control":"up-to-date"}}] What are those messages in the response data? When you request shape data using the HTTP API you're actually requesting entries from a log of database operations affecting the data in the shape. This is called the [Shape Log](/docs/api/http#shape-log) . The `offset` that you see in the messages and provide as the `?offset=...` query parameter in your request identifies a position in the log. The messages you see in the response are shape log entries (the ones with `value`s and `operation` headers) and control messages (the ones with `control` headers). At this point, you could continue to fetch data using HTTP requests. However, let's switch up to fetch the same shape to use in a React app instead. React app [​](#react-app) -------------------------- Run the following to create a standard React app: sh npm create --yes vite@latest react-app -- --template react-ts Change into the `react-app` subfolder and install the `@electric-sql/react` package: sh cd react-app npm install @electric-sql/react Replace the contents of `src/App.tsx` with the following. Note that we're requesting the same shape as before: tsx import { useShape } from '@electric-sql/react' function Component() { const { data } = useShape({ url: `http://localhost:3000/v1/shape`, params: { table: `foo` } }) return (
{ JSON.stringify(data, null, 2) }
) } export default Component Finally run the dev server to see it all in action! sh npm run dev Navigate to [http://localhost:5173](http://localhost:5173) in your web browser. You should see output like this: json [\ {\ "id": 1,\ "name": "Alice",\ "value": 3.14\ },\ {\ "id": 2,\ "name": "Bob",\ "value": 2.71\ },\ {\ "id": 3,\ "name": "Charlie",\ "value": -1.618\ },\ {\ "id": 4,\ "name": "David",\ "value": 1.414\ },\ {\ "id": 5,\ "name": "Eve",\ "value": 0\ }\ ] #### Postgres as a real-time database [​](#postgres-as-a-real-time-database) Note that the row with id `2` has the name `"Bob"`. Go back to your Postgres client and update the name of that row. It'll instantly be synced to your component! sql UPDATE foo SET name = 'James' WHERE id = 2; Congratulations! You've built your first real-time, reactive Electric app! --- # Shapes - Guide | ElectricSQL Return to top ![](/img/icons/shapes.svg) Shapes [​](#shapes) ==================== Shapes are the core primitive for controlling sync in the ElectricSQL system. What is a Shape? [​](#what-is-a-shape) --------------------------------------- Electric syncs little subsets of your Postgres data into local apps and services. Those subsets are defined using Shapes. ### Little subsets [​](#little-subsets) Imagine a Postgres database in the cloud with lots of data stored in it. It's often impractical or unwanted to sync all of this data over the network onto a local device. A shape is a way of defining a subset of that data that you'd like to sync into a local app. Defining shapes allows you to sync just the data you want and just the data that's practical to sync onto the local device. ![Illustration of syncing a shape](/assets/sync-shape.m7CbOrKk.svg) A client can choose to sync one shape, or lots of shapes. Many clients can sync the same shape. Multiple shapes can overlap. Defining shapes [​](#defining-shapes) -------------------------------------- Shapes are defined by: * a [table](#table) , such as `items` * an optional [where clause](#where-clause) to filter which rows are included in the shape * an optional [columns](#columns) clause to select which columns are included A shape contains all of the rows in the table that match the where clause, if provided. If a columns clause is provided, the synced rows will only contain those selected columns. Limitations Shapes are currently [single table](#single-table) . Shape definitions are [immutable](#immutable) . ### Table [​](#table) This is the root table of the shape. All shapes must specify a table and it must match a table in your Postgres database. The value can be just a tablename like `projects`, or can be a qualified tablename prefixed by the database schema using a `.` delimiter, such as `foo.projects`. If you don't provide a schema prefix, then the table is assumed to be in the `public.` schema. #### Partitioned Tables [​](#partitioned-tables) Electric supports subscribing to [declaratively partitioned tables](https://www.postgresql.org/docs/current/ddl-partitioning.html#DDL-PARTITIONING-DECLARATIVE) , both individual partitions and the root table of all partitions. Consider the following partitioned schema: sql CREATE TABLE measurement ( city_id int not null, logdate date not null, peaktemp int, unitsales int ) PARTITION BY RANGE (logdate); CREATE TABLE measurement_y2025m02 PARTITION OF measurement FOR VALUES FROM ('2025-02-01') TO ('2025-03-01'); CREATE TABLE measurement_y2025m03 PARTITION OF measurement FOR VALUES FROM ('2025-03-01') TO ('2025-04-01'); We create 2 shapes, one on the root table `measurement` and one on the `measurement_y2025m03` partition: sh curl -i 'http://localhost:3000/v1/shape?table=measurement&offset=-1' curl -i 'http://localhost:3000/v1/shape?table=measurement_y2025m03&offset=-1' The shape based on the `measurement_y2025m03` partition will only receive writes that fall within the partition range, that is with `logdate >= '2025-02-01' AND logdate < '2025-03-01'` whereas the shape based on the root `measurements` table will receive all writes to all partitions. ### Where clause [​](#where-clause) Shapes can define an optional where clause to filter out which rows from the table are included in the shape. Only rows that match the where clause will be included. The where clause must be a valid [PostgreSQL query expression](https://www.postgresql.org/docs/current/queries-table-expressions.html#QUERIES-WHERE) in SQL syntax, e.g.: * `title='Electric'` * `status IN ('backlog', 'todo')` Where clauses support: 1. columns of numerical types, `boolean`, `uuid`, `text`, `interval`, date and time types (with the exception of `timetz`), [Arrays](https://github.com/electric-sql/electric/issues/1767) (but not yet [Enums](https://github.com/electric-sql/electric/issues/1709) , except when explicitly casting them to `text`) 2. operators that work on those types: arithmetics, comparisons, logical/boolean operators like `OR`, string operators like `LIKE`, etc. You can use `AND` and `OR` to group multiple conditions, e.g.: * `title='Electric' OR title='SQL'` * `title='Electric' AND status='todo'` Where clauses are limited in that they: 1. can only refer to columns in the target row 2. can't perform joins or refer to other tables 3. can't use non-deterministic SQL functions like `count()` or `now()` See [`known_functions.ex`](https://github.com/electric-sql/electric/blob/main/packages/sync-service/lib/electric/replication/eval/env/known_functions.ex) and [`parser.ex`](https://github.com/electric-sql/electric/blob/main/packages/sync-service/lib/electric/replication/eval/parser.ex) for the source of truth on which types, operators and functions are currently supported. If you need a feature that isn't supported yet, please [raise a feature request](https://github.com/electric-sql/electric/discussions/categories/feature-requests) . Throughput Where clause evaluation impacts [data throughput](#throughput) . Some where clauses are [optimized](#optimized-where-clauses) . ### Columns [​](#columns) This is an optional list of columns to select. When specified, only the columns listed are synced. When not specified all columns are synced. For example: * `columns=id,title,status` - only include the `id`, `title` and `status` columns * `columns=id,"Status-Check"` - only include `id` and `Status-Check` columns, quoting the identifiers where necessary The specified columns must always include the primary key column(s), and should be formed as a comma separated list of column names — exactly as they are in the database schema. If the identifier was defined as case sensitive and/or with special characters, then you must quote it. Subscribing to shapes [​](#subscribing-to-shapes) -------------------------------------------------- Local clients establish shape subscriptions, typically using [client libraries](/docs/api/clients/typescript) . These sync data from the [Electric sync engine](/product/electric) into the client using the [HTTP API](/docs/api/http) . The sync service maintains shape subscriptions and streams any new data and data changes to the local client. In the client, shapes can be held as objects in memory, for example using a [`useShape`](/docs/integrations/react) hook, or in a normalised store or database like [PGlite](/product/pglite) . ### HTTP [​](#http) You can sync shapes manually using the [`GET /v1/shape`](/openapi#/paths/~1v1~1shape~1%7Btable%7D/get) endpoint. First make an initial sync request to get the current data for the Shape, such as: sh curl -i 'http://localhost:3000/v1/shape?table=foo&offset=-1' Then switch into a live mode to use long-polling to receive real-time updates: sh curl -i 'http://localhost:3000/v1/shape?table=foo&live=true&offset=...&handle=...' These requests both return an array of [Shape Log](/docs/api/http#shape-log) entries. You can process these manually, or use a higher-level client. ### Typescript [​](#typescript) You can use the [Typescript Client](/docs/api/clients/typescript) to process the Shape Log and materialised it into a `Shape` object for you. First install using: sh npm i @electric-sql/client Instantiate a `ShapeStream` and materialise into a `Shape`: ts import { ShapeStream, Shape } from '@electric-sql/client' const stream = new ShapeStream({ url: `http://localhost:3000/v1/shape`, params: { table: `foo` } }) const shape = new Shape(stream) // Returns promise that resolves with the latest shape data once it's fully loaded await shape.rows You can register a callback to be notified whenever the shape data changes: ts shape.subscribe(({ rows }) => { // rows is an array of the latest value of each row in a shape. }) Or you can use framework integrations like the [`useShape`](/docs/integrations/react) hook to automatically bind materialised shapes to your components. See the [Quickstart](/docs/quickstart) and [HTTP API](/docs/api/http) docs for more information. Throughput [​](#throughput) ---------------------------- Electric evaluates [where clauses](#where-clause) when processing changes from Postgres and matching them to [shape logs](/docs/api/http#shape-log) . If there are lots of shapes, this means we have to evaluate lots of where clauses. This has an impact on data throughput. There are two kinds of where clauses: 1. [optimized where clauses](#optimized-where-clauses) : a subset of clauses that we've optimized the evaluation of 2. non-optimized where clauses: all other where clauses With non-optimized where clauses, throughput is inversely proportional to the number of shapes. If you have 10 shapes, Electric can process 1,400 changes per second. If you have 100 shapes, throughput drops to 140 changes per second. With optimized where clauses, Electric can evaluate millions of clauses at once and maintain a consistent throughput of ~5,000 row changes per second **no matter how many shapes you have**. If you have 10 shapes, Electric can process 5,000 changes per second. If you have 1,000 shapes, throughput remains at 5,000 changes per second. For more details see the [benchmarks](/docs/reference/benchmarks#_7-write-throughput-with-optimized-where-clauses) . ### Optimized where clauses [​](#optimized-where-clauses) We currently optimize the evaluation of the following clauses: * `field = constant` - literal equality checks against a constant value. We optimize this by indexing shapes by their constant, allowing a single lookup to retrieve all shapes for that constant instead of evaluating the where clause for each shape. Note that this index is internal to Electric and unrelated to Postgres indexes. * `field = constant AND another_condition` - the `field = constant` part of the where clause is optimized as above, and any shapes that match are iterated through to check the other condition. Providing the first condition is enough to filter out most of the shapes, the write processing will be fast. If however `field = const` matches for a large number of shapes, then the write processing will be slower since each of the shapes will need to be iterated through. * `a_non_optimized_condition AND field = constant` - as above. The order of the clauses is not important (Electric will filter by optimized clauses first). Need additional where clause optimization? We plan to optimize a much larger subset of Postgres where clauses. If you need a particular clause optimized, please [raise an issue on GitHub](https://github.com/electric-sql/electric) or [let us know on Discord](https://discord.electric-sql.com) . ### Row filtering [​](#row-filtering) We use [row filtering](https://www.postgresql.org/docs/17/logical-replication-row-filter.html) where possible to reduce the amount of data sent over the replication stream. Based on the active shapes and their where clauses, we can determine which rows should be included in the replication stream to be filtered directly in Postgres. When using custom data types in where clauses, like enums or domains, row filtering at the replication level [is not available](https://www.postgresql.org/docs/17/sql-createpublication.html#:~:text=The%20row%20filter%20allows%20simple%20expressions%20that%20don%27t%20have%20user%2Ddefined%20functions%2C%20user%2Ddefined%20operators%2C%20user%2Ddefined%20types%2C%20user%2Ddefined%20collations%2C%20non%2Dimmutable%20built%2Din%20functions%2C%20or%20references%20to%20system%20columns.) , and thus all changes will be sent over the replication stream for the relevant tables. Limitations [​](#limitations) ------------------------------ ### Single table [​](#single-table) Shapes are currently single table only. In the [old version of Electric](https://legacy.electric-sql.com/docs/usage/data-access/shapes) , Shapes had an include tree that allowed you to sync nested relations. The new Electric has not yet implemented support for include trees. You can upvote and discuss adding support for include trees here: * [Shape support for include trees #1608](https://github.com/electric-sql/electric/discussions/1608) Include tree workarounds There are some practical workarounds you can already use to sync related data, based on subscribing to multiple shapes and joining in the client. For a one-level deep include tree, such as "sync this project with its issues", you can sync one shape for projects `where="id=..."` and another for issues `where="project_id=..."`. For multi-level include trees, such as "sync this project with its issues and their comments", you can denormalise the `project_id` onto the lower tables so that you can also sync comments `where="project_id=1234"`. Where necessary, you can use triggers to update these denormalised columns. ### Immutable [​](#immutable) Shape definitions are currently immutable. Once a shape subscription has been started, it's definition cannot be changed. If you want to change the data in a shape, you need to start a new subscription. You can upvote and discuss adding support for mutable shapes here: * [Editable shapes #1677](https://github.com/electric-sql/electric/discussions/1677) ### Dropping tables [​](#dropping-tables) When dropping a table from Postgres you need to _manually_ delete all shapes that are defined on that table. This is especially important if you intend to recreate the table afterwards (possibly with a different schema) as the shape will contain stale data from the old table. Therefore, recreating the table only works if you first delete the shape. Electric does not yet automatically delete shapes when tables are dropped because Postgres does not stream DDL statements (such as `DROP TABLE`) on the logical replication stream that Electric uses to detect changes. However, we are actively exploring approaches for automated shape deletion in this [GitHub issue](https://github.com/electric-sql/electric/issues/1733) . --- # Writes - Guide | ElectricSQL Return to top ![](/img/icons/writes.svg) Writes [​](#writes) ==================== How to do local writes and write-path sync with Electric. Includes patterns for [online writes](#online-writes) , [optimistic state](#optimistic-state) , [shared persistent optimistic state](#shared-persistent) and [through-the-database sync](#through-the-db) . With accompanying code in the [write-patterns example](https://github.com/electric-sql/electric/tree/main/examples/write-patterns) . Local writes with Electric [​](#local-writes-with-electric) ------------------------------------------------------------ Electric does [read-path sync](/product/electric) . It syncs data out-of Postgres, into local apps and services. Electric does not do write-path sync. It doesn't provide (or prescribe) a built-in solution for getting data back into Postgres from local apps and services. So how do you handle local writes with Electric? Well, the [design philosophy](/blog/2024/07/17/electric-next) behind Electric is to be composable and [integrate with your existing stack](/blog/2024/11/21/local-first-with-your-existing-api) . So, just as you can sync into [any client](/docs/guides/client-development) you like, you can implement writes in any way you like, using a variety of different patterns. Patterns [​](#patterns) ------------------------ This guide describes four different patterns for handling writes with Electric. It shows code examples and discusses trade-offs to consider when choosing between them. 1. [online writes](#online-writes) 2. [optimistic state](#optimistic-state) 3. [shared persistent optimistic state](#shared-persistent) 4. [through-the-database sync](#through-the-db) All of the patterns use Electric for the read-path sync (i.e.: to sync data from Postgres into the local app) and use a different approach for the write-path (i.e.: how they handle local writes and get data from the local app back into Postgres). They are introduced in order of simplicity. So the simplest and easiest to implement first and the more powerful but more complex patterns further down ‐ where you may prefer to reach for a [framework](#tools) rather than implement yourself. Write-patterns example on GitHub This guide has an accompanying [write-patterns example](https://github.com/electric-sql/electric/tree/main/examples/write-patterns) on GitHub. This implements each of the patterns described below and combines them into a single React application. You can see the example running online at [write-patterns.examples.electric-sql.com](https://write-patterns.examples.electric-sql.com) ### 1\. Online writes [​](#online-writes) ([source code](https://github.com/electric-sql/electric/tree/main/examples/write-patterns/patterns/1-online-writes) ) The first pattern is simply to use online writes. Not every app needs local, offline writes. Some apps are read-only. Some only have occasional writes or are fine requiring the user to be online in order to edit data. In this case, you can combine Electric sync with web service calls to send writes to a server. For example, the implementation in [`patterns/1-online-writes`](https://github.com/electric-sql/electric/tree/main/examples/write-patterns/patterns/1-online-writes) runs a simple Node server (in [`api.js`](https://github.com/electric-sql/electric/blob/thruflo/writes-guide/examples/write-patterns/shared/backend/api.js) ) and uses REST API calls for writes: tsx import React from 'react' import { v4 as uuidv4 } from 'uuid' import { useShape } from '@electric-sql/react' import api from '../../shared/app/client' import { ELECTRIC_URL, envParams } from '../../shared/app/config' type Todo = { id: string title: string completed: boolean created_at: Date } export default function OnlineWrites() { // Use Electric's `useShape` hook to sync data from Postgres // into a React state variable. const { isLoading, data } = useShape({ url: `${ELECTRIC_URL}/v1/shape`, params: { table: 'todos', ...envParams, }, parser: { timestamptz: (value: string) => new Date(value), }, }) const todos = data ? data.sort((a, b) => +a.created_at - +b.created_at) : [] // Handle user input events by making requests to the backend // API to create, update and delete todos. async function createTodo(event: React.FormEvent) { event.preventDefault() const form = event.target as HTMLFormElement const formData = new FormData(form) const title = formData.get('todo') as string const path = '/todos' const data = { id: uuidv4(), title: title, created_at: new Date(), } await api.request(path, 'POST', data) form.reset() } async function updateTodo(todo: Todo) { const path = `/todos/${todo.id}` const data = { completed: !todo.completed, } await api.request(path, 'PUT', data) } async function deleteTodo(event: React.MouseEvent, todo: Todo) { event.preventDefault() const path = `/todos/${todo.id}` await api.request(path, 'DELETE') } if (isLoading) { return
Loading …
} // prettier-ignore return (

1. Online writes

) } #### Benefits [​](#benefits) Online writes are very simple to implement [with your existing API](/blog/2024/11/21/local-first-with-your-existing-api) . The pattern allows you to create apps that are fast and available offline for reading data. Good use-cases include: * live dashboards, data analytics and data visualisation * AI applications that generate embeddings in the cloud * systems where writes require online integration anyway, e.g.: making payments #### Drawbacks [​](#drawbacks) You have the network on the write path. This can be slow and laggy with the user left watching loading spinners. The UI doesn't update until the server responds. Applications won't work offline. ### 2\. Optimistic state [​](#online-writes) ([source code](https://github.com/electric-sql/electric/tree/main/examples/write-patterns/patterns/2-optimistic-state) ) The second pattern extends the online pattern above with support for local offline writes with simple optimistic state. Optimistic state is state that you display "optimistically" whilst waiting for an asynchronous operation, like sending data to a server, to complete. This allows local writes to be accepted when offline and displayed immediately to the user, by merging the synced state with the optimistic state when rendering. When the writes do succeed, they are automatically synced back to the app via Electric and the local optimistic state can be discarded. The example implementation in [`patterns/2-optimistic-state`](https://github.com/electric-sql/electric/tree/main/examples/write-patterns/patterns/2-optimistic-state) uses the same REST API calls as the online example above, along with React's built in [`useOptimistic`](https://react.dev/reference/react/useOptimistic) hook to apply and discard the optimistic state. tsx import React, { useOptimistic, useTransition } from 'react' import { v4 as uuidv4 } from 'uuid' import { matchBy, matchStream } from '@electric-sql/experimental' import { useShape } from '@electric-sql/react' import api from '../../shared/app/client' import { ELECTRIC_URL, envParams } from '../../shared/app/config' type Todo = { id: string title: string completed: boolean created_at: Date } type PartialTodo = Partial & { id: string } type Write = { operation: 'insert' | 'update' | 'delete' value: PartialTodo } export default function OptimisticState() { const [isPending, startTransition] = useTransition() // Use Electric's `useShape` hook to sync data from Postgres // into a React state variable. // // Note that we also unpack the `stream` from the useShape // return value, so that we can monitor it below to detect // local writes syncing back from the server. const { isLoading, data, stream } = useShape({ url: `${ELECTRIC_URL}/v1/shape`, params: { table: 'todos', ...envParams, }, parser: { timestamptz: (value: string) => new Date(value), }, }) const sorted = data ? data.sort((a, b) => +a.created_at - +b.created_at) : [] // Use React's built in `useOptimistic` hook. This provides // a mechanism to apply local optimistic state whilst writes // are being sent-to and syncing-back-from the server. const [todos, addOptimisticState] = useOptimistic( sorted, (synced: Todo[], { operation, value }: Write) => { switch (operation) { case 'insert': return synced.some((todo) => todo.id === value.id) ? synced : [...synced, value as Todo] case 'update': return synced.map((todo) => todo.id === value.id ? { ...todo, ...value } : todo ) case 'delete': return synced.filter((todo) => todo.id !== value.id) } } ) // These are the same event handler functions from the online // example, extended with `startTransition` -> `addOptimisticState` // to apply local optimistic state. // // Note that the local state is applied: // // 1. whilst the HTTP request is being made to the API server; and // 2. until the write syncs back through the Electric shape stream // // This is slightly different from most optimistic state examples // because we wait for the sync as well as the api request. async function createTodo(event: React.FormEvent) { event.preventDefault() const form = event.target as HTMLFormElement const formData = new FormData(form) const title = formData.get('todo') as string const path = '/todos' const data = { id: uuidv4(), title: title, created_at: new Date(), completed: false, } startTransition(async () => { addOptimisticState({ operation: 'insert', value: data }) const fetchPromise = api.request(path, 'POST', data) const syncPromise = matchStream( stream, ['insert'], matchBy('id', data.id) ) await Promise.all([fetchPromise, syncPromise]) }) form.reset() } async function updateTodo(todo: Todo) { const { id, completed } = todo const path = `/todos/${id}` const data = { id, completed: !completed, } startTransition(async () => { addOptimisticState({ operation: 'update', value: data }) const fetchPromise = api.request(path, 'PUT', data) const syncPromise = matchStream(stream, ['update'], matchBy('id', id)) await Promise.all([fetchPromise, syncPromise]) }) } async function deleteTodo(event: React.MouseEvent, todo: Todo) { event.preventDefault() const { id } = todo const path = `/todos/${id}` startTransition(async () => { addOptimisticState({ operation: 'delete', value: { id } }) const fetchPromise = api.request(path, 'DELETE') const syncPromise = matchStream(stream, ['delete'], matchBy('id', id)) await Promise.all([fetchPromise, syncPromise]) }) } if (isLoading) { return
Loading …
} // The template below the heading is identical to the other patterns. // prettier-ignore return (

2. Optimistic state

) } #### Benefits [​](#benefits-1) Using optimistic state allows you to take the network off the write path and allows you to create apps that are fast and available offline for both reading and writing data. The pattern is simple to implement. You can handle writes [using your existing API](/blog/2024/11/21/local-first-with-your-existing-api) . Good use-cases include: * management apps and interactive dashboards * apps that want to feel fast and avoid loading spinners on write * mobile apps that want to be resilient to patchy connectivity #### Drawbacks [​](#drawbacks-1) This example illustrates a "simple" approach where the optimistic state: 1. is component-scoped, i.e.: is only available within the component that makes the write 2. is not persisted This means that other components may display inconsistent information and users may be confused by the optimistic state dissapearing if they unmount the component or reload the page. These limitations are addressed by the more comprehensive approach in the next pattern. ### 3\. Shared persistent optimistic state[​](#online-writes) ([source code](https://github.com/electric-sql/electric/tree/main/examples/write-patterns/patterns/3-shared-persistent) ) The third pattern extends the second pattern above by storing the optimistic state in a shared, persistent local store. This makes offline writes more resilient and avoids components getting out of sync. It's a compelling point in the design space: providing good UX and DX without introducing too much complexity or any heavy dependencies. This pattern can be implemented with a variety of client-side state management and storage mechanisms. This example in [`patterns/3-shared-persistent`](https://github.com/electric-sql/electric/tree/main/examples/write-patterns/patterns/3-shared-persistent) uses [valtio](https://valtio.dev) with localStorage for a shared, persistent, reactive store. This allows us to keep the code very similar to the simple optimistic state example above (with a valtio `useSnapshot` and plain reduce function replacing `useOptimistic`). tsx import React, { useTransition } from 'react' import { v4 as uuidv4 } from 'uuid' import { subscribe, useSnapshot } from 'valtio' import { proxyMap } from 'valtio/utils' import { type Operation, ShapeStream } from '@electric-sql/client' import { matchBy, matchStream } from '@electric-sql/experimental' import { useShape } from '@electric-sql/react' import api from '../../shared/app/client' import { ELECTRIC_URL, envParams } from '../../shared/app/config' const KEY = 'electric-sql/examples/write-patterns/shared-persistent' type Todo = { id: string title: string completed: boolean created_at: Date } type PartialTodo = Partial & { id: string } type LocalWrite = { id: string operation: Operation value: PartialTodo } // Define a shared, persistent, reactive store for local optimistic state. const optimisticState = proxyMap( JSON.parse(localStorage.getItem(KEY) || '[]') ) subscribe(optimisticState, () => { localStorage.setItem(KEY, JSON.stringify([...optimisticState])) }) /* * Add a local write to the optimistic state */ function addLocalWrite(operation: Operation, value: PartialTodo): LocalWrite { const id = uuidv4() const write: LocalWrite = { id, operation, value, } optimisticState.set(id, write) return write } /* * Subscribe to the shape `stream` until the local write syncs back through it. * At which point, delete the local write from the optimistic state. */ async function matchWrite( stream: ShapeStream, write: LocalWrite ): Promise { const { operation, value } = write const matchFn = operation === 'delete' ? matchBy('id', value.id) : matchBy('write_id', write.id) try { await matchStream(stream, [operation], matchFn) } catch (_err) { return } optimisticState.delete(write.id) } /* * Make an HTTP request to send the write to the API server. * If the request fails, delete the local write from the optimistic state. * If it succeeds, return the `txid` of the write from the response data. */ async function sendRequest( path: string, method: string, { id, value }: LocalWrite ): Promise { const data = { ...value, write_id: id, } let response: Response | undefined try { response = await api.request(path, method, data) } catch (_err) { // ignore } if (response === undefined || !response.ok) { optimisticState.delete(id) } } export default function SharedPersistent() { const [isPending, startTransition] = useTransition() // Use Electric's `useShape` hook to sync data from Postgres. const { isLoading, data, stream } = useShape({ url: `${ELECTRIC_URL}/v1/shape`, params: { table: 'todos', ...envParams, }, parser: { timestamptz: (value: string) => new Date(value), }, }) const sorted = data ? data.sort((a, b) => +a.created_at - +b.created_at) : [] // Get the local optimistic state. const localWrites = useSnapshot>(optimisticState) const computeOptimisticState = ( synced: Todo[], writes: LocalWrite[] ): Todo[] => { return writes.reduce( (synced: Todo[], { operation, value }: LocalWrite): Todo[] => { switch (operation) { case 'insert': return [...synced, value as Todo] case 'update': return synced.map((todo) => todo.id === value.id ? { ...todo, ...value } : todo ) case 'delete': return synced.filter((todo) => todo.id !== value.id) default: return synced } }, synced ) } const todos = computeOptimisticState(sorted, [...localWrites.values()]) // These are the same event handler functions from the previous optimistic // state pattern, adapted to add the state to the shared, persistent store. async function createTodo(event: React.FormEvent) { event.preventDefault() const form = event.target as HTMLFormElement const formData = new FormData(form) const title = formData.get('todo') as string const path = '/todos' const data = { id: uuidv4(), title: title, completed: false, created_at: new Date(), } startTransition(async () => { const write = addLocalWrite('insert', data) const fetchPromise = sendRequest(path, 'POST', write) const syncPromise = matchWrite(stream, write) await Promise.all([fetchPromise, syncPromise]) }) form.reset() } async function updateTodo(todo: Todo) { const { id, completed } = todo const path = `/todos/${id}` const data = { id, completed: !completed, } startTransition(async () => { const write = addLocalWrite('update', data) const fetchPromise = sendRequest(path, 'PUT', write) const syncPromise = matchWrite(stream, write) await Promise.all([fetchPromise, syncPromise]) }) } async function deleteTodo(event: React.MouseEvent, todo: Todo) { event.preventDefault() const { id } = todo const path = `/todos/${id}` const data = { id, } startTransition(async () => { const write = addLocalWrite('delete', data) const fetchPromise = sendRequest(path, 'DELETE', write) const syncPromise = matchWrite(stream, write) await Promise.all([fetchPromise, syncPromise]) }) } if (isLoading) { return
Loading …
} // The template below the heading is identical to the other patterns. // prettier-ignore return (

3. Shared persistent

) } #### Benefits [​](#benefits-2) This is a powerful and pragmatic pattern, occupying a compelling point in the design space. It's relatively simple to implement. Persisting optimistic state makes local writes more resilient. Storing optimistic state in a shared store allows all your components to see and react to it. This avoids the weaknesses with ephemoral, component-scoped optimistic state and makes this pattern more suitable for more complex, real world apps. Seperating immutable synced state from mutable local state also makes it easy to reason about and implement rollback strategies. Worst case, you can always just wipe the local state and/or re-sync the server state, without having to unpick some kind of merged mutable store. Good use-cases include: * building local-first software * interactive SaaS applications * collaboration and authoring software #### Drawbacks [​](#drawbacks-2) Combining data on-read makes local reads slightly slower. Whilst a persistent local store is used for optimistic state, writes are still made via an API. This can often be helpful and pragmatic, allowing you to [re-use your existing API](/blog/2024/11/21/local-first-with-your-existing-api) . However, you may prefer to avoid this, with a purer local-first approach based on syncing [through a local embedded database](#through-the-db) . #### Implementation notes [​](#implementation-notes) The merge logic in the `matchWrite` function differs from the previous optimistic state example in that it supports rebasing local optimistic state on concurrent updates from other users. The entrypoint for handling rollbacks has the local write context available. So it's able to rollback individual writes, rather than wiping the whole local state. Because it has the shared store available, it would also be possible to extend this to implement more sophisticated strategies. Such as also removing other local writes that causally depended-on or were related-to the rejected write. ### 4\. Through the database sync [​](#online-writes) ([source code](https://github.com/electric-sql/electric/tree/main/examples/write-patterns/patterns/4-through-the-db) ) The fourth pattern extends the concept of shared, persistent optimistic state all the way to an embedded local database. This provides a pure local-first experience, where the application code talks directly to a local database and changes sync automatically in the background. This "power" comes at the cost of increased complexity in the form of an embedded database, complex local schema and loss of context when handling rollbacks. The example in [`patterns/4-through-the-db`](https://github.com/electric-sql/electric/tree/main/examples/write-patterns/patterns/4-through-the-db) uses [PGlite](https://electric-sql.com/product/pglite) to store both synced and local optimistic state. Specifically, it: 1. syncs data into an immutable `todos_synced` table 2. persists optimistic state in a shadow `todos_local` table; and 3. combines the two on read using a `todos` view. For the write path sync it: 4. uses `INSTEAD OF` triggers to * redirect writes made to the `todos` view to the `todos_local` table * keep a log of local writes in a `changes` table 5. uses `NOTIFY` to drive a sync utility * which sends the changes to the server Through this, the implementation: * automatically manages optimistic state lifecycle * presents a single table interface for reads and writes * auto-syncs the local writes to the server The application code in [`index.tsx`](https://github.com/electric-sql/electric/blog/main/examples/write-patterns/patterns/4-through-the-db/index.tsx) stays very simple. Most of the complexity is abstracted into the local database schema, defined in [`local-schema.sql`](https://github.com/electric-sql/electric/blog/main/examples/write-patterns/patterns/4-through-the-db/local-schema.sql) . The write-path sync utility in [`sync.ts`](https://github.com/electric-sql/electric/blog/main/examples/write-patterns/patterns/4-through-the-db/local-schema.sql) handles sending data to the server. These are shown in the three tabs below: index.tsxlocal-schema.sqlsync.ts tsx import React, { useEffect, useState } from 'react' import { v4 as uuidv4 } from 'uuid' import { PGliteProvider, useLiveQuery, usePGlite, } from '@electric-sql/pglite-react' import { type PGliteWithLive } from '@electric-sql/pglite/live' import loadPGlite from './db' import ChangeLogSynchronizer from './sync' type Todo = { id: string title: string completed: boolean created_at: Date } /* * Setup the local PGlite database, with automatic change detection and syncing. * * See `./local-schema.sql` for the local database schema, including view * and trigger machinery. * * See `./sync.ts` for the write-path sync utility, which listens to changes * using pg_notify, as per https://pglite.dev/docs/api#listen */ export default function Wrapper() { const [db, setDb] = useState() useEffect(() => { let isMounted = true let writePathSync: ChangeLogSynchronizer async function init() { const pglite = await loadPGlite() if (!isMounted) { return } writePathSync = new ChangeLogSynchronizer(pglite) writePathSync.start() setDb(pglite) } init() return () => { isMounted = false if (writePathSync !== undefined) { writePathSync.stop() } } }, []) if (db === undefined) { return
Loading …
} return ( ) } function ThroughTheDB() { const db = usePGlite() const results = useLiveQuery('SELECT * FROM todos ORDER BY created_at') async function createTodo(event: React.FormEvent) { event.preventDefault() const form = event.target as HTMLFormElement const formData = new FormData(form) const title = formData.get('todo') as string await db.sql` INSERT INTO todos ( id, title, completed, created_at ) VALUES ( ${uuidv4()}, ${title}, ${false}, ${new Date()} ) ` form.reset() } async function updateTodo(todo: Todo) { const { id, completed } = todo await db.sql` UPDATE todos SET completed = ${!completed} WHERE id = ${id} ` } async function deleteTodo(event: React.MouseEvent, todo: Todo) { event.preventDefault() await db.sql` DELETE FROM todos WHERE id = ${todo.id} ` } if (results === undefined) { return
Loading …
} const todos = results.rows // The template below the heading is identical to the other patterns. // prettier-ignore return (

4. Through the DB

) } #### Benefits [​](#benefits-3) This provides full offline support, shared optimistic state and allows your components to interact purely with the local database, rather than coding over the network. Data fetching and sending is abstracted away behind Electric (for reads) and the sync utility processing the change log (for writes). Good use-cases include: * building local-first software * mobile and desktop applications * collaboration and authoring software #### Drawbacks [​](#drawbacks-3) Using a local embedded database adds quite a heavy dependency to your app. The shadow table and trigger machinery complicate your client side schema definition. Syncing changes in the background complicates any potential [rollback handling](#rollbacks) . In the [shared persistent optimistic state](#shared-persistent) pattern, you can detect a write being rejected by the server whilst in context, still handling user input. With through-the-database sync, this context is harder to reconstruct. #### Implementation notes [​](#implementation-notes-1) The [merge logic](#merge-logic) in the `delete_local_on_synced_insert_and_update_trigger` in [`./local-schema.sql`](https://github.com/electric-sql/electric/blog/main/examples/write-patterns/patterns/4-through-the-db/local-schema.sql) supports rebasing local optimistic state on concurrent updates from other users. The rollback strategy in the `rollback` method of the `ChangeLogSynchronizer` in [`./sync.ts`](https://github.com/electric-sql/electric/blog/main/examples/write-patterns/patterns/4-through-the-db/sync.ts) is very naive: clearing all local state and writes in the event of any write being rejected by the server. You may want to implement a more nuanced strategy. For example, to provide information to the user about what is happening and / or minimise data loss by only clearing local-state that's causally dependent on a rejected write. This opens the door to a lot of complexity that may best be addressed by [using an existing framework](#framework) or one of the [simpler patterns](#patterns) . Advanced [​](#advanced) ------------------------ This is an advanced section. You don't need to engage with these complexities to [get started with Electric](/docs/quickstart) . There are two key complexities introduced by handling offline writes or local writes with optimistic state: 1. merge logic when receiving synced state from the server 2. handling rollbacks when writes are rejected ### Merge logic [​](#merge-logic) When a change syncs in over the Electric replication stream, the application has to decide how to handle any overlapping optimistic state. This can be complicated by concurrency, when changes syncing in may be made by other users (or devices, or even tabs). The third and fourth examples both demonstrate approaches to rebasing the local state on the synced state, rather than just naively clearing the local state, in order to preserve local changes. [Linearlite](https://github.com/electric-sql/electric/blog/main/examples/linearlite) is another example of through-the-DB sync with more sophisticated merge logic. ### Rollbacks [​](#rollbacks) If an offline write is rejected by the server, the local application needs to find some way to revert the local state and potentially notify the user. A baseline approach can be to clear all local state if any write is rejected. More sophisticated and forgiving strategies are possible, such as: * marking local writes as rejected and displaying for manual conflict resolution * only clearing the set of writes that are causally dependent on the rejected operation One consideration is the indirection between making a write and handling a rollback. When sending write operations directly to an API, your application code can handle a rollback with the write context still available. When syncing through the database, the original write context is much harder to reconstruct. ### YAGNI [​](#yagni) [Adam Wiggins](/about/team#angels) , one of the authors of the [local-first paper](https://www.inkandswitch.com/local-first/) , developed a canvas-based thinking tool called [Muse](https://adamwiggins.com/muse-retrospective/) , explicitly designed to support concurrent, collaborative editing of an infinite canvas. Having operated at scale with a large user base, one of his main findings [reported back](https://www.youtube.com/watch?v=WEFuEY3fHd0) at the first local-first meetup in Berlin in 2023 was that in reality, conflicts are extremely rare and can be mitigated well by strategies like presence. If you're crafting a highly concurrent, collaborative experience, you may want to engage with the complexities of merge logic and rebasing local state. However, blunt strategies like those illustrated by the [patterns in this guide](#patterns) can be much easier to implement and reason about — and are perfectly serviceable for many applications. Tools [​](#tools) ------------------ Below we list some useful tools that work well for implementing writes with Electric. ### Libraries [​](#libraries) * [React `useOptimistic`](https://react.dev/reference/react/useOptimistic) * [React Router](https://reactrouter.com/start/framework/pending-ui) * [SolidJS](https://docs.solidjs.com/solid-router/reference/data-apis/action) * [Svelte Optimistic Store](https://github.com/Der-Penz/svelte-optimistic-store) * [TanStack Query](/docs/integrations/tanstack) * [Valtio](https://valtio.dev) * [Vue `vue-useoptimistic`](https://github.com/shoko31/vue-useoptimistic) ### Frameworks [​](#frameworks) * [LiveStore](/docs/integrations/livestore) * [TinyBase](https://tinybase.org) * [tRPC](https://github.com/KyleAMathews/trpc-crdt) See also the list of local-first projects on the [alternatives page](/docs/reference/alternatives#local-first) . --- # HTTP API | ElectricSQL Return to top HTTP API [​](#http-api) ======================== The HTTP API is the primary, low level API for syncing data with Electric. HTTP API specification [​](#http-api-specification) ---------------------------------------------------- API documentation is published as an [OpenAPI](https://www.openapis.org/what-is-openapi) specification: * [download the specification file](https://github.com/electric-sql/electric/blob/main/website/electric-api.yaml) to view or use with other OpenAPI [tooling](https://tools.openapis.org/) * [view the HTML documentation](/openapi) generated using [Redocly](https://redocly.com) The rest of this page will describe the features of the API. 💡 If you haven't already, you may like to walkthrough the [Quickstart](/docs/quickstart) to get a feel for using the HTTP API. Syncing shapes [​](#syncing-shapes) ------------------------------------ The API allows you to sync [Shapes](/docs/guides/shapes) of data out of Postgres using the [`GET /v1/shape`](/openapi#/paths/~1v1~1shape/get) endpoint. The pattern is as follows. First you make an initial sync request to get the current data for the Shape, such as: sh curl -i 'http://localhost:3000/v1/shape?table=foo&offset=-1' Then you switch into a live mode to use long-polling to receive real-time updates. We'll go over these steps in more detail below. First a note on the data that the endpoint returns. ### Shape Log [​](#shape-log) When you sync a shape from Electric, you get the data in the form of a log of logical database operations. This is the **Shape Log**. The `offset` that you see in the messages and provide as the `?offset=...` query parameter in your request identifies a position in the log. The messages you see in the response are shape log entries (the ones with `value`s and `action` headers) and control messages (the ones with `control` headers). The Shape Log is similar conceptually to the logical replication stream from Postgres. Except that instead of getting all the database operations, you're getting the ones that affect the data in your Shape. It's then the responsibility of the client to consume the log and materialize out the current value of the shape. [![Shape log flow diagramme](/img/api/shape-log.png)](/img/api/shape-log.jpg) Shape log flow diagramme. The values included in the shape log are strings formatted according to Postgres' display settings. The [OpenAPI specification](/openapi) defines the display settings the HTTP API adheres to. ### Initial sync request [​](#initial-sync-request) When you make an initial sync request, with `offset=-1`, you're telling the server that you want the whole log, from the start for a given shape. When a shape is first requested, Electric queries Postgres for the data and populates the log by turning the query results into insert operations. This allows you to sync shapes without having to pre-define them. Electric then streams out the log data in the response. Sometimes a log can fit in a single response. Sometimes it's too big and requires multiple requests. In this case, the first request will return a batch of data and an `electric-offset` header. An HTTP client should then continue to make requests setting the `offset` parameter to this header value. This allows the client to paginate through the shape log until it has received all of the current data. ### Control messages [​](#control-messages) The client will then receive an `up-to-date` control message at the end of the response data: json {"headers": {"control": "up-to-date"}} This indicates that the client has all the data that the server was aware of when fulfilling the request. The client can then switch into live mode to receive real-time updates. Must-refetch Note that the other control message is `must-refetch` which indicates that the client must throw away their local shape data and re-sync from scratch: json {"headers": {"control": "must-refetch"}} ### Live mode [​](#live-mode) Once a client is up-to-date, it can switch to live mode to receive real-time updates, by making requests with `live=true`, an `offset` and a shape `handle`, e.g.: sh curl -i 'http://localhost:3000/v1/shape?table=foo&live=true&handle=3833821-1721812114261&offset=0_0' The `live` parameter puts the server into live mode, where it will hold open the connection, waiting for new data arrive. This allows you to implement a long-polling strategy to consume real-time updates. The server holds open the request until either a timeout (returning `204 No content`) or when new data is available, which it sends back as the response. The client then reconnects and the server blocks again for new content. This way the client is always updated as soon as new data is available. ### Clients [​](#clients) The algorithm for consuming the HTTP API described above can be implemented from scratch for your application. Howerver, it's typically implemented by clients that can be re-used and provide a simpler interface for application code. There are a number of existing clients, such as the [TypeScript](/docs/api/clients/typescript) and [Elixir](/docs/api/clients/elixir) clients. If one doesn't exist for your language or environment, we hope that the pattern is simple enough that you should be able to [write your own client](/docs/guides/client-development) relatively easily. Caching [​](#caching) ---------------------- HTTP API responses contain cache headers, including `cache-control` with `max-age` and `stale-age` and `etag`. These work out-of-the-box with caching proxies, such as [Nginx](https://nginx.org/en) , [Caddy](https://caddyserver.com) or [Varnish](https://varnish-cache.org) , or a CDN like [Cloudflare](https://www.cloudflare.com/en-gb/application-services/products/cdn) or [Fastly](https://www.fastly.com/products/cdn) . There are three aspects to caching: 1. [accelerating initial sync](#accelerating-initial-sync) 2. [caching in the browser](#caching-in-the-browser) 3. [collapsing live requests](#collapsing-live-requests) ### Accelerating initial sync [​](#accelerating-initial-sync) When a client makes a `GET` request to fetch shape data at a given `offset`, the response can be cached. Subsequent clients requesting the same data can be served from the proxy or CDN. This removes load from Electric (and from Postrgres) and allows data to be served extremely quickly, at the edge by an optimised CDN. You can see an example Nginx config at [packages/sync-service/dev/nginx.conf](https://github.com/electric-sql/electric/blob/main/packages/sync-service/dev/nginx.conf) : nginx worker_processes 1; events { worker_connections 1024; } http { include mime.types; default_type application/octet-stream; # Enable gzip gzip on; gzip_types text/plain text/css application/javascript image/svg+xml application/json; gzip_min_length 1000; gzip_vary on; # Enable caching proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=my_cache:10m max_size=1g inactive=60m use_temp_path=off; server { listen 3002; location / { proxy_pass http://host.docker.internal:3000; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; # Enable caching proxy_cache my_cache; proxy_cache_revalidate on; proxy_cache_min_uses 1; proxy_cache_methods GET HEAD; proxy_cache_use_stale error timeout; proxy_cache_background_update on; proxy_cache_lock on; # Add proxy cache status header add_header X-Proxy-Cache $upstream_cache_status; add_header X-Cache-Date $upstream_http_date; } } } ### Caching in the browser [​](#caching-in-the-browser) Requests are also designed to be cached by the browser. This allows apps to cache and avoid re-fetching data. For example, say a page loads data by syncing a shape. [![Console showing initial request loading from the network](/assets/initial-request.BmdXXKqa.png)](/assets/initial-request.BmdXXKqa.png) [![Console showing initial request loading from the network](/assets/initial-request.sm.CFKXqfx3.png)](/assets/initial-request.BmdXXKqa.png) The next time the user navigates to the same page, the data is in the browser file cache. [![Console showing subsequent requests loading from the browser's file cache](/assets/subsequent-request.c7HveaKo.png)](/assets/subsequent-request.c7HveaKo.png) [![Console showing subsequent requests loading from the browser's file cache](/assets/subsequent-request.sm.TVSfJOK7.png)](/assets/subsequent-request.c7HveaKo.png) This can make data access instant and available offline, even without using a persistent local store. ### Collapsing live requests [​](#collapsing-live-requests) Once a client has requested the initial data for a shape, it switches into [live mode](#live-mode) , using long polling to wait for new data. When new data arrives, the client reconnects to wait for more data, and so on. Most caching proxies and CDNs support a feature called [request collapsing](https://info.varnish-software.com/blog/two-minutes-tech-tuesdays-request-coalescing) (sometimes also called request coalescing). This identifies requests to the same resource, queues them on a waiting list, and only sends a single request to the origin. Electric takes advantage of this to optimise realtime delivery to large numbers of concurrent clients. Instead of Electric holding open a connection per client, this is handled at the CDN level and allows us to coalesce concurrent long-polling requests in live mode. This is how Electric can support millions of concurrent clients with minimal load on the sync service and no load on the source Postgres. --- # Installation - Guide | ElectricSQL Return to top ![](/img/icons/install.svg) Installation [​](#installation) ================================ You need to have a [Postgres](https://www.postgresql.org) database and to run the [Electric sync service](/product/electric) in front of it. How to run Electric [​](#how-to-run-electric) ---------------------------------------------- Electric is a web application published as a Docker image at [electricsql/electric](https://hub.docker.com/r/electricsql/electric) . It connects to Postgres via a `DATABASE_URL`. Recommended [​](#recommended) ------------------------------ The simplest way to run Electric is using Docker. ### Using Docker [​](#using-docker) You can run a fresh Postgres and Electric connected together using [Docker Compose](https://docs.docker.com/compose) with this [`docker-compose.yaml`](https://github.com/electric-sql/electric/blob/main/website/public/docker-compose.yaml) : yaml version: "3.3" name: "electric_quickstart" services: postgres: image: postgres:16-alpine environment: POSTGRES_DB: electric POSTGRES_USER: postgres POSTGRES_PASSWORD: password ports: - 54321:5432 tmpfs: - /var/lib/postgresql/data - /tmp command: - -c - listen_addresses=* - -c - wal_level=logical electric: image: electricsql/electric environment: DATABASE_URL: postgresql://postgres:password@postgres:5432/electric?sslmode=disable ports: - "3000:3000" depends_on: - postgres For example you can run this using: sh curl -O https://electric-sql.com/docker-compose.yaml docker compose up Alternatively, you can run the Electric sync service on its own and connect it to an existing Postgres database, e.g.: sh docker run \ -e "DATABASE_URL=postgresql://..." \ -p 3000:3000 \ -t \ electricsql/electric:latest ### Postgres requirements [​](#postgres-requirements) You can use any Postgres (new or existing) that has [logical replication](https://www.postgresql.org/docs/current/logical-replication-config.html) enabled. You also need to connect as a database user that has the [`REPLICATION`](https://www.postgresql.org/docs/current/logical-replication-security.html) role. Advanced [​](#advanced) ------------------------ You can also choose to build and run Electric [from source](https://github.com/electric-sql/electric) as an [Elixir](https://elixir-lang.org) application. ### Build from source [​](#build-from-source) Clone the Electric repo: sh git clone https://github.com/electric-sql/electric.git cd electric Install the system dependencies with [asdf](https://asdf-vm.com) . Versions are defined in [.tool-versions](https://github.com/electric-sql/electric/blob/main/.tool-versions) : sh asdf plugin-add elixir asdf plugin-add erlang asdf plugin-add nodejs asdf plugin-add pnpm asdf install Install the [packages/sync-service](https://github.com/electric-sql/electric/tree/main/packages/sync-service) dependencies using [Mix](https://hexdocs.pm/mix/1.12/Mix.html) .: sh cd packages/sync-service mix deps.get Run the development server: sh mix run --no-halt This will try to connect to Postgres using the `DATABASE_URL` configured in [packages/sync-service/.env.dev](https://github.com/electric-sql/electric/blob/main/packages/sync-service/.env.dev) , which defaults to: shell ELECTRIC_LOG_LEVEL=debug DATABASE_URL=postgresql://postgres:password@localhost:54321/electric?sslmode=disable ELECTRIC_ENABLE_INTEGRATION_TESTING=true ELECTRIC_CACHE_MAX_AGE=1 ELECTRIC_CACHE_STALE_AGE=3 # using a small chunk size of 10kB for dev to speed up tests ELECTRIC_SHAPE_CHUNK_BYTES_THRESHOLD=10000 # configuring a second database for multi-tenancy integration testing OTHER_DATABASE_URL=postgresql://postgres:password@localhost:54322/electric?sslmode=disable ELECTRIC_PROFILE_WHERE_CLAUSES=false ELECTRIC_OTEL_SAMPLING_RATIO=1 ELECTRIC_OTEL_DEBUG=false You can edit this file to change the configuration. To run the tests, you'll need a Postgres running that matches the `:test` env config in [config/runtime.exs](https://github.com/electric-sql/electric/blob/main/packages/sync-service/config/runtime.exs) and then: sh mix test If you need any help, [ask on Discord](https://discord.electric-sql.com) . --- # Deployment - Guide | ElectricSQL Return to top ![](/img/icons/deploy.png) Deployment [​](#deployment) ============================ How to deploy the [Electric sync engine](/product/electric) , with links to integration docs for specific platforms like [Supabase](/docs/integrations/supabase) , [Neon](/docs/integrations/neon) , [Render](/docs/integrations/render) and [AWS](/docs/integrations/aws) . Electric Cloud – the simplest way to use Electric The simplest way to use Electric is via the [Electric Cloud](/product/cloud) , which is a simple, scalable, low-cost, managed Electric hosting service. [View Cloud](/product/cloud) The ingredients of a successful deployment [​](#the-ingredients-of-a-successful-deployment) -------------------------------------------------------------------------------------------- An Electric deployment has three main components. Your Postgres database, the Electric sync service and your app. Electric connects to your Postgres using a `DATABASE_URL`. Your app connects to Electric [over HTTP](/docs/api/http) , usually using a [Client library](/docs/api/clients/typescript) . [![Illustration of the main components of a successfull deployment](/assets/components.qKqz322G.png)![Illustration of the main components of a successfull deployment](/assets/components.sm.BR1Ct5je.png)](/assets/components.DNCChOI5.jpg) As a result, there are three ingredients to a successful Electric deployment: 1. you need to be [running a Postgres database](#_1-running-postgres) 2. you need to [run and connect the Electric sync service](#_2-running-electric) 3. you need your app/client to [connect to Electric over HTTP](#_3-connecting-your-app) ### Proxying requests to Electric [​](#proxying-requests-to-electric) You also often want to proxy requests to Electric through your API, or other proxy. For example, to implement [auth](./auth) and/or [caching](/docs/api/http#caching) . In these cases, you'll also need to deploy your API and/or proxy layer in front of Electric. Note also that, when running Electric behind a CDN, you may want your proxy in front of the CDN. This is where primitives like [edge functions](/docs/integrations/supabase#sync-into-edge-function) and [edge workers](/docs/integrations/cloudflare#workers) can be very useful. ### Securing data access [​](#securing-data-access) By default, Electric exposes public access to the contents of your database. You generally don't want to expose the contents of your database, so you need to [lock down access](/docs/guides/security#secure-data-access) to the Electric HTTP API. See the [Security guide](/docs/guides/security) for information. 1\. Running Postgres [​](#_1-running-postgres) ----------------------------------------------- You can use _**any standard Postgres**_, version 14 and above. This includes Postgres you host yourself, or Postgres hosted by managed database hosting providers, including: * [Supabase](/docs/integrations/supabase) * [Neon](/docs/integrations/neon) * [AWS (RDS and Aurora)](/docs/integrations/aws) * [GCP (Cloud SQL and Alloy)](/docs/integrations/gcp) * [Digital Ocean](/docs/integrations/digital-ocean) * [Crunchy](/docs/integrations/crunchy) Postgres must have [logical replication](https://www.postgresql.org/docs/current/logical-replication-config.html) enabled. You also need to connect as a database role that has the [`REPLICATION`](https://www.postgresql.org/docs/current/logical-replication-security.html) attribute. ### Data model compatibility [​](#data-model-compatibility) Electric is compatible with _**any Postgres data model**_. Electric will work as a drop on to any existing data model. There are no limitations on the database features, data types or extensions you can use. ### Connecting to Postgres [​](#connecting-to-postgres) You connect to Postgres using a [`DATABASE_URL`](/docs/api/config#database-url) env var. This connection string contains your user credentials and an `sslmode` parameter. You usually want to connect directly to Postgres and not via a connection pool. This is because Electric uses logical replication and most connection poolers don't support it. (pgBouncer does support logical replication, [as of version 1.23](https://www.pgbouncer.org/changelog#pgbouncer-123x) so this may change in future). Troubleshooting common errors If you get a TCP connection error saying `non-existing domain - :nxdomain` or `network is unreachable - :enetunreach` then you may need to connect using IPv6. You can enable this by setting [`ELECTRIC_DATABASE_USE_IPV6=true`](/docs/api/config#database-use-ipv6) . If you get a TCP connection `timeout` error then make sure you're connecting directly to Postgres and not via a connection pool. For example, when using [Supabase](/docs/integrations/supabase) you need to untick their "Use connection pooling" option on the database settings page. If you're using IPv6 with Docker, then assuming the machine you're running Electric on has IPv6 connectivity, you may also need to enable IPv6 for the Docker daemon. You can do this by [defining an IPv6-capable network](https://docs.docker.com/engine/daemon/ipv6/#create-an-ipv6-network) ) in your Compose file and then adding the `networks` key to the Electric service definition. ### Database resources [​](#database-resources) Electric creates a logical replication [publication](https://www.postgresql.org/docs/current/logical-replication-publication.html) and [replication slot](https://www.postgresql.org/docs/current/logical-replication-subscription.html#LOGICAL-REPLICATION-SUBSCRIPTION-SLOT) inside Postgres. These are called `electric_publication_default` and `electric_slot_default` by default. You can configure the name suffix using the [`ELECTRIC_REPLICATION_STREAM_ID`](/docs/api/config#replication-stream-id) env var. When running, Electric also keeps a pool of active database connections open. The size of this pool defaults to `20` and can be configured using [`ELECTRIC_DB_POOL_SIZE`](/docs/api/config#electric-db-pool-size) . Cleaning up resources If you decide to stop using Electric with a given Postgres database or switch to a different database but keep the old one around, make sure to clean up both the publication and the replication slot. See this [troubleshooting advice](./troubleshooting#wal-growth-mdash-why-is-my-postgres-database-storage-filling-up) for details. 2\. Running Electric [​](#_2-running-electric) ----------------------------------------------- The [Electric sync engine](/product/electric) is an Elixir web service, packaged using Docker. You can deploy it anywhere you can run a container with a filesystem and exposed HTTP port. This includes cloud and application hosting platforms like: * [AWS](/docs/integrations/aws) * [GCP](/docs/integrations/gcp) * [Digital Ocean](/docs/integrations/digital-ocean) * [Fly.io](/docs/integrations/fly) * [Render](/docs/integrations/render) ### Docker container [​](#docker-container) Images are deployed to Docker Hub at [electricsql/electric](https://hub.docker.com/r/electricsql/electric) . ### Optimizing for disk [​](#optimizing-for-disk) Electric caches [Shape logs](/docs/api/http#shape-log) and metadata on the filesystem. Your Electric host must provide a persistent filesystem. Ideally this should be large, fast and locally mounted, such as a NVMe SSD. If you're configuring a machine and you want to optimise it for Electric, the factors to optimise for, in order of important, are: 1. disk speed — low latency, high throughput reads and writes 2. memory 3. CPU For example, on AWS, [Storage Optimized](https://aws.amazon.com/ec2/instance-types/#Storage_Optimized) instances such as the `i3en.large`, or on Hetzner the [SX-line](https://www.hetzner.com/dedicated-rootserver/matrix-sx/) of dedicated servers would both be great choices. ### Configuring storage [​](#configuring-storage) The path to Electric's persistent storage can be configured via the [`ELECTRIC_STORAGE_DIR`](/docs/api/config#electric-storage-dir) environment variable, e.g. `ELECTRIC_STORAGE_DIR=/var/lib/electric/persistent`. Electric will create the directory at that path if it doesn't exist yet. However, you need to make sure that the OS user that Electric is running as has the necessary permissions in the parent directory. The file system location configured via `ELECTRIC_STORAGE_DIR` and the data Electric stores there must survive sync service's restarts. For example, when using Kubernetes, you'll want to create a persistent volume and attach it to your Electric deployment. Clear one, clear the other The persistent state that Electric maintains in Postgres (via the logical replication publication and replication slot) **must** stay in sync with the shape data cached on disk by Electric. If you change the value of `ELECTRIC_STORAGE_DIR` or switch to a different `DATABASE_URL` at any point, you **must** clean up the other location by hand, whether it's removing a directory tree on disk or dropping the replication slot and publication in Postgres. How much storage space? Electric trades storage for low memory use and fast sync. How much storage you need is highly application dependent. We encourage you to test with your own workload. We plan to implement [compaction](https://github.com/electric-sql/electric/issues/1582) and other features to limit and optimise storage use, such as [garbage collecting LRU shapes](https://github.com/electric-sql/electric/issues/1529) . ### HTTP port [​](#http-port) Electric provides an HTTP API exposed on a configurable [`ELECTRIC_PORT`](/docs/api/config#electric-port) . You should make sure this is exposed to the Internet. ### Caching proxy [​](#caching-proxy) Electric is designed to run behind a caching proxy, such as [Nginx](https://nginx.org/en) , [Caddy](https://caddyserver.com) , [Varnish](https://varnish-cache.org) or a CDN like [Cloudflare](https://www.cloudflare.com/en-gb/application-services/products/cdn) or [Fastly](https://www.fastly.com/products/cdn) . You don't _have_ to run a proxy in front of Electric but you will benefit from radically better performance if you do. See the [Caching section](/docs/api/http#caching) of the HTTP API docs for more information. 3\. Connecting your app [​](#_3-connecting-your-app) ----------------------------------------------------- You can then connect your app to Electric [over HTTP](/docs/api/http) . Typically you use a [Client library](/docs/api/clients/typescript) and configure the URL in the constructor, e.g.: ts const stream = new ShapeStream({ url: `https://your-electric-service.example.com/v1/shape`, params: { table: `foo` } }) const shape = new Shape(stream) You can connect to Electric from any language/environment that speaks HTTP. See the [HTTP API](/docs/api/http) and [Client docs](/docs/api/clients/typescript) for more information. --- # Elixir Client | ElectricSQL Return to top Elixir client [​](#elixir-client) ================================== Electric provides an [Elixir client](#how-to-use) that wraps the [HTTP API](/docs/api/http) into a higher-level stream interface and a [Phoenix integration](#phoenix-integration) that adds sync to your Phoenix application. How to use [​](#how-to-use) ---------------------------- The [`Electric.Client`](https://hex.pm/packages/electric_client) library allows you to stream [Shapes](/docs/guides/shapes) into your Elixir application. It's published to Hex as the [`electric_client`](https://hex.pm/packages/electric_client) package. ### Stream [​](#stream) The client exposes a [`stream/3`](https://hexdocs.pm/electric_client/Electric.Client.html#stream/3) that streams a [Shape Log](/docs/api/http#shape-log) into an [`Enumerable`](https://hexdocs.pm/elixir/Enumerable.html) : elixir Mix.install([:electric_client]) {:ok, client} = Electric.Client.new(base_url: "http://localhost:3000") stream = Electric.Client.stream(client, "my_table", where: "something = true") stream |> Stream.each(&IO.inspect/1) |> Stream.run() You can materialise the shape stream into a variety of data structures. For example by matching on insert, update and delete operations and applying them to a Map or an Ecto struct. (See the [Redis example](/demos/redis) example and Typescript [Shape class](/docs/api/clients/typescript#shape) for reference). ### Ecto queries [​](#ecto-queries) The `stream/3` function also supports deriving the shape definition from an [`Ecto.Query`](https://hexdocs.pm/ecto/Ecto.Query.html) : elixir import Ecto.Query, only: [from: 2] query = from(t in MyTable, where: t.something == true) stream = Electric.Client.stream(client, query) See the documentation at [hexdocs.pm/electric\_client](https://hexdocs.pm/electric_client) for more details. Phoenix integration [​](#phoenix-integration) ---------------------------------------------- Electric also provides an [`Electric.Phoenix`](https://hex.pm/packages/electric_phoenix) integration allows you to: * sync data into a [front-end app](/docs/integrations/phoenix#front-end-sync) from a Postgres-backed Phoenix application; and * add real-time streaming from Postgres into Phoenix LiveView via [Phoenix.Streams](/docs/integrations/phoenix#liveview-sync) See the [Phoenix framework integration page](/docs/integrations/phoenix) for more details. --- # Client development - Guide | ElectricSQL Return to top ![](/img/icons/coding.svg) Client development [​](#client-development) ============================================ How to write an Electric client for any language that speaks HTTP and JSON. HTTP and JSON [​](#http-and-json) ---------------------------------- You can create a client for Electric by: 1. implementing a long-polling strategy to [consume the HTTP API](#consume-the-http-api) 2. (optionally) [materialising the shape log](#materialise-the-shape-log) into a data structure or local store 3. (optionally) [providing reactivity bindings](#reactivity-bindings) Before you start It's worth looking through the source code for the existing [Typescript](https://github.com/electric-sql/electric/tree/main/packages/typescript-client) and [Elixir](https://github.com/electric-sql/electric/tree/main/packages/elixir-client) clients. You're also welcome to [raise an issue on GitHub](https://github.com/electric-sql/electric) and [flag up your plans on Discord](https://discord.electric-sql.com) . Your client also needs to be able to talk to [a running instance of Electric](./installation) . Consume the HTTP API [​](#consume-the-http-api) ------------------------------------------------ The [Electric sync service](/product/electric) syncs data over an [HTTP API](/docs/api/http) . The primary job of a client is to consume this API using HTTP requests. The HTTP API exposes [Shapes](/docs/guides/shapes) . There are two phases to syncing a shape: 1. [initial sync](#initial-sync) where you load all the data the server is currently aware of 2. [live mode](#live-mode) where you wait for and consume live updates in real-time ### Initial sync [​](#initial-sync) In the initial sync phase, you make a series of requests to get Shape data, increasing the `offset` parameter until you get an `up-to-date` message. #### Construct your shape URL [​](#construct-your-shape-url) Encode a [shape definition](/docs/guides/shapes#defining-shapes) into a `GET /v1/shape` URL. See the [specification for the URL structure here](/openapi#/paths/~1v1~1shape~1%7Broot_table%7D/get) . For example, a Shape that contains all of the rows in the `items` table would be requested with: http GET /v1/shape?table=items #### Make the initial `offset=-1` request [​](#make-the-initial-offset-1-request) The first request to a shape should set the `offset` parameter to `-1`. This indicates to Electric that you want to consume all of the data from the beginning of the [Shape log](/docs/api/http#shape-log) . For example, you might make a request to: http GET /v1/shape?table=items?offset=-1 The body of the response will contain a JSON array of messages. The headers of the response will contain two pieces of important information: * `electric-handle` an ephemeral identifier to an existing shape log * `electric-offset` the offset value for your next request If the last message in the response body contains an `up-to-date` control message: json {"headers":{"control":"up-to-date"}} Then the response will also contain an: * `electric-up-to-date` header Either of which indicate that you can [process the messages](#materialise-the-shape-log) and switch into [live mode](#live-mode) . Otherwise, you should continue to accumulate messages by making additional requests to the same URL, with the new shape handle and offset. For example: http GET /v1/shape?table=items&handle=38083685-1729874417404&offset=0_0 In this way, you keep making GET requests with increasing offsets until you load all the data that the server is aware of, at which point you get the `up-to-date` message. ### Live mode [​](#live-mode) In live mode, if the server doesn't have any new data, it will hold open your request until either a timeout or new data arrives. This allows you to implement long polling, where you keep the request open, and reconnect immediately when new data arrives. Why not websockets?! Consuming data over HTTP allows us to [leverage CDNs](/docs/api/http#caching) , simplifies observability and allows you to implement auth (and other capabilities) [using HTTP proxies](/docs/guides/auth#recommended-pattern) . #### Add `live` and `cursor` parameters [​](#add-live-and-cursor-parameters) Set `live=true` to switch Electric into live mode. Make sure your request timeout is higher than the server timeout (which defaults to `20s`) If the previous response contains an `electric-cursor` header, then also set the `cursor` parameter to its value. (This is an extra cache-busting parameter used to normalise [request-collapsing](/docs/api/http#collapsing-live-requests) behaviour across different CDNs). For example: http GET /v1/shape?table=items&handle=38083685-1729874417404&offset=27344208_0&cursor=1674440&live=true #### Keep polling [​](#keep-polling) Live requests will either timeout, returning `204 No content`, or will return an array of messages and headers, just as with non live responses. Keep pooling and whenever you get new data with an `up-to-date` header/message then [process the messages](#materialise-the-shape-log) . Materialise the shape log [​](#materialise-the-shape-log) ---------------------------------------------------------- How you choose to process shape log messages is up-to you. You can: * [stream the shape log messages](#streaming-messages) through * materialise the shape log [into a data structure](#into-a-data-structure) or [database](#into-a-database) ### Streaming messages [​](#streaming-messages) If you just want a stream of logical database operations, you can simply stream or broadcast these onwards. This is what both the Typescript client [`ShapeStream`](/docs/api/clients/typescript#shapestream) class and Elixir client [`stream/3`](/docs/api/clients/elixir#stream) function do. ### Into a data structure [​](#into-a-data-structure) If you want to maintain a materialised Shape in your client, you can apply the operations in the shape log to a data structure. This is what both the Typescript client [`Shape`](/docs/api/clients/typescript#shape) class and [Redis example](/demos/redis) do. Shape log messages are either control messages or logical `insert`, `update` or `delete` operations. You can materialise a Shape by applying these to your chosen data structure. For example, for a Javascript `Map`: ts switch (message.headers.operation) { case `insert`: data.set(message.key, message.value) break case `update`: data.set(message.key, { ...data.get(message.key)!, ...message.value, }) break case `delete`: data.delete(message.key) break } ### Into a database [​](#into-a-database) As well as just a single data structure, it's possible to materialise one or more shapes into a local store. This can be very simple -- just update entries in a normalised store, no matter which shape they came through -- or can be complex, when aiming to maintain database invariants in a local embedded database such as [PGlite](/product/pglite) . Syncing into a database is out of scope of this guide If you're interested in implementing it, [raise an Issue](https://github.com/electric-sql/electric) or [ask on Discord](https://discord.electric-sql.com) . ### Transactions [​](#transactions) Only apply logical operations to your materialised structure when you get an `up-to-date` message. Then either apply that batch of operations to your data structure or store atomically, for example using some kind of transactional application primitive, or only [trigger reactivity](#reactivity-bindings) once all the changes are applied. Reactivity bindings [​](#reactivity-bindings) ---------------------------------------------- If you maintain a materialised data structure, it's often useful to know when it changes. This is what the Typescript client's [`Shape.subscribe`](/docs/api/clients/typescript#shape) function enables, for example. This can then be used by a framework to trigger re-rendering. See the [`useShape` React hook source code](https://github.com/electric-sql/electric/blob/main/packages/react-hooks/src/react-hooks.tsx) for a real example but in short, e.g.: for a React component: tsx import { useEffect, useState } from 'react' const MyComponent = ({ shapeDefinition }) => { const [ data, setData ] = useState([]) useEffect(() => { const stream = new ShapeStream(shapeDefinition) const shape = new Shape(stream) shape.subscribe(setData) return () => { shape.unsubscribe() } }, [shapeDefinition]) } How you choose to provide this kind of API is very language dependent. You could support registering callbacks (like `shape.subscribe`) and then call these whenever you've finished materialising your shape, or you could some kind of broadcast mechanism. Examples [​](#examples) ------------------------ Let's walk through the process of implementing a client in a real programming language. ### Brainfuck [​](#brainfuck) shell ++++++++[>++++++++++>++++++++++++++>+++++++++++++++>++++>+++++++>+++++<<<<<<-]>-.>--.--.>+.>.<<--.+++++.----.--.+++++.-------.>>.>+++.>+. ### Python [​](#python) Let's build a simple happy-path client in Python to materialise a Shape into a `dict`. First create a new folder and make it a Python package: shell mkdir example-client cd example-client touch __init__.py Install the [Requests](https://docs.python-requests.org) HTTP client: shell # Optionally in a virtualenv: # virtualenv .venv # source .venv/bin/activate python -m pip install requests Now let's write our `Shape` client, saving the following in `client.py`: python import requests from urllib.parse import urlencode class Shape(object): """Syncs a shape log and materialises it into a `data` dict.""" def __init__(self, base_url='http://localhost:3000', offset=-1, handle=None, table=None, where=None): if table is None: raise "Must provide a table" # Request state used to build the URL. self.base_url = base_url self.cursor = None self.handle = handle self.live = False self.offset = offset self.table = table self.where = where # Materialiased data. self.data = {} # Accumulated messages (waiting for an `up-to-date` to apply). self.messages = [] # Registered callbacks to notify when the data changes. self.subscribers = [] def subscribe(self, callback): """Register a function that's called whenever the data changes.""" self.subscribers.append(callback) def sync(self): """Start syncing. Note that this blocks the current thread.""" while True: self.request() def request(self): """Make a request to `GET /v1/shape` and process the response.""" # Build the URL based on the current parameters. url = self.build_url() # Fetch the response. response = requests.get(url) # This is a happy path example, so we just log error codes. # A real client should handle errors, backoff, reconnect, etc. if response.status_code > 204: raise Exception("Error: {}".format(response.status_code)) # If the response is 200 then we may have new data to process. if response.status_code == 200: self.messages.append(response.json()) # If we're up-to-date, switch into live mode and process # the accumulated messages. if 'electric-up-to-date' in response.headers: self.live = True self.process_messages() # Set the shape handle, offset and optionally cursor for # the next request from the response headers. self.handle = response.headers['electric-handle'] self.offset = response.headers['electric-offset'] if 'electric-cursor' in response.headers: self.cursor = r.headers['electric-cursor'] def process_messages(self): """Process any batched up messages. If the data has changed, notify the subscribers. """ has_changed = False # Process the accumulated messages. for batch in self.messages: for message in batch: if 'operation' in message.get('headers', {}): op_changed = self.apply_operation(message) if op_changed: has_changed = True # Clear the queue. self.messages = [] # If the data has changed, notify the subscribers. if has_changed: self.notify_subscribers() def apply_operation(self, message): """Apply a logical operation message to the data dict. Return whether the data has changed. """ key = message['key'].replace('"', '').split("/")[-1] value = message.get('value') operation = message['headers']['operation'] if operation == 'insert': self.data[key] = value return True if operation == 'update': has_changed = False current_value = self.data[key] for k, v in value: if current_value.get(k) != v: has_changed = True current_value.update(new_value) return has_changed if operation == 'delete': if key in self.data: del self.data[key] return True return False def notify_subscribers(self): for callback in self.subscribers: callback(self.data) def build_url(self): params = { 'offset': self.offset, 'table': self.table } if self.cursor is not None: params['cursor'] = self.cursor if self.handle is not None: params['handle'] = self.handle if self.live: params['live'] = True if self.where is not None: params['where'] = self.where return "{}/v1/shape?{}".format(self.base_url, urlencode(params)) Now let's create a test file to test running the client. Save the following in `client.test.py`: python import multiprocessing import unittest from client import Shape class TestClient(unittest.TestCase): def test_shape_sync(self): parent_conn, child_conn = multiprocessing.Pipe() shape = Shape(table='items') shape.subscribe(child_conn.send) p = multiprocessing.Process(target=shape.sync) p.start() data = parent_conn.recv() self.assertEqual(type(data), dict) p.kill() if __name__ == '__main__': unittest.main() Make sure you [have Electric running](/docs/guides/installation) and then: shell $ python client.test.py . ---------------------------------------------------------------------- Ran 1 test in 0.087s OK --- # MobX - Integrations | ElectricSQL Return to top ![](/img/integrations/mobx.svg) MobX [​](#mobx) ================ [MobX](https://mobx.js.org) is a framework for simple, scalable, client-side state management. Electric and MobX [​](#electric-and-mobx) ------------------------------------------ Electric can be integrated with MobX by syncing data into a [Shape](/docs/api/clients/typescript#shape) and then [making the shape observable](https://mobx.js.org/observable-state.html) . Help wanted [Good first issue](https://github.com/electric-sql/electric/labels/good%20first%20issue) ----------------------------------------------------------------------------------------------------- We have [an open GitHub issue](https://github.com/electric-sql/electric/issues/1477) tracking this if you'd like to contribute an example or library integrating Electric and MobX. Please [leave a comment](https://github.com/electric-sql/electric/issues/1477) or [ask on Discord](https://discord.electric-sql.com) if you'd like any pointers or to discuss how best to approach this. --- # Sync service | ElectricSQL Return to top Sync service configuration [​](#sync-service-configuration) ============================================================ This page documents the config options for [self-hosting](/docs/guides/deployment) the [Electric sync engine](/product/electric) . Advanced only You don't need to worry about this if you're using [Electric Cloud](/product/cloud) . Also, the only required configuration is `DATABASE_URL`. Configuration [​](#configuration) ---------------------------------- The sync engine is an [Elixir](https://elixir-lang.org) application developed at [packages/sync-service](https://github.com/electric-sql/electric/tree/main/packages/sync-service) and published as a [Docker](https://docs.docker.com/get-started/docker-overview) image at [electricsql/electric](https://hub.docker.com/r/electricsql/electric) . Configuration options can be provided as environment variables, e.g.: shell docker run \ -e "DATABASE_URL=postgresql://..." \ -e "ELECTRIC_DB_POOL_SIZE=10" \ -p 3000:3000 \ electricsql/electric These are passed into the application via [config/runtime.exs](https://github.com/electric-sql/electric/blob/main/packages/sync-service/config/runtime.exs) . Database [​](#database) ------------------------ ### DATABASE\_URL [​](#database-url) | | | | --- | --- | | Variable | `DATABASE_URL`required | | Description | Postgres connection string. Used to connect to the Postgres database.

The connection string must be in the [libpg Connection URI format](https://www.postgresql.org/docs/current/libpq-connect.html#LIBPQ-CONNSTRING-URIS)
of `postgresql://[userspec@][hostspec][/dbname][?sslmode=]`.

The `userspec` section of the connection string specifies the database user that Electric connects to Postgres as. They must have the `REPLICATION` role.

For a secure connection, set the `sslmode` query parameter to `require`. | | Example | `DATABASE_URL=postgresql://user:password@example.com:54321/electric` | ### ELECTRIC\_DATABASE\_USE\_IPV6 [​](#electric-database-use-ipv6) | | | | --- | --- | | Variable | `ELECTRIC_DATABASE_USE_IPV6` | | Default | `false` | | Description | Set to `true` to prioritise connecting to the database over IPv6. Electric will fall back to an IPv4 DNS lookup if the IPv6 lookup fails. | | Example | `ELECTRIC_DATABASE_USE_IPV6=true` | ### ELECTRIC\_DB\_POOL\_SIZE [​](#electric-db-pool-size) | | | | --- | --- | | Variable | `ELECTRIC_DB_POOL_SIZE` | | Default | `20` | | Description | How many connections Electric opens as a pool for handling shape queries. | | Example | `ELECTRIC_DB_POOL_SIZE=10` | ### ELECTRIC\_REPLICATION\_STREAM\_ID [​](#electric-replication-stream-id) | | | | --- | --- | | Variable | `ELECTRIC_REPLICATION_STREAM_ID` | | Default | `default` | | Description | Suffix for the logical replication publication and slot name. | | Example | `ELECTRIC_REPLICATION_STREAM_ID=my-app` | Electric [​](#electric) ------------------------ ### ELECTRIC\_INSTANCE\_ID [​](#electric-instance-id) | | | | --- | --- | | Variable | `ELECTRIC_INSTANCE_ID` | | Default | `Electric.Utils.uuid4()` | | Description | A unique identifier for the Electric instance. Defaults to a randomly generated UUID. | | Example | `ELECTRIC_INSTANCE_ID=some-unique-instance-identifier` | ### ELECTRIC\_SERVICE\_NAME [​](#electric-service-name) | | | | --- | --- | | Variable | `ELECTRIC_SERVICE_NAME` | | Default | `electric` | | Description | Name of the electric service. Used as a resource identifier and namespace. | | Example | `ELECTRIC_SERVICE_NAME=my-electric-service` | ### ELECTRIC\_ENABLE\_INTEGRATION\_TESTING [​](#electric-enable-integration-testing) | | | | --- | --- | | Variable | `ELECTRIC_ENABLE_INTEGRATION_TESTING` | | Default | `false` | | Description | Expose some unsafe operations that faciliate integration testing. Do not enable this production. | | Example | `ELECTRIC_ENABLE_INTEGRATION_TESTING=true` | ### ELECTRIC\_LISTEN\_ON\_IPV6 [​](#electric-listen-on-ipv6) | | | | --- | --- | | Variable | `ELECTRIC_LISTEN_ON_IPV6` | | Default | `false` | | Description | By default, Electric binds to IPv4. Enable this to listen on IPv6 addresses as well. | | Example | `ELECTRIC_LISTEN_ON_IPV6=true` | ### ELECTRIC\_SHAPE\_CHUNK\_BYTES\_THRESHOLD [​](#electric-shape-chunk-bytes-threshold) | | | | --- | --- | | Variable | `ELECTRIC_SHAPE_CHUNK_BYTES_THRESHOLD` | | Default | `10485760` | | Description | Limit the maximum size of a shape log response, to ensure they are cached by upstream caches. Defaults to 10MB (10 \* 1024 \* 1024).

See [#1581](https://github.com/electric-sql/electric/issues/1581)
for context. | | Example | `ELECTRIC_SHAPE_CHUNK_BYTES_THRESHOLD=20971520` | ### ELECTRIC\_PORT [​](#electric-port) | | | | --- | --- | | Variable | `ELECTRIC_PORT` | | Default | `3000` | | Description | Port that the [HTTP API](/docs/api/http)
is exposed on. | | Example | `ELECTRIC_PORT=8080` | Caching [​](#caching) ---------------------- ### ELECTRIC\_CACHE\_MAX\_AGE [​](#electric-cache-max-age) | | | | --- | --- | | Variable | `ELECTRIC_CACHE_MAX_AGE` | | Default | `60` | | Description | Default `max-age` for the cache headers of the HTTP API. | | Example | `ELECTRIC_CACHE_MAX_AGE=5` | ### ELECTRIC\_CACHE\_STALE\_AGE [​](#electric-cache-stale-age) | | | | --- | --- | | Variable | `ELECTRIC_CACHE_STALE_AGE` | | Default | `300` | | Description | Default `stale-age` for the cache headers of the HTTP API. | | Example | `ELECTRIC_CACHE_STALE_AGE=5` | Storage [​](#storage) ---------------------- ### ELECTRIC\_PERSISTENT\_STATE [​](#electric-persistent-state) | | | | --- | --- | | Variable | `ELECTRIC_PERSISTENT_STATE` | | Default | `FILE` | | Description | Where to store shape metadata. Defaults to storing on the filesystem. If provided must be one of `MEMORY` or `FILE`. | | Example | `ELECTRIC_PERSISTENT_STATE=MEMORY` | ### ELECTRIC\_STORAGE [​](#electric-storage) | | | | --- | --- | | Variable | `ELECTRIC_STORAGE` | | Default | `FILE` | | Description | Where to store shape logs. Defaults to storing on the filesystem. If provided must be one of `MEMORY` or `FILE`. | | Example | `ELECTRIC_STORAGE=MEMORY` | ### ELECTRIC\_STORAGE\_DIR [​](#electric-storage-dir) | | | | --- | --- | | Variable | `ELECTRIC_STORAGE_DIR` | | Default | `./persistent` | | Description | Path to root folder for storing data on the filesystem. | | Example | `ELECTRIC_STORAGE_DIR=/var/example` | Telemetry [​](#telemetry) -------------------------- These environment variables allow configuration of metric and trace export for visibility into performance of the Electric instance. ### ELECTRIC\_OTLP\_ENDPOINT [​](#electric-otlp-endpoint) | | | | --- | --- | | Variable | `ELECTRIC_OTLP_ENDPOINT`optional | | Description | Set an [OpenTelemetry](https://opentelemetry.io/docs/what-is-opentelemetry/)
endpoint URL to enable telemetry. | | Example | `ELECTRIC_OTLP_ENDPOINT=https://example.com` | ### ELECTRIC\_OTEL\_DEBUG [​](#electric-otel-debug) | | | | --- | --- | | Variable | `ELECTRIC_OTEL_DEBUG` | | Default | `false` | | Description | Debug tracing by printing spans to stdout, without batching. | | Example | `ELECTRIC_OTEL_DEBUG=true` | ### ELECTRIC\_HNY\_API\_KEY [​](#electric-hny-api-key) | | | | --- | --- | | Variable | `ELECTRIC_HNY_API_KEY`optional | | Description | [Honeycomb.io](https://www.honeycomb.io)
api key. Specify along with `HNY_DATASET` to export traces directly to Honeycomb, without the need to run an OpenTelemetry Collector. | | Example | `ELECTRIC_HNY_API_KEY=your-api-key` | ### ELECTRIC\_HNY\_DATASET [​](#electric-hny-dataset) | | | | --- | --- | | Variable | `ELECTRIC_HNY_DATASET`optional | | Description | Name of your Honeycomb Dataset. | | Example | `ELECTRIC_HNY_DATASET=your-dataset-name` | ### ELECTRIC\_PROMETHEUS\_PORT [​](#electric-prometheus-port) | | | | --- | --- | | Variable | `ELECTRIC_PROMETHEUS_PORT`optional | | Description | Expose a prometheus reporter for telemetry data on the specified port. | | Example | `ELECTRIC_PROMETHEUS_PORT=9090` | ### ELECTRIC\_STATSD\_HOST [​](#electric-statsd-host) | | | | --- | --- | | Variable | `ELECTRIC_STATSD_HOST`optional | | Description | Enable sending telemetry data to a StatsD reporting endpoint. | | Example | `ELECTRIC_STATSD_HOST=https://example.com` | Logging [​](#logging) ---------------------- ### ELECTRIC\_LOG\_LEVEL [​](#electric-log-level) | | | | --- | --- | | Variable | `ELECTRIC_LOG_LEVEL`optional | | Description | Verbosity of Electric's log output.

Available levels, in the order of increasing verbosity:

* `error`
* `warning`
* `info`
* `debug` | | Example | `ELECTRIC_LOG_LEVEL=debug` | ### ELECTRIC\_LOG\_COLORS [​](#electric-log-colors) | | | | --- | --- | | Variable | `ELECTRIC_LOG_COLORS`optional | | Description | Enable or disable ANSI coloring of Electric's log output.

By default, coloring is enabled when Electric's stdout is connected to a terminal. This may be undesirable in certain runtime environments, such as AWS which displays ANSI color codes using escape sequences and may incorrectly split log entries into multiple lines. | | Example | `ELECTRIC_LOG_COLORS=false` | ### ELECTRIC\_LOG\_OTP\_REPORTS [​](#electric-log-otp-reports) | | | | --- | --- | | Variable | `ELECTRIC_LOG_OTP_REPORTS` | | Default | `false` | | Description | Enable [OTP SASL](https://www.erlang.org/doc/apps/sasl/sasl_app.html)
reporting at runtime. | | Example | `ELECTRIC_LOG_OTP_REPORTS=true` | Usage reporting [​](#usage-reporting) -------------------------------------- ### ELECTRIC\_USAGE\_REPORTING [​](#electric-usage-reporting) These environment variables allow configuration of anonymous usage data reporting back to [https://electric-sql.com](https://electric-sql.com) | | | | --- | --- | | Variable | `ELECTRIC_USAGE_REPORTING` | | Default | `true` | | Description | Configure anonymous usage data about the instance being sent to a central checkpoint service. Collected information is anonymised and doesn't contain any information from the replicated data. You can read more about it in our [telemetry docs](./../reference/telemetry#anonymous-usage-data)
. | | Example | `ELECTRIC_USAGE_REPORTING=true` | --- # Typescript Client | ElectricSQL Return to top TypeScript client [​](#typescript-client) ========================================== The TypeScript client is a higher-level client interface that wraps the [HTTP API](/docs/api/http) to make it easy to sync [Shapes](/docs/guides/shapes) in the web browser and other JavaScript environments. Defined in [packages/typescript-client](https://github.com/electric-sql/electric/tree/main/packages/typescript-client) , it provides a [ShapeStream](#shapestream) primitive to subscribe to a change stream and a [Shape](#shape) primitive to get the whole shape whenever it changes. Install [​](#install) ---------------------- The client is published on NPM as [`@electric-sql/client`](https://www.npmjs.com/package/@electric-sql/client) : sh npm i @electric-sql/client How to use [​](#how-to-use) ---------------------------- The client exports: * a [`ShapeStream`](#shapestream) class for consuming a [Shape Log](./../http#shape-log) ; and * a [`Shape`](#shape) class for materialising the log stream into a shape object These compose together, e.g.: ts import { ShapeStream, Shape } from '@electric-sql/client' const stream = new ShapeStream({ url: `http://localhost:3000/v1/shape`, params: { table: 'items' } }) const shape = new Shape(stream) // The callback runs every time the Shape data changes. shape.subscribe(data => console.log(data)) ### ShapeStream [​](#shapestream) The [`ShapeStream`](https://github.com/electric-sql/electric/blob/main/packages/typescript-client/src/client.ts#L163) is a low-level primitive for consuming a [Shape Log](./../http#shape-log) . Construct with a shape definition and options and then either subscribe to the shape log messages directly or pass into a [`Shape`](#shape) to materialise the stream into an object. tsx import { ShapeStream } from '@electric-sql/client' // Passes subscribers rows as they're inserted, updated, or deleted const stream = new ShapeStream({ url: `http://localhost:3000/v1/shape`, params: { table: `foo` } }) stream.subscribe(messages => { // messages is an array with one or more row updates // and the stream will wait for all subscribers to process them // before proceeding }) #### Options [​](#options) The `ShapeStream` constructor takes [the following options](https://github.com/electric-sql/electric/blob/main/packages/typescript-client/src/client.ts#L39) : ts /** * Options for constructing a ShapeStream. */ export interface ShapeStreamOptions { /** * The full URL to where the Shape is hosted. This can either be the Electric * server directly or a proxy. E.g. for a local Electric instance, you might * set `http://localhost:3000/v1/shape` */ url: string /** * PostgreSQL-specific parameters for the shape. * This includes table, where clause, columns, and replica settings. */ params: { /** * The root table for the shape. */ table: string /** * The where clauses for the shape. */ where?: string /** * The columns to include in the shape. * Must include primary keys, and can only include valid columns. */ columns?: string[] /** * If `replica` is `default` (the default) then Electric will only send the * changed columns in an update. * * If it's `full` Electric will send the entire row with both changed and * unchanged values. * * Setting `replica` to `full` will obviously result in higher bandwidth * usage and so is not recommended. */ replica?: Replica /** * Additional request parameters to attach to the URL. * These will be merged with Electric's standard parameters. */ [key: string]: string | string[] | undefined } /** * The "offset" on the shape log. This is typically not set as the ShapeStream * will handle this automatically. A common scenario where you might pass an offset * is if you're maintaining a local cache of the log. If you've gone offline * and are re-starting a ShapeStream to catch-up to the latest state of the Shape, * you'd pass in the last offset and shapeId you'd seen from the Electric server * so it knows at what point in the shape to catch you up from. */ offset?: Offset /** * Similar to `offset`, this isn't typically used unless you're maintaining * a cache of the shape log. */ shapeId?: string /** * HTTP headers to attach to requests made by the client. * Can be used for adding authentication headers. */ headers?: Record /** * Automatically fetch updates to the Shape. If you just want to sync the current * shape and stop, pass false. */ subscribe?: boolean /** * Signal to abort the stream. */ signal?: AbortSignal /** * Custom fetch client implementation. */ fetchClient?: typeof fetch /** * Custom parser for handling specific Postgres data types. */ parser?: Parser /** * A function for handling errors. * This is optional, when it is not provided any shapestream errors will be thrown. * If the function returns an object containing parameters and/or headers * the shapestream will apply those changes and try syncing again. * If the function returns void the shapestream is stopped. */ onError?: ShapeStreamErrorHandler backoffOptions?: BackoffOptions } type RetryOpts = { params?: ParamsRecord headers?: Record } type ShapeStreamErrorHandler = ( error: Error ) => void | RetryOpts | Promise Note that certain parameter names are reserved for Electric's internal use and cannot be used in custom params: * `offset` * `handle` * `live` * `cursor` * `source_id` The following PostgreSQL-specific parameters should be included within the `params` object: * `table` - The root table for the shape * `where` - SQL where clause for filtering rows * `columns` - List of columns to include * `replica` - Controls whether to send full or partial row updates Example with PostgreSQL-specific parameters: typescript const stream = new ShapeStream({ url: 'http://localhost:3000/v1/shape', params: { table: 'users', where: 'age > 18', columns: ['id', 'name', 'email'], replica: 'full' } }) You can also include additional custom parameters in the `params` object alongside the PostgreSQL-specific ones: typescript const stream = new ShapeStream({ url: 'http://localhost:3000/v1/shape', params: { table: 'users', customParam: 'value' } }) #### Dynamic Options [​](#dynamic-options) Both `params` and `headers` support function options that are resolved when needed. These functions can be synchronous or asynchronous: typescript const stream = new ShapeStream({ url: 'http://localhost:3000/v1/shape', params: { table: 'items', userId: () => getCurrentUserId(), filter: async () => await getUserPreferences() }, headers: { 'Authorization': async () => `Bearer ${await getAccessToken()}`, 'X-Tenant-Id': () => getCurrentTenant() } }) Function options are resolved in parallel, making this pattern efficient for multiple async operations like fetching auth tokens and user context. Common use cases include: * Authentication tokens that need to be refreshed * User-specific parameters that may change * Dynamic filtering based on current state * Multi-tenant applications where context determines the request #### Messages [​](#messages) A `ShapeStream` consumes and emits a stream of messages. These messages can either be a `ChangeMessage` representing a change to the shape data: ts export type ChangeMessage = Row> = { key: string value: T headers: Header & { operation: `insert` | `update` | `delete` } offset: Offset } Or a `ControlMessage`, representing an instruction to the client, as [documented here](./../http#control-messages) . #### Parsing and Custom Parsing [​](#parsing-and-custom-parsing) To understand the type of each column in your shape, you can check the `electric-schema` response header in the shape response. This header contains the PostgreSQL type information for each column. By default, when constructing a `ChangeMessage.value`, `ShapeStream` parses the following Postgres types into native JavaScript values: * `int2`, `int4`, `float4`, and `float8` are parsed into JavaScript `Number` * `int8` is parsed into a JavaScript `BigInt` * `bool` is parsed into a JavaScript `Boolean` * `json` and `jsonb` are parsed into JavaScript values/arrays/objects using `JSON.parse` * Postgres Arrays are parsed into JavaScript arrays, e.g. `"{{1,2},{3,4}}"` is parsed into `[[1,2],[3,4]]` All other types aren't parsed and are left in the string format as they were served by the HTTP endpoint. You can extend the default parsing behavior by defining custom parsers for specific PostgreSQL data types. This is particularly useful when you want to transform string representations of dates, JSON, or other complex types into their corresponding JavaScript objects. Here's an example: ts // Define row type type CustomRow = { id: number title: string created_at: Date // We want this to be a Date object } const stream = new ShapeStream({ url: 'http://localhost:3000/v1/shape', params: { table: 'posts' }, parser: { // Parse timestamp columns into JavaScript Date objects timestamptz: (date: string) => new Date(date) } }) const shape = new Shape(stream) shape.subscribe(data => { console.log(data.created_at instanceof Date) // true }) #### Replica full [​](#replica-full) By default Electric sends the modified columns in an update message, not the complete row. To be specific: * an `insert` operation contains the full row * an `update` operation contains the primary key column(s) and the changed columns * a `delete` operation contains just the primary key column(s) If you'd like to recieve the full row value for updates and deletes, you can set the `replica` option of your `ShapeStream` to `full`: tsx import { ShapeStream } from "@electric-sql/client" const stream = new ShapeStream({ url: `http://localhost:3000/v1/shape`, params: { table: `foo`, replica: `full` } }) This is less efficient and will use more bandwidth for the same shape (especially for tables with large static column values). Note also that shapes with different `replica` settings are distinct, even for the same table and where clause combination. #### Authentication with Dynamic Tokens [​](#authentication-with-dynamic-tokens) When working with authentication tokens that need to be refreshed, the recommended approach is to use a function-based header: ts const stream = new ShapeStream({ url: 'http://localhost:3000/v1/shape', params: { table: 'items' }, headers: { 'Authorization': async () => `Bearer ${await getToken()}` }, onError: async (error) => { if (error instanceof FetchError && error.status === 401) { // Force token refresh await refreshToken() // Return empty object to trigger a retry with the new token // that will be fetched by our function-based header return {} } // Rethrow errors we can't handle throw error } }) This approach automatically handles token refresh as the function is called each time a request is made. You can also combine this with an error handler for more complex scenarios. ### Shape [​](#shape) The [`Shape`](https://github.com/electric-sql/electric/blob/main/packages/typescript-client/src/shape.ts) is the main primitive for working with synced data. It takes a [`ShapeStream`](#shapestream) , consumes the stream, materialises it into a Shape object and notifies you when this changes. tsx import { ShapeStream, Shape } from '@electric-sql/client' const stream = new ShapeStream({ url: `http://localhost:3000/v1/shape`, params: { table: `foo` } }) const shape = new Shape(stream) // Returns promise that resolves with the latest shape data once it's fully loaded await shape.rows // passes subscribers shape data when the shape updates shape.subscribe(({ rows }) => { // rows is an array of the latest value of each row in a shape. }) ### Subscribing to updates [​](#subscribing-to-updates) The `subscribe` method allows you to receive updates whenever the shape changes. It takes two arguments: 1. A message handler callback (required) 2. An error handler callback (optional) typescript const stream = new ShapeStream({ url: 'http://localhost:3000/v1/shape', params: { table: 'issues' } }) // Subscribe to both message and error handlers stream.subscribe( (messages) => { // Process messages console.log('Received messages:', messages) }, (error) => { // Get notified about errors console.error('Error in subscription:', error) } ) You can have multiple active subscriptions to the same stream. Each subscription will receive the same messages, and the stream will wait for all subscribers to process their messages before proceeding. To stop receiving updates, you can either: * Unsubscribe a specific subscription using the function returned by `subscribe` * Unsubscribe all subscriptions using `unsubscribeAll()` typescript // Store the unsubscribe function const unsubscribe = stream.subscribe(messages => { console.log('Received messages:', messages) }) // Later, unsubscribe this specific subscription unsubscribe() // Or unsubscribe all subscriptions stream.unsubscribeAll() ### Error Handling [​](#error-handling) The ShapeStream provides two ways to handle errors: 1. Using the `onError` handler (recommended): typescript const stream = new ShapeStream({ url: 'http://localhost:3000/v1/shape', params: { table: 'issues' }, onError: (error) => { // Handle all stream errors here if (error instanceof FetchError) { console.error('HTTP error:', error.status, error.message) } else { console.error('Stream error:', error) } } }) If no `onError` handler is provided, the ShapeStream will throw errors that occur during streaming. 2. Individual subscribers can optionally handle errors specific to their subscription: typescript stream.subscribe( (messages) => { // Process messages }, (error) => { // Handle errors for this specific subscription console.error('Subscription error:', error) } ) #### Error Types [​](#error-types) The following error types may be encountered: **Initialization Errors** (thrown by constructor): * `MissingShapeUrlError`: Missing required URL parameter * `InvalidSignalError`: Invalid AbortSignal instance * `ReservedParamError`: Using reserved parameter names **Runtime Errors** (handled by `onError` or thrown): * `FetchError`: HTTP errors during shape fetching * `FetchBackoffAbortError`: Fetch aborted using AbortSignal * `MissingShapeHandleError`: Missing required shape handle * `ParserNullValueError`: Parser encountered NULL value in a column that doesn't allow NULL values See the [Demos](/demos) and [integrations](/docs/integrations/react) for more usage examples. --- # Troubleshooting - Guide | ElectricSQL Return to top ![](/img/icons/troubleshoot.svg) Troubleshooting [​](#troubleshooting) ====================================== Tips and answers to FAQs about how to run Electric successfully. Local development [​](#local-development) ------------------------------------------ ### Slow shapes — why are my shapes slow in the browser in local development? [​](#slow-shapes-mdash-why-are-my-shapes-slow-in-the-browser-in-local-development) Sometimes people encounter a mysterious slow-down with Electric in local development, when your web app is subscribed to 6 or more shapes. This slow-down is caused by a limitation of the legacy version of HTTP, 1.1. With HTTP/1.1, browsers only allow 6 simultaneous requests to a specific backend. This is because each HTTP/1.1 request uses its own expensive TCP connection. As shapes are loaded over HTTP, this means only 6 shapes can be getting updates with HTTP/1.1 due to this browser restriction. All other requests pause until there's an opening. Luckily, HTTP/2, introduced in 2015, fixes this problem by _multiplexing_ each request to a server over the same TCP connection. This allows essentially unlimited connections. HTTP/2 is standard across the vast majority of hosts now. Unfortunately it's not yet standard in local dev environments. ##### Solution — run Caddy [​](#solution-mdash-run-caddy) To fix this, you can setup a local reverse-proxy using the popular [Caddy server](https://caddyserver.com) . Caddy automatically sets up HTTP/2 and proxies requests to Electric, getting around the 6 requests limitation with HTTP/1.1 in the browser. 1. Install Caddy for your OS — [https://caddyserver.com/docs/install](https://caddyserver.com/docs/install) 2. Run `caddy trust` so Caddy can install its certificate into your OS. This is necessary for http/2 to Just Work™ without SSL warnings/errors in your browser — [https://caddyserver.com/docs/command-line#caddy-trust](https://caddyserver.com/docs/command-line#caddy-trust) Note — it's really important you run Caddy directly from your computer and not in e.g. a Docker container as otherwise, Caddy won't be able to use http/2 and will fallback to http/1 defeating the purpose of using it! Once you have Caddy installed and have added its certs — you can run this command to start Caddy listening on port 3001 and proxying shape requests to Electric on port 3000. If you're loading shapes through your API or framework dev server, replace `3000` with the port that your API or dev server is listening on. The browser should talk directly to Caddy. sh caddy run \ --config - \ --adapter caddyfile \ < { const { id, title } = item const enc = new TextEncoder() const encoded = enc.encode(title) const iv = crypto.getRandomValues(new Uint8Array(12)) const encrypted = await crypto.subtle.encrypt( { iv, name: 'AES-GCM', }, key, encoded ) const ciphertext = base64.fromByteArray(new Uint8Array(encrypted)) const iv_str = base64.fromByteArray(iv) return { id, ciphertext, iv: iv_str, } } /* * Decrypt an `EncryptedItem` to an `Item`. */ async function decrypt(item: EncryptedItem): Promise { const { id, ciphertext, iv: iv_str } = item const encrypted = base64.toByteArray(ciphertext) const iv = base64.toByteArray(iv_str) const decrypted = await crypto.subtle.decrypt( { iv, name: 'AES-GCM', }, key, encrypted ) const dec = new TextDecoder() const title = dec.decode(decrypted) return { id, title, } } export const Example = () => { const [items, setItems] = useState() const { data } = useShape({ url: `${ELECTRIC_URL}/v1/shape`, params: { table: 'items', }, }) const rows = data !== undefined ? data : [] // There are more efficient ways of updating state than always decrypting // all the items on any change but just to demonstate the decryption ... useEffect(() => { async function init() { const items = await Promise.all( rows.map(async (row) => await decrypt(row)) ) setItems(items) } init() }, [rows]) /* * Handle adding an item by creating the item data, encrypting it * and sending it to the API */ async function createItem(event: React.FormEvent) { event.preventDefault() const form = event.target as HTMLFormElement const formData = new FormData(form) const title = formData.get('title') as string const id = crypto.randomUUID() const item = { id, title, } const data = await encrypt(item) const url = `${API_URL}/items` const options = { method: 'POST', body: JSON.stringify(data), headers: { 'Content-Type': 'application/json', }, } await fetch(url, options) form.reset() } if (items === undefined) { return
Loading...
} return (
{items.map((item: Item, index: number) => (

{item.title}

))}
) } ### Key management [​](#key-management) One of the primary challenges with encryption is key management. I.e.: choosing which data to encrypt with which keys and sharing the right keys with the right users. Electric doesn't provide or prescribe any specific key management solution. You're free to use any existing key management system, such as Hashicorp Vault, for key management. However, for end-to-end encryption of shared data, you will at some point need to share keys between clients. This is a job that Electric is good at: syncing the right data to the right users. For example, imagine you store keys in a seperate, extra secure, Postgres database and you segment your encryption by tenant (or group, or some other shared resource). You could sync keys to the client using a shape like this: ts import { ShapeStream } from '@electric-sql/client' const stream = new ShapeStream({ url: `${ELECTRIC_URL}/v1/shape`, params: { table: 'tenants', columns: [\ 'keys'\ ], where: `id in ('${user.tenant_ids.join(`', '`)}')` } }) You could then put a denormalised `tenant_id` column on all of the synced tables in your main database and lookup the correct key to use when decrypting and encrypting the row in the client. --- # LiveStore - Integrations | ElectricSQL Return to top ![](/img/integrations/livestore.svg) LiveStore [​](#livestore) ========================== [LiveStore](https://github.com/livestorejs) is a reactive SQLite-based state management library focused on high-performance client-side reactivity, originally based on [Riffle](https://riffle.systems) . Electric and LiveStore [​](#electric-and-livestore) ---------------------------------------------------- LiveStore is under active development by [Johannes Schickling](https://www.schickling.dev) . ElectricSQL are sponsoring LiveStore development and building a sync integration. Some early docs are published at [livestore.dev](https://livestore.dev/getting-started/react-web) . You can also see the [Expo](./expo) launch day [blog post](https://expo.dev/blog/local-first-application-development-with-livestore) and video: Early access [​](#early-access) -------------------------------- LiveStore is now open access for GitHub Sponsors. You can get access by [sponsoring Johannes here](https://github.com/sponsors/schickling) for $20 per month. --- # TanStack - Integrations | ElectricSQL Return to top ![](/img/integrations/tanstack.svg) TanStack [​](#tanstack) ======================== [TanStack](https://tanstack.com/) is a set of utilities for building web applications. [TanStack Query](https://tanstack.com/query/latest) is a data-fetching and state management library. Electric and TanStack [​](#electric-and-tanstack) -------------------------------------------------- Electric works very well together with TanStack Query, where Electric provides the read-path sync and TanStack provides a [local write-path with optimistic state](https://tanstack.com/query/latest/docs/framework/react/guides/optimistic-updates#via-the-cache) . [![Illustration of an Electric - TanStack integration](/assets/data-flow.CYO8XsN2.png)![Illustration of an Electric - TanStack integration](/assets/data-flow.sm.DRVsMAs5.png)](/assets/data-flow.Bfy4bsQw.jpg) _Green shows read-path sync via Electric. Red shows write-path via TanStack._ In this configuration, Electric and TanStack can provide a fully offline-capable system with active-active replication of both reads and writes. ### Example [​](#example) The example below shows a simple todo application that uses Electric for read-path sync and TanStack for local optimistic writes. Electric is used to sync a shape. TanStack is used to apply mutations and maintain optimistic state. When a mutation is confirmed, it cleares the optimistic state. When the component renders, it merges the optimistic state into the shape data. tsx import { getShapeStream, useShape } from "@electric-sql/react" import { useMutation, useMutationState, useQueryClient, } from "@tanstack/react-query" import { matchStream } from "./match-stream" import { v4 as uuidv4 } from "uuid" import "./Example.css" type Item = { id: string } const baseUrl = import.meta.env.ELECTRIC_URL ?? `http://localhost:3000` const baseApiUrl = `http://localhost:3001` const itemsUrl = new URL(`/items`, baseApiUrl) const itemShape = () => ({ url: new URL(`/v1/shape`, baseUrl).href, params: { table: `items`, }, }) async function createItem(newId: string) { const itemsStream = getShapeStream(itemShape()) // Match the insert const findUpdatePromise = matchStream({ stream: itemsStream, operations: [`insert`], matchFn: ({ message }) => message.value.id === newId, }) // Insert item const fetchPromise = fetch(itemsUrl, { method: `POST`, body: JSON.stringify({ id: newId }), }) return await Promise.all([findUpdatePromise, fetchPromise]) } async function clearItems(numItems: number) { const itemsStream = getShapeStream(itemShape()) // Match the delete const findUpdatePromise = numItems > 0 ? matchStream({ stream: itemsStream, operations: [`delete`], // First delete will match matchFn: () => true, }) : Promise.resolve() // Delete all items const fetchPromise = fetch(itemsUrl, { method: `DELETE` }) return await Promise.all([findUpdatePromise, fetchPromise]) } export const Example = () => { const queryClient = useQueryClient() const { data: items } = useShape(itemShape()) const submissions: Item[] = useMutationState({ filters: { status: `pending` }, select: (mutation) => mutation.state.context as Item, }).filter((item) => item !== undefined) const { mutateAsync: addItemMut } = useMutation({ scope: { id: `items` }, mutationKey: [`add-item`], mutationFn: (newId: string) => createItem(newId), onMutate: (id) => { const optimisticItem: Item = { id } return optimisticItem }, }) const { mutateAsync: clearItemsMut, isPending: isClearing } = useMutation({ scope: { id: `items` }, mutationKey: [`clear-items`], mutationFn: (numItems: number) => clearItems(numItems), onMutate: () => { const addMutations = queryClient .getMutationCache() .findAll({ mutationKey: [`add-item`] })! addMutations?.forEach((mut) => queryClient.getMutationCache().remove(mut)) }, }) // Merge data from shape & optimistic data from fetchers. This removes // possible duplicates as there's a potential race condition where // useShape updates from the stream slightly before the action has finished. const itemsMap = new Map() if (!isClearing) { items.concat(submissions).forEach((item) => { itemsMap.set(item.id, { ...itemsMap.get(item.id), ...item }) }) } else { submissions.forEach((item) => itemsMap.set(item.id, item)) } return (
{[...itemsMap.values()].map((item: Item, index: number) => (

{item.id}

))}
) } See the [Tanstack example](/demos/tanstack) for the full source code. Help wanted [Good first issue](https://github.com/electric-sql/electric/labels/good%20first%20issue) ----------------------------------------------------------------------------------------------------- We have [an open GitHub issue](https://github.com/electric-sql/electric/issues/1882) tracking this if you'd like to contribute a library based on the `tanstack` example that integrates Electric and TanStack into a higher level interface. Please [leave a comment](https://github.com/electric-sql/electric/issues/1882) or [ask on Discord](https://discord.electric-sql.com) if you'd like any pointers or to discuss how best to approach this. --- # Yjs - Integrations | ElectricSQL Return to top ![](/img/integrations/yjs.svg) Yjs [​](#yjs) ============== [Yjs](https://docs.yjs.dev) is a CRDT library for building collaborative applications. Electric and Yjs [​](#electric-and-yjs) ---------------------------------------- Electric can be used with Yjs to create collaborative applications on top of Postgres. See the [Notes](/demos/notes) and [Yjs provider](/demos/yjs) examples for two different example apps showing how to integrate Electric with Yjs. Help wanted [Good first issue](https://github.com/electric-sql/electric/labels/good%20first%20issue) ----------------------------------------------------------------------------------------------------- We have [an open GitHub issue](https://github.com/electric-sql/electric/issues/2121) tracking this if you'd like to contribute a Yjs integration library based on the `notes` and `yjs-provider` examples. Please [leave a comment](https://github.com/electric-sql/electric/issues/2121) or [ask on Discord](https://discord.electric-sql.com) if you'd like any pointers or to discuss how best to approach this. --- # Cloudflare - Integrations | ElectricSQL Return to top ![](/img/integrations/cloudflare.svg) Cloudflare [​](#cloudflare) ============================ Cloudflare is a global network and edge-cloud platform. Electric and Cloudflare [​](#electric-and-cloudflare) ------------------------------------------------------ You can use Cloudflare [as a CDN](#cdn) in front of Electric and as a sync target to sync data into Cloudflare primitives including [Workers](#workers) and [Durable Objects](#durable-objects) . Need context? See the [Deployment guide](/docs/guides/deployment) for more details. ### CDN [​](#cdn) Cloudflare provides a [global content delivery network](https://developers.cloudflare.com/cache/get-started/) . [This guide](https://loadforge.com/guides/steps-to-set-up-cloudflare-cdn-for-your-website) walks through the process of using it. Basically you need to create a DNS rule resolving to the Electric service and enable Cloudflare as a proxy for it. Electric's [HTTP API caching](/docs/api/http#caching) will work out of the box. ### Workers [​](#workers) You can also use [Cloudflare Workers](https://workers.cloudflare.com) in front of the CDN to handle concerns like authorization and routing. #### Auth example [​](#auth-example) For example, you could validate an auth token to protect access to a shape and then proxy the request through to Electric: ts export default { async fetch(request): Promise { const ELECTRIC_URL = 'https://my-electric.example.com' const headers = request.headers const authHeader = request.headers.get('Authorization') const isValid = (header) => { /* ... e.g.: verify JWT ... */ } if (!isValid(authHeader)) { return new Response('Forbidden', {status: 403}) } if (request.method != `GET`) { return new Response('Method Not Allowed', {status: 405}) } const url = new URL(request.url) const shapeUrl = `${ELECTRIC_URL}${url.pathname}${url.search}` const clonedHeaders = new Headers(new Request(request).headers) return await fetch( shapeUrl, { headers: clonedHeaders, cf: { cacheEverything: true } } ) }, } satisfies ExportedHandler; #### Syncing data into the worker [​](#syncing-data-into-the-worker) Or you can use Electric to hydrate data quickly into an edge worker. For example, you could sync data into an edge worker to dynamically redirect the request: ts import { ShapeStream, Shape } from '@electric-sql/client' export default { async fetch(request): Promise { const ELECTRIC_URL = 'https://my-electric.example.com' const stream = new ShapeStream({ url: `${ELECTRIC_URL}/v1/shape`, params: { table: 'routes' } }) const shape = new Shape(stream) const routes = await shape.value const url = new URL(request.url) const match = routes.find(x => x.path == url.pathname) if (!match) { return new Response('Not Found', {status: 404}) } return Response.redirect(match.redirect, 301) }, } satisfies ExportedHandler; ### Durable Objects [​](#durable-objects) You can implement a similar pattern to the [sync example above](#syncing-data-into-the-worker) to sync data into a Durable Object. The key difference is that with a [Durable Object](https://developers.cloudflare.com/durable-objects/) , the data can be persisted across requests. This allows you to sync a shape log into the Durable Object, materialise the shape into persistent storage and then re-sync the latest changes whenever the Durable Object is accessed. You can see a demo of this pattern, using SQLite to persist the Shape data, at [KyleAMathews/electric-demo-cloudflare-sqlite](https://github.com/KyleAMathews/electric-demo-cloudflare-sqlite) : Combining CDN and Durable Objects Note that if you sync data into a Durable Object (or a Worker) [from Cloudflare's CDN](#cdn) it can be _extremely fast_ — with high bandwidth and low network latency. Help wanted [Good first issue](https://github.com/electric-sql/electric/labels/good%20first%20issue) ----------------------------------------------------------------------------------------------------- We have [an open GitHub issue](https://github.com/electric-sql/electric/issues/1884) tracking this if you'd like to contribute example apps using Cloudflare and/or wrap up the code samples above into a library. Please [leave a comment](https://github.com/electric-sql/electric/issues/1884) or [ask on Discord](https://discord.electric-sql.com) if you'd like any pointers or to discuss how best to approach this. --- # React - Integrations | ElectricSQL Return to top ![](/img/integrations/react.svg) React [​](#react) ================== React is a popular library for building declarative, component-based UI. Electric and React [​](#electric-and-react) -------------------------------------------- Electric has first-class support for React. We maintain a [react-hooks](https://github.com/electric-sql/electric/tree/main/packages/react-hooks) package that provides a number of [React Hooks](https://react.dev/reference/react/hooks) to bind Shape data to your components. How to use [​](#how-to-use) ---------------------------- ### Install [​](#install) The package is published on NPM as [`@electric-sql/react`](https://www.npmjs.com/package/@electric-sql/react) . Install using e.g.: shell npm i @electric-sql/react ### `useShape` [​](#useshape) [`useShape`](https://github.com/electric-sql/electric/blob/main/packages/react-hooks/src/react-hooks.tsx#L131) binds a materialised [Shape](/docs/api/clients/typescript#shape) to a state variable. For example: tsx import { useShape } from '@electric-sql/react' const MyComponent = () => { const { isLoading, data } = useShape<{title: string}>({ url: `http://localhost:3000/v1/shape`, params: { table: 'items' } }) if (isLoading) { return
Loading ...
} return (
{data.map(item =>
{item.title}
)}
) } You can also include additional PostgreSQL-specific parameters: tsx const MyFilteredComponent = () => { const { isLoading, data } = useShape<{id: number, title: string}>({ url: `http://localhost:3000/v1/shape`, params: { table: 'items', where: 'status = \'active\'', columns: ['id', 'title'] } }) // ... } `useShape` takes the same options as [ShapeStream](/docs/api/clients/typescript#options) . The return value is a `UseShapeResult`: tsx export interface UseShapeResult = Row> { /** * The array of rows that make up the materialised Shape. * @type {T[]} */ data: T[] /** * The Shape instance used by this useShape * @type {Shape} */ shape: Shape /** True during initial fetch. False afterwise. */ isLoading: boolean /** Unix time at which we last synced. Undefined when `isLoading` is true. */ lastSyncedAt?: number /** Unix time at which we last synced. Undefined when `isLoading` is true. */ isError: boolean error: Shape[`error`] } ### `preloadShape` [​](#preloadshape) [`preloadShape`](https://github.com/electric-sql/electric/blob/main/packages/react-hooks/src/react-hooks.tsx#L17) is useful to call in route loading functions or elsewhere when you want to ensure Shape data is loaded before rendering a route or component. tsx export const clientLoader = async () => { return await preloadShape({ url: `http://localhost:3000/v1/shape`, params: { table: 'items' } }) } You can also preload filtered data: tsx export const filteredLoader = async () => { return await preloadShape({ url: `http://localhost:3000/v1/shape`, params: { table: 'items', where: 'category = \'electronics\'', columns: ['id', 'name', 'price'] } }) } It takes the same options as [ShapeStream](/docs/api/clients/typescript#options) . ### `getShapeStream` [​](#getshapestream) [`getShapeStream`](https://github.com/electric-sql/electric/blob/main/packages/react-hooks/src/react-hooks.tsx#L30) get-or-creates a `ShapeStream` off the global cache. tsx const itemsStream = getShapeStream({ url: `http://localhost:3000/v1/shape`, params: { table: 'items' } }) This allows you to avoid consuming multiple streams for the same shape log. ### `getShape` [​](#getshape) [`getShape`](https://github.com/electric-sql/electric/blob/main/packages/react-hooks/src/react-hooks.tsx#L49) get-or-creates a `Shape` off the global cache. tsx const itemsShape = getShape({ url: `http://localhost:3000/v1/shape`, params: { table: 'items' } }) This allows you to avoid materialising multiple shapes for the same stream. ### How to abort a shape subscription — `AbortController` [​](#how-to-abort-a-shape-subscription-—-abortcontroller) If you'd like to abort the shape's subscription to live updates e.g. after unmounting a component or navigating away from a route, you can use the [`AbortController`](https://developer.mozilla.org/en-US/docs/Web/API/AbortController) . The following is a simple example which aborts the subscription when the component is unmounted. tsx function MyComponent() { const [controller, _] = useState(new AbortController()) const { data } = useShape({ ... signal: controller.signal }) useEffect(() => { return () { // Live updates are now disabled. controller.abort() } }, []) ... } Note that if you have multiple components using the same component, this will stop updates for all subscribers. Which is probably not what you want. We plan to add a better API for unsubscribing from updates & cleaning up shapes that are no longer needed. If interested, please file an issue to start a discussion. --- # Redis - Integrations | ElectricSQL Return to top ![](/img/integrations/redis.svg) Redis [​](#redis) ================== Redis is an in-memory "data structure server", often used as a cache. Electric and Redis [​](#electric-and-redis) -------------------------------------------- Many applications use [Redis](https://redis.io/docs/latest/develop/use/client-side-caching/) as a local cache. With Electric, you can define a [Shape](/docs/guides/shapes) and sync it into a [Redis data structure](https://redis.io/docs/latest/develop/data-types/hashes/) . ### Example [​](#example) The shape stream comes through as a [log](/docs/api/http#shape-log) of insert, update and delete messages. Apply these to the Redis hash and the cache automatically stays up-to-date: ts import { createClient } from 'redis' import { ShapeStream, Message, isChangeMessage } from '@electric-sql/client' // Create a Redis client const REDIS_HOST = `localhost` const REDIS_PORT = 6379 const client = createClient({ url: `redis://${REDIS_HOST}:${REDIS_PORT}`, }) client.connect().then(async () => { console.log(`Connected to Redis server`) // Clear out old data on the hash. client.del(`items`) // Lua script for updating hash field. We need to merge in partial updates // from the shape log. const script = ` local current = redis.call('HGET', KEYS[1], KEYS[2]) local parsed = {} if current then parsed = cjson.decode(current) end for k, v in pairs(cjson.decode(ARGV[1])) do parsed[k] = v end local updated = cjson.encode(parsed) return redis.call('HSET', KEYS[1], KEYS[2], updated) ` // Load the script into Redis and get its SHA1 digest const updateKeyScriptSha1 = await client.SCRIPT_LOAD(script) const itemsStream = new ShapeStream({ url: `http://localhost:3000/v1/shape`, params: { table: `items`, }, }) itemsStream.subscribe(async (messages: Message[]) => { // Begin a Redis transaction // // FIXME The Redis docs suggest only sending 10k commands at a time // to avoid excess memory usage buffering commands. const pipeline = client.multi() // Loop through each message and make writes to the Redis hash for action messages messages.forEach((message) => { if (!isChangeMessage(message)) return console.log(`message`, message) // Upsert/delete switch (message.headers.operation) { case `delete`: pipeline.hDel(`items`, message.key) break case `insert`: pipeline.hSet( `items`, String(message.key), JSON.stringify(message.value) ) break case `update`: { pipeline.evalSha(updateKeyScriptSha1, { keys: [`items`, String(message.key)], arguments: [JSON.stringify(message.value)], }) break } } }) // Execute all commands as a single transaction try { await pipeline.exec() console.log(`Redis hash updated successfully with latest shape updates`) } catch (error) { console.error(`Error while updating hash:`, error) } }) }) See the [Redis example](/demos/redis) for more details. Help wanted [Good first issue](https://github.com/electric-sql/electric/labels/good%20first%20issue) ----------------------------------------------------------------------------------------------------- We have [an open GitHub issue](https://github.com/electric-sql/electric/issues/1881) tracking this if you'd like to contribute a library that wraps up the `redis-sync` example into an `@electric-sql/redis` integration library. Please [leave a comment](https://github.com/electric-sql/electric/issues/1881) or [ask on Discord](https://discord.electric-sql.com) if you'd like any pointers or to discuss how best to approach this. --- # Phoenix - Integrations | ElectricSQL Return to top ![](/img/integrations/phoenix.svg) Phoenix [​](#phoenix) ====================== [Phoenix](https://www.phoenixframework.org) is a full-stack web development framework for [Elixir](https://elixir-lang.org) . Electric and Phoenix [​](#electric-and-phoenix) ------------------------------------------------ Electric is [developed in Elixir](/product/electric#how-does-it-work) and provides [an Elixir client](/docs/api/clients/elixir) . We've leveraged this to develop a batteries-included Phoenix integration for: * [front-end sync](#front-end-sync) : into a front-end client from a Postgres-backed Phoenix application * [LiveView sync](#liveview-sync) : into Phoenix LiveView from Postgres in realtime via [Phoenix.Streams](/docs/integrations/phoenix#liveview-sync) `Electric.Phoenix` is published on Hex as [hex.pm/packages/electric\_phoenix](https://hex.pm/packages/electric_phoenix) . ### Inspiration [​](#inspiration) It was inspired by [`josevalim/sync`](https://github.com/josevalim/sync) . You can read José's [original design document](https://github.com/josevalim/sync/blob/main/DESIGN.md) . How to use [​](#how-to-use) ---------------------------- ### Front-end sync [​](#front-end-sync) Phoenix is a general framework that provides a number of different methods to get data from the server to the client. These include exposing [REST APIs](https://hexdocs.pm/phoenix/routing.html#resources) and using [Absinthe](https://hexdocs.pm/absinthe/overview.html) to expose a GraphQL endpoint. `Electric.Phoenix` provides an alternative method: exposing [Shapes](/docs/guides/shapes) that sync data directly from Postgres into the client. With this, shapes are exposed and configured in your Phoenix Router. For example, here we expose a predefined shape of all visible todos, deriving the shape definition from an Ecto query using your existing data model: elixir defmodule MyAppWeb.Router do use Phoenix.Router alias MyApp.Todos.Todo scope "/shapes" do pipe_through :browser get "/todos", Electric.Phoenix.Plug, shape: Electric.Client.shape!(Todo, where: "visible = true") end end Because the shape is defined in your Router, it can use Plug middleware for authorization. See [Parameter-based shapes](https://hexdocs.pm/electric_phoenix/Electric.Phoenix.Plug.html#module-parameter-based-shapes) for more details. ### LiveView sync [​](#liveview-sync) [Phoenix LiveView](https://hexdocs.pm/phoenix_live_view) allows you to develop interactive web applications in Elixir/Phoenix, often without writing any front-end code. LiveView provides a primitive, called [Phoenix.Streams](https://fly.io/phoenix-files/phoenix-dev-blog-streams) that allows you to stream data into a LiveView. `Electric.Phoenix` provides a wrapper around this to automatically stream a [Shape](/docs/guides/shapes) into a LiveView. The key primitive is an [`electric_stream/4`](https://hexdocs.pm/electric_phoenix/Electric.Phoenix.LiveView.html#electric_stream/4) function that wraps [`Phoenix.LiveView.stream/4`](https://hexdocs.pm/phoenix_live_view/Phoenix.LiveView.html#stream/4) to provide a live updating collection of items. elixir def mount(_params, _session, socket) do socket = Electric.Phoenix.LiveView.electric_stream( socket, :visible_todos, from(t in Todo, where: t.visible == true) ) {:ok, socket} end This makes your LiveView applications real-time. In fact, it allows you to build interactive, real-time multi-user applications straight out of your existing Ecto schema, without writing any JavaScript at all 🤯 ### More details [​](#more-details) For more details and full documentation see [hexdocs.pm/electric\_phoenix](https://hexdocs.pm/electric_phoenix) . Examples [​](#examples) ------------------------ ### Phoenix LiveView [​](#phoenix-liveview) See the [Phoenix LiveView example](/demos/phoenix-liveview) . This is an example Phoenix LiveView application that uses [`Electric.Phoenix.LiveView.electric_stream/4`](https://hexdocs.pm/electric_phoenix/Electric.Phoenix.LiveView.html#electric_stream/4) to sync data from Postgres into a LiveView using [Phoenix Streams](https://fly.io/phoenix-files/phoenix-dev-blog-streams/) . This keeps the LiveView automatically in-sync with Postgres, without having to re-run queries or trigger any change handling yourself. ### Gatekeeper Auth [​](#gatekeeper-auth) The [Gatekeeper auth](/demos/gatekeeper-auth) example also contains a Phoenix application that uses [`Electric.Phoenix.Plug`](https://hexdocs.pm/electric_phoenix/Electric.Phoenix.Plug.html) to authorize shape access and issue shape-scoped access tokens. Help wanted [Good first issue](https://github.com/electric-sql/electric/labels/good%20first%20issue) ----------------------------------------------------------------------------------------------------- We have [an open GitHub issue](https://github.com/electric-sql/electric/issues/1878) tracking this if you'd like to contribute an equivalent integration for other server-side frameworks, such as Rails, Laravel, Django, etc. Please [leave a comment](https://github.com/electric-sql/electric/issues/1878) or [ask on Discord](https://discord.electric-sql.com) if you'd like any pointers or to discuss how best to approach this. --- # Next.js - Integrations | ElectricSQL Return to top ![](/img/integrations/next.svg) Next.js [​](#next-js) ====================== [Next.js](https://mobx.js.org) is a full-stack React framework. Electric and Next.js [​](#electric-and-next-js) ------------------------------------------------ Next.js is based on React. Electric [works with React](./react) . You can integrate Electric into your Next.js application like any other npm / React library. ### Examples [​](#examples) #### Next.js example [​](#next-js-example) See the [Nextjs example](/demos/nextjs) on GitHub. This demonstrates using Electric for read-path sync and a Next.js API for handling writes: tsx "use client" import { v4 as uuidv4 } from "uuid" import { useOptimistic } from "react" import { useShape, getShapeStream } from "@electric-sql/react" import "./Example.css" import { matchStream } from "./match-stream" import { ShapeStreamOptions } from "@electric-sql/client/*" const itemShape = (): ShapeStreamOptions => { if (typeof window !== `undefined`) { return { url: new URL(`/shape-proxy`, window?.location.origin).href, params: { table: `items`, }, } } else { return { url: new URL(`https://not-sure-how-this-works.com/shape-proxy`).href, params: { table: `items`, }, } } } type Item = { id: string } async function createItem(newId: string) { const itemsStream = getShapeStream(itemShape()) // Match the insert const findUpdatePromise = matchStream({ stream: itemsStream, operations: [`insert`], matchFn: ({ message }) => message.value.id === newId, }) // Generate new UUID and post to backend const fetchPromise = fetch(`/api/items`, { method: `POST`, body: JSON.stringify({ uuid: newId }), }) return await Promise.all([findUpdatePromise, fetchPromise]) } async function clearItems() { const itemsStream = getShapeStream(itemShape()) // Match the delete const findUpdatePromise = matchStream({ stream: itemsStream, operations: [`delete`], // First delete will match matchFn: () => true, }) // Post to backend to delete everything const fetchPromise = fetch(`/api/items`, { method: `DELETE`, }) return await Promise.all([findUpdatePromise, fetchPromise]) } export default function Home() { const { data: items } = useShape(itemShape()) const [optimisticItems, updateOptimisticItems] = useOptimistic< Item[], { newId?: string; isClear?: boolean } >(items, (state, { newId, isClear }) => { if (isClear) { return [] } if (newId) { // Merge data from shape & optimistic data from fetchers. This removes // possible duplicates as there's a potential race condition where // useShape updates from the stream slightly before the action has finished. const itemsMap = new Map() state.concat([{ id: newId }]).forEach((item) => { itemsMap.set(item.id, { ...itemsMap.get(item.id), ...item }) }) return Array.from(itemsMap.values()) } return [] }) return (
{ const intent = formData.get(`intent`) const newId = formData.get(`new-id`) as string if (intent === `add`) { updateOptimisticItems({ newId }) await createItem(newId) } else if (intent === `clear`) { updateOptimisticItems({ isClear: true }) await clearItems() } }} >
{optimisticItems.map((item: Item, index: number) => (

{item.id}

))}
) } It also demonstrates using a [shape-proxy endpoint](https://github.com/electric-sql/electric/blob/main/examples/nextjs-example/app/shape-proxy/route.ts) for proxying access to the Electric sync service. This allows you to implement [auth](/docs/guides/auth) and routing in-front-of Electric (and other concerns like transforming or decrypting the stream) using your Next.js backend: ts export async function GET(request: Request) { const url = new URL(request.url) const originUrl = new URL( process.env.ELECTRIC_URL ? `${process.env.ELECTRIC_URL}/v1/shape` : `http://localhost:3000/v1/shape` ) url.searchParams.forEach((value, key) => { originUrl.searchParams.set(key, value) }) if (process.env.ELECTRIC_SOURCE_ID) { originUrl.searchParams.set(`source_id`, process.env.ELECTRIC_SOURCE_ID) } const headers = new Headers() if (process.env.ELECTRIC_SOURCE_SECRET) { originUrl.searchParams.set( `source_secret`, process.env.ELECTRIC_SOURCE_SECRET ) } const newRequest = new Request(originUrl.toString(), { method: `GET`, headers, }) // When proxying long-polling requests, content-encoding & content-length are added // erroneously (saying the body is gzipped when it's not) so we'll just remove // them to avoid content decoding errors in the browser. // // Similar-ish problem to https://github.com/wintercg/fetch/issues/23 let resp = await fetch(newRequest) if (resp.headers.get(`content-encoding`)) { const headers = new Headers(resp.headers) headers.delete(`content-encoding`) headers.delete(`content-length`) resp = new Response(resp.body, { status: resp.status, statusText: resp.statusText, headers, }) } return resp } #### ElectroDrizzle [​](#electrodrizzle) [ElectroDrizzle](https://github.com/LeonAlvarez/ElectroDrizzle) is an example application by [Leon Alvarez](https://github.com/LeonAlvarez) using Next.js, [Drizzle](https://orm.drizzle.team) , [PGLite](/product/pglite) and Electric together. See the [Getting Started guide here](https://github.com/LeonAlvarez/ElectroDrizzle?tab=readme-ov-file#getting-started) . #### SSR [​](#ssr) Next.js supports SSR. We are currently [experimenting with patterns](https://github.com/electric-sql/electric/pull/1596) to use Electric with SSR in a way that supports server rendering _and_ client-side components seamlessly moving into realtime sync. Help wanted [Good first issue](https://github.com/electric-sql/electric/labels/good%20first%20issue) ----------------------------------------------------------------------------------------------------- We have [a pull request](https://github.com/electric-sql/electric/issues/1596) open if you'd like to contribute to improving our Next.js documentation, patterns and framework integrations. Please [leave a comment](https://github.com/electric-sql/electric/issues/1596) or [ask on Discord](https://discord.electric-sql.com) if you'd like any pointers or to discuss how best to approach this. --- # Amazon Web Services (AWS) - Integrations | ElectricSQL Return to top ![](/img/integrations/aws.svg) Amazon Web Services (AWS) [​](#amazon-web-services-aws) ======================================================== AWS is a cloud infrastructure platform. Electric and AWS [​](#electric-and-aws) ---------------------------------------- You can use AWS to deploy any or all components of the Electric stack: * [deploy a Postgres database](#deploy-postgres) * [an Electric sync service](#deploy-electric) * [your client application](#deploy-your-app) If you already run Postgres in AWS, potentially using RDS or Aurora, then it's a great idea to also deploy Electric within the same network. Need context? See the [Deployment guide](/docs/guides/deployment) for more details. ### Deploy Postgres [​](#deploy-postgres) AWS provides Postgres hosting via RDS and Aurora. Electric works with either. You need to configure them to enable logical replication and connect with the right user. The default `wal_level` is `minimal` for RDS and `replica` for Aurora. It can be changed to `logical` by creating a [custom parameter group](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_WorkingWithParamGroups.html) for the RDS instance (or the Aurora DB cluster) and setting the value of the `rds.logical_replication` parameter to `1` and rebooting the instance. The default `postgres` user has the `REPLICATION` role. If you need to add it to another user you can do so by granting the `rds_replication` role, e.g.: sql GRANT rds_replication TO someuser; ### Deploy Electric [​](#deploy-electric) AWS provides a [wide range of container hosting](https://aws.amazon.com/containers) . For example, you can deploy Electric to [AWS Elastic Container Service](https://aws.amazon.com/efs) using [AWS Fargate](https://aws.amazon.com/fargate) . You should store Shape logs to a persistent disk (not an ephemoral filesystem). For example using [Amazon Elastic File System](https://aws.amazon.com/efs) . ### Deploy your app [​](#deploy-your-app) AWS provides a range of [website hosting options](https://aws.amazon.com/getting-started/hands-on/host-static-website/) . For example you can deploy a static app to [AWS Amplify](https://aws.amazon.com/amplify) . Examples [​](#examples) ------------------------ ### AWS Terraform [​](#aws-terraform) We have an example Terraform repo at [electric-sql/terraform-aws](https://github.com/electric-sql/terraform-aws) . --- # Crunchy Data - Integrations | ElectricSQL Return to top ![](/img/integrations/crunchy.svg) Crunchy Data [​](#crunchy-data) ================================ Crunchy is a Postgres hosting provider. Electric and Crunchy [​](#electric-and-crunchy) ------------------------------------------------ You can use Electric with [Crunchy Bridge](https://www.crunchydata.com/products/crunchy-bridge) , their managed cloud Postgres product. Need context? See the [Deployment guide](/docs/guides/deployment) for more details. ### Deploy Postgres [​](#deploy-postgres) [Sign up to Crunchy Bridge](https://crunchybridge.com/register) and go through the steps to create a cluster. Go to the "Connection" tab, select "URL", set the role to "postgres (superuser)" and copy the connection string. You can then run Electric with this connection string as the `DATABASE_URL`, e.g.: shell docker run -it \ -e "DATABASE_URL=postgres://postgres:****@p.YOUR_CLUSTER_ID.db.postgresbridge.com:5432/postgres" \ electricsql/electric:latest You can also use the `postgres` superuser to create other users with the `REPLICATION` role, e.g.: sql CREATE ROLE electric WITH REPLICATION LOGIN PASSWORD '...'; GRANT ALL PRIVILEGES ON DATABASE "postgres" to electric; You can then connect as the new `electric` user. Need somewhere to host Electric? If you need somewhere to deploy Electric then [Crunchy works well](https://neon.tech/docs/guides/render) with [Render](./render#deploy-electric) . --- # Neon - Integrations | ElectricSQL Return to top ![](/img/integrations/neon.svg) Neon [​](#neon) ================ [Neon](https://neon.tech) is a serverless Postgres hosting platform. Electric and Neon [​](#electric-and-neon) ------------------------------------------ You can use Electric with Neon's [serverless Postgres hosting](https://neon.tech/docs/introduction/serverless) . Need context? See the [Deployment guide](/docs/guides/deployment) for more details. ### Deploy Postgres [​](#deploy-postgres) [Sign up to Neon](https://neon.tech/docs/get-started-with-neon/signing-up) and go through the steps to create a database. On the project page, go to `Settings -> Logical Replication` and click "Enable". Neon and logical replication See the [Neon guide on logical replication](https://neon.tech/docs/guides/logical-replication-neon) for information about how logical replication works with the rest of the Neon feature set. ### Connect Electric [​](#connect-electric) Go to the Dashboard page and copy the database connection string. Make sure you **don't** check "Pooled connection". You want the direct connection string in order to use logical replication. You can then run Electric with this connection string as the `DATABASE_URL`, e.g.: shell docker run -it \ -e "DATABASE_URL=YOUR_NEON_CONNECTION_STRING" \ electricsql/electric:latest Need somewhere to host Electric? If you need somewhere to deploy Electric then [Neon works well](https://neon.tech/docs/guides/render) with [Render](./render#deploy-electric) . PGlite [​](#pglite) -------------------- Electric and Neon have also collaborated to develop [PGlite](/product/pglite) , which was started as a project by Neon's CTO, [Stas Kelvich](https://github.com/kelvich) . --- # Digital Ocean - Integrations | ElectricSQL Return to top ![](/img/integrations/digital-ocean.svg) Digital Ocean [​](#digital-ocean) ================================== Digital Ocean is a cloud hosting platform. Electric and Digital Ocean [​](#electric-and-digital-ocean) ------------------------------------------------------------ You can use Digital Ocean to deploy any or all components of the Electric stack: * [deploy a Postgres database](#deploy-postgres) * [an Electric sync service](#deploy-electric) * [your client application](#deploy-your-app) If you already run a Managed Postgres in Digital Ocean, then it's a great idea to also deploy Electric within the same network. Need context? See the [Deployment guide](/docs/guides/deployment) for more details. ### Deploy Postgres [​](#deploy-postgres) Digital Ocean provides [Managed Postgres](https://docs.digitalocean.com/products/databases/postgresql/) . This has logical replication enabled and works with Electric out of the box. Use `doadmin` for older Postgres versions If you're using Postgres version 15 or lower, you will need to connect to your Managed Postgres as the `doadmin` user. This is the default user and the only user with the `REPLICATION` role. (With later Postgres versions its fine to create other users and use the `doadmin` user to grant them the `REPLICATION` role). ### Deploy Electric [​](#deploy-electric) Digital Ocean has a number of different ways to deploy web services. We recommend using a [Docker Droplet](https://marketplace.digitalocean.com/apps/docker) . Below we walk through the steps to deploy Electric using a Docker Droplet. First you create the Droplet. Then setup some Docker / SSH networking so your local Docker can talk to it. Then use Docker Compose to run Electric inside the Droplet. Don't use App Platform We **don't recommend** that you use [App Platform](https://docs.digitalocean.com/products/app-platform/) to deploy the Electric sync service because App Platform does not provide persistent file storage for Shape logs. #### Create Droplet [​](#create-droplet) Go to the [Docker marketplace page](https://marketplace.digitalocean.com/apps/docker) and click on the "Create Docker Droplet" button. Follow the prompts. You **must** use key-based SSH authentication (so that you can set up your local Docker to talk to the remote daemon). It's a good idea to change the hostname to something like `electric-sync` as well. Create the Droplet and wait until its ready with an IPv4 address. Copy the address and use it in place of `YOUR_IP_ADDRESS` in the instructions that follow. #### Connect Docker [​](#connect-docker) Connect to your new Droplet using `ssh` in order to verify the authenticity of the host and add its public key to your local `known_hosts` file. console $ ssh root@YOUR_IP_ADDRESS ... Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added 'YOUR_IP_ADDRESS' (ED25519) to the list of known hosts. Permission denied? If the output from the command above ends with: ... Permission denied (publickey). Then you need to add a section to your `~/.ssg/config` to tell it to use your SSH key when connecting to `YOUR_IP_ADDRESS`. Something like this will do: Host YOUR_IP_ADDRESS Port 22 Hostname YOUR_IP_ADDRESS AddKeysToAgent yes IdentitiesOnly yes IdentityFile ~/.ssh/path_to_your_private_ssh_key TCPKeepAlive yes UseKeychain yes Now set the `DOCKER_HOST` environment variable to point to your Droplet's IP address: shell export DOCKER_HOST=ssh://root@YOUR_IP_ADDRESS #### Deploy [​](#deploy) Save the following contents into a file called `compose.yaml`, changing the `DATABASE_URL` and setting [any other environment variables](/docs/api/config) to match your setup. yaml services: electric: image: electricsql/electric:latest environment: DATABASE_URL: "postgresql://..." ports: - 80:3000 restart: always Now launch on the remote server, with output that should look something like this: console $ docker compose up [+] Running 8/8 ✔ electric 7 layers [⣿⣿⣿⣿⣿⣿⣿] 0B/0B Pulled 8.2s ✔ efc2b5ad9eec Pull complete 3.4s ✔ 2cb0d575dcef Pull complete 4.5s ✔ c1b251d76665 Pull complete 4.6s ✔ c82981779fd9 Pull complete 4.7s ✔ 65b429e477c5 Pull complete 4.8s ✔ 1fd7ee9efb04 Pull complete 6.0s ✔ 87053f06541e Pull complete 6.1s [+] Running 2/2 ✔ Network electric-sync-droplet_default Created 0.2s ✔ Container electric-sync-droplet-electric-1 Created 0.2s Attaching to electric-sync-droplet-electric-1 electric-sync-droplet-electric-1 | =INFO REPORT==== 23-Oct-2024::13:16:01.777082 === electric-sync-droplet-electric-1 | Loading 140 CA(s) from otp store electric-sync-droplet-electric-1 | 13:16:01.832 [info] Running Electric.Plug.Router with Bandit 1.5.5 at 0.0.0.0:3000 (http) electric-sync-droplet-electric-1 | 13:16:01.935 [info] Acquiring lock from postgres with name electric_slot_default electric-sync-droplet-electric-1 | 13:16:01.937 [info] Lock acquired from postgres with name electric_slot_default electric-sync-droplet-electric-1 | 13:16:02.006 [info] Postgres server version = 160004, system identifier = 7428958789530034185, timeline_id = 1 electric-sync-droplet-electric-1 | 13:16:02.145 [info] No previous timeline detected. electric-sync-droplet-electric-1 | 13:16:02.146 [info] Connected to Postgres and timeline electric-sync-droplet-electric-1 | 13:16:02.147 [info] Starting shape replication pipeline electric-sync-droplet-electric-1 | 13:16:02.150 [info] Starting replication from postgres You can hit the health check endpoint to verify that everything is running OK: console $ curl http://YOUR_IP_ADDRESS/v1/health {"status":"active"} ### Deploy your app [​](#deploy-your-app) You can deploy [your client app to Digital Ocean using App Platform](https://www.digitalocean.com/community/tutorials/how-to-deploy-a-static-website-to-the-cloud-with-digitalocean-app-platform) . --- # Expo - Integrations | ElectricSQL Return to top ![](/img/integrations/expo.svg) Expo [​](#expo) ================ Expo is a platform that helps you deploy React Native applications. Electric and Expo [​](#electric-and-expo) ------------------------------------------ Expo applications are developed in Javacript (or Typescript) using [React Native](https://reactnative.dev) . You can use the Electric [Typescript client](/docs/api/clients/typescript) in your Expo applications. This allows you to sync data from Electric into mobile apps. Need context? See the [Deployment guide](/docs/guides/deployment) for more details. Example [​](#example) ---------------------- Follow the [Expo Quickstart](https://docs.expo.dev/get-started/create-a-project/) to create an Expo app. Replace the generated `./app/(tabs)/index.tsx` with the following: tsx import { Text } from 'react-native' import { useShape } from '@electric-sql/react' // Edit to match your setup. const ELECTRIC_URL = 'https://my-electric-sync-service.example.com' export default function HomeScreen() { const { isLoading, data } = useShape({ url: `${ELECTRIC_URL}/v1/shape`, params: { table: 'items' } }) if (isLoading) { return null } return ( { JSON.stringify(data, null, 4) } ) } Install `@electric-sql/react` (if necessary using `--force` to work around a React dependency version mismatch): shell npm install '@electric-sql/react' --force Run, e.g. in the browser: shell npm run web If there's data in the `items` table of your Postgres, you should see it syncing into your app. PGlite [​](#pglite) -------------------- [PGlite](https://pglite.dev) doesn't _yet_ work in React Native. We have an [open issue tracking support for it](https://github.com/electric-sql/pglite/issues/87) . When it does, we hope to work with the Expo team to get an official `expo-pglite` package published. --- # Fly.io - Integrations | ElectricSQL Return to top ![](/img/integrations/fly.svg) Fly.io [​](#fly-io) ==================== [Fly.io](https://fly.io) is a public cloud built for developers who ship. Electric and Fly [​](#electric-and-fly) ---------------------------------------- You can use Fly to deploy any or all components of the Electric stack: * [deploy a Postgres database](#deploy-postgres) * [an Electric sync service](#deploy-electric) * [your client application](#deploy-your-app) One of Fly's specialities is deploying Elixir applications. So Fly is especially good for [deploying the Electric sync service](#deploy-electric) and/or [Phoenix applications](./phoenix) using Electric. Need context? See the [Deployment guide](/docs/guides/deployment) for more details. ### Deploy Postgres [​](#deploy-postgres) [Fly Postgres](https://fly.io/docs/postgres) is not a managed Postgres service. See the general advice on [Running Postgres](/docs/guides/deployment#_1-running-postgres) in the deployment guide for information on how to configure Postgres to work with Electric. Fly's [Supabase Postgres](https://fly.io/docs/supabase) is a managed Postgres service, powered by [Supabase](./supabase) . If you use it, make sure to connect on the IPv6 `DATABASE_URL` rather than the `DATABASE_POOLER_URL`. See the [Supabase deployment docs](./supabase#deploy-postgres) and the IPv6 section of the [troubleshooting guide](/docs/guides/troubleshooting#ipv6-support) for more information about IPv6 support. ### Deploy Electric [​](#deploy-electric) Copy the following config into a file called `fly.toml`, replacing the app name and `DATABASE_URL`: toml app = "YOUR_UNIQUE_APP_NAME" [build] image = "electricsql/electric:latest" [env] DATABASE_URL = "postgresql://..." ELECTRIC_DATABASE_USE_IPV6 = true [http_service] internal_port = 3000 force_https = true [[http_service.checks]] interval = "10s" timeout = "2s" grace_period = "20s" method = "GET" path = "/v1/health" Using the [`flyctl` client](https://fly.io/docs/flyctl/install/) , in the same directory as `fly.toml`, run: shell flyctl launch --copy-config --ha=false Hit the health check endpoint to verify that everything is running OK: console $ curl https://YOUR_UNIQUE_APP_NAME.fly.dev/v1/health {"status":"active"} ### Deploy your app [​](#deploy-your-app) You can run most kinds of apps on Fly, including [static sites](https://fly.io/docs/languages-and-frameworks/static/) . --- # Google Cloud Platform (GCP) - Integrations | ElectricSQL Return to top ![](/img/integrations/gcp.svg) Google Cloud Platform (GCP) [​](#google-cloud-platform-gcp) ============================================================ GCP is a cloud infrastructure platform. Electric and GCP [​](#electric-and-gcp) ---------------------------------------- You can use GCP to deploy any or all components of the Electric stack: * [deploy a Postgres database](#deploy-postgres) * [an Electric sync service](#deploy-electric) * [your client application](#deploy-your-app) If you already run Postgres in GCP, then it's a great idea to also deploy Electric within the same network. Need context? See the [Deployment guide](/docs/guides/deployment) for more details. ### Deploy Postgres [​](#deploy-postgres) GCP provides Postgres hosting via [Cloud SQL](https://cloud.google.com/sql/docs/postgres/) or [AlloyDB](https://cloud.google.com/alloydb) . Electric works with either. You need to configure them to enable logical replication and connect with the right user. #### Cloud SQL [​](#cloud-sql) The default `wal_level` is `replica`. Change it to `logical` by [setting the `cloudsql.logical_decoding` flag to `on`](https://cloud.google.com/sql/docs/postgres/replication/configure-logical-replication#configure-your-postgresql-instance) . Customise your instance on setup You can set flags in the "Flags" panel of the "Customise your instance" section of the [create database page](https://console.cloud.google.com/sql/instances/create;engine=PostgreSQL) in the console, when setting up your database. Be careful to connect using the "Outgoing IP address", not the "Public IP address". You will also need to create a new database user with `REPLICATION`. Log in using the default `postgres` user and then run something like this, changing the username and database name as necessary: sql CREATE ROLE electric WITH REPLICATION LOGIN PASSWORD '...'; GRANT ALL PRIVILEGES ON DATABASE "postgres" to electric; You can then connect to Postgres from Electric as that user, which you can verify using e.g.: shell docker run -it -e DATABASE_URL=postgresql://electric:YOUR_PASSWORD@YOUR_OUTGOING_IP/postgres electricsql/electric:latest #### AlloyDB [​](#alloydb) For AlloyDB, the flag to enable logical replication is called `alloydb.logical_decoding`. ### Deploy Electric [​](#deploy-electric) GCP provides a [wide range of container hosting](https://cloud.google.com/containers) . We recommend using [Containers on Compute Engine](https://cloud.google.com/compute/docs/containers/deploying-containers) or [Google Kubernetes Engine (GKE)](https://cloud.google.com/kubernetes-engine) . For example, you can deploy Electric on a [Container-Optimized OS](https://cloud.google.com/container-optimized-os/docs) with a [Persistent Disk](https://cloud.google.com/compute/docs/disks/#pdspecs) for storing Shape logs. Don't use Cloud Run We **don't recommend** that you use [Cloud Run](https://cloud.google.com/run) to deploy the Electric sync service because Cloud Run uses an in-memory filesystem and does not provide persistent file storage for Shape logs. IPv6 support If you're connecting to Postgres over IPv6, (for example, if you're [connecting to Supabase Postgres](./supabase#deploy-postgres) ) then you may need to [enable IPv6 support](/docs/guides/troubleshooting#ipv6-support) and be on a [Premium Network Tier](https://cloud.google.com/vpc/docs/subnets#ipv6-ranges) . ### Deploy your app [​](#deploy-your-app) GCP provides a range of [website hosting options](https://cloud.google.com/solutions/web-hosting?hl=en) . For example you can deploy a static app to [Google Storage](https://cloud.google.com/storage/docs/hosting-static-website) with [Cloud Build](https://cloud.google.com/build/docs/overview) . --- # Netlify - Integrations | ElectricSQL Return to top ![](/img/integrations/netlify.svg) Netlify [​](#netlify) ====================== [Netlify](https://www.netlify.com/) is an [application deployment platform](https://www.netlify.com/platform/) . Electric and Netlify [​](#electric-and-netlify) ------------------------------------------------ Netlify is a great choice for deploying client-side web apps that use Electric. Need context? See the [Deployment guide](/docs/guides/deployment) for more details. ### Deploy your app [​](#deploy-your-app) [Create your app](https://docs.netlify.com/welcome/add-new-site/) , connect it to Netlify and [deploy via `git push`](https://docs.netlify.com/site-deploys/create-deploys/#deploy-with-git) . ### Connect to Electric [​](#connect-to-electric) You need Electric (and Postgres) running somewhere else The easiest way is to use the [Electric Cloud](/product/cloud) . Or see the [Deployment guide](/docs/guides/deployment) . Copy the URL to your Electric instance and use it when [syncing data](/docs/api/clients/typescript#shape) into your app. E.g.: by [setting an environment variable](https://docs.netlify.com/environment-variables/get-started/#site-environment-variables) and using it in your code: tsx const ELECTRIC_URL = process.env.ELECTRIC_URL const stream = new ShapeStream({ url: `${ELECTRIC_URL}/v1/shape`, params: { table: 'items' } }) See the [Client docs](/docs/api/clients/typescript) for more information. Example [​](#example) ---------------------- ### Deploy example app [​](#deploy-example-app) Deploy our [standalone-basic-example](https://github.com/electric-sql/standalone-basic-example) app using the form below: `ELECTRIC_URL`: required, must start with `https://`. [![](https://www.netlify.com/img/deploy/button.svg)](https://app.netlify.com/start/deploy?repository=https://github.com/electric-sql/standalone-basic-example#VITE_ELECTRIC_URL=undefined) --- # Render - Integrations | ElectricSQL Return to top ![](/img/integrations/render.svg) Render [​](#render) ==================== [Render](https://render.com) is a cloud infrastructure and web hosting platform. Electric and Render [​](#electric-and-render) ---------------------------------------------- You can use Render to deploy [an Electric sync service](#deploy-electric) and [your client application](#deploy-your-app) . Postgres on Render and logical replication Render does provide [managed Postgres hosting](https://docs.render.com/postgresql) . However, this [doesn't yet](https://feedback.render.com/features/p/allow-for-postgres-logical-replication) support logical replication, so you can't currently use Electric with it. If you need Postgres hosting to use with Render, [Neon](./neon) and [Supabase](./supabase) both work great. Need context? See the [Deployment guide](/docs/guides/deployment) for more details. ### Deploy Electric [​](#deploy-electric) Deploy Electric as a [Web Service](https://docs.render.com/web-services) using their [deploy from a container registry](https://docs.render.com/web-services#deploy-from-a-container-registry) option. In the Render dashboard, create a new Web Service, select Existing Image and paste `electricsql/electric` as the image URL. Then on the next screen set a `DATABASE_URL` and [any other config](/docs/api/config) as environment variables. You can also optionally enter `/v1/health` as the path for a health check. Under "Advanced" make sure you add a Persistent Disk and set the Mount path to e.g.: `/var/electric`. Then also set the [`ELECTRIC_STORAGE_DIR` environment variable](/docs/api/config#storage-dir) to the same mount path, e.g.: `ELECTRIC_STORAGE_DIR=/var/electric`. ### Deploy your app [​](#deploy-your-app) You can deploy your app on Render as a [Static Site](https://docs.render.com/static-sites) . For example, you can deploy our [standalone-basic-example](https://github.com/electric-sql/standalone-basic-example) by: * selecting "Public GitHub Repository" and pasting `https://github.com/electric-sql/standalone-basic-example` as the value * setting the publish directory to `dist` * setting a `VITE_ELECTRIC_URL` environment variable to the URL of your Electric web service, such as `https://YOUR_WEB_SERVICE_NAME.onrender.com` Then make sure that your Postgres database has an `items` table with an `id` column and insert some data into it. Example [​](#example) ---------------------- Render supports [Blueprints](https://docs.render.com/infrastructure-as-code) to deploy infrastructure as code. The following example shows how to deploy Electric and an example web app that connects to it. Requires an existing Postgres running somewhere else The Blueprint above requires a `DATABASE_URL` to an existing Postgres database hosted somewhere else. Also, as per [the example above](#deploy-your-app) , the example app it deploys assumes you have an `items` table in your database. ### `render.yaml` Blueprint [​](#render-yaml-blueprint) Clone [github.com/electric-sql/render-blueprint](https://github.com/electric-sql/render-blueprint) or copy the following config into a `render.yaml` file: yaml services: - type: web runtime: image name: electric image: url: electricsql/electric:latest disk: name: storage mountPath: /var/electric sizeGB: 20 envVars: - key: DATABASE_URL sync: false - key: ELECTRIC_STORAGE_DIR value: "/var/electric" - type: web runtime: static name: app buildCommand: VITE_ELECTRIC_URL="https://${ELECTRIC_HOST}.onrender.com" npm run build staticPublishPath: ./dist envVars: - key: ELECTRIC_HOST fromService: name: electric type: web property: host You can then follow [the instructions here](https://docs.render.com/infrastructure-as-code#setup) to deploy the Blueprint on Render. In short, you push the `render.yaml` to a repo, open the [Render Dashboard](https://dashboard.render.com/) , click "New > Blueprint", connect the repo and enter your `DATABASE_URL` when prompted. --- # Supabase - Integrations | ElectricSQL Return to top ![](/img/integrations/supabase.svg) Supabase [​](#supabase) ======================== [Supabase](https://supabase.com) is a Postgres hosting and backend-as-a-service platform for building web, mobile and AI applications. Electric and Supabase [​](#electric-and-supabase) -------------------------------------------------- You can use Electric on Supabase's [hosted Postgres](#deploy-postgres) . You can also use Electric to [sync data into Supabase Edge Functions](#sync-into-edge-function) . Need context? See the [Deployment guide](/docs/guides/deployment) for more details. ### Deploy Postgres [​](#deploy-postgres) [Supabase Postgres databases](https://supabase.com/docs/guides/database/overview) come with logical replication enabled and the necessary permissions for Electric to work. Create a database on [Supabase.com](https://supabase.com) . Click the "Connect" button in the top right to get the connection string. Make sure you untick the "Display connection pooler" option to get the direct access URL, because the pooled URL does not support logical replication. Note that this direct access URL only works with IPv6, which means you will need to [configure Electric to connect over IPv6](#troubleshooting-ipv6) . ### Connect Electric [​](#connect-electric) Configure Electric to connect to the direct access `DATABASE_URL` you copied above. Set [`ELECTRIC_DATABASE_USE_IPV6`](/docs/api/config#database-use-ipv6) to `true`, e.g.: shell docker run -it \ -e "DATABASE_URL=postgresql://postgres:[YOUR_PASSWORD]@db.[YOUR_PROJECT_ID].supabase.co:5432/postgres" \ -e "ELECTRIC_DATABASE_USE_IPV6=true" \ -p 3000:3000 \ electricsql/electric:latest #### Troubleshooting IPv6 [​](#troubleshooting-ipv6) When connecting to a Supabase Postgres, you either need to make sure Electric and its network supports IPv6, or you need to be on a Pro or Team plan with Supabase Platform to enable their IPv4 add-on. See the [troubleshooting guide on IPv6](/docs/guides/troubleshooting#ipv6-support) for tips on enabling IPv6 support for Electric. Or see [this Supabase guide](https://supabase.com/docs/guides/platform/ipv4-address#enabling-the-add-on) for information about enabling their IPv4 add-on. Need somewhere to host Electric? If you need to deploy Electric, then [Supabase works great](https://supabase.com/blog/postgres-on-fly-by-supabase) with [Fly.io](./fly#deploy-electric) . ### Sync into Edge Function [​](#sync-into-edge-function) You can also use Electric to sync data into a Supabase [Edge Function](https://supabase.com/docs/guides/functions) . Install the [Supabase CLI](https://supabase.com/docs/guides/local-development/cli/getting-started) and follow the steps in [this Quickstart](https://supabase.com/docs/guides/functions/quickstart) to initialise a new project and create an edge function, e.g.: shell supabase init supabase functions new hello-electric Start Supabase and serve the functions locally: shell supabase start supabase functions serve Run `tail` to see the `curl` command at the bottom of the generated `supabase/functions/hello-electric/index.ts` file: shell tail supabase/functions/hello-electric/index.ts Copy the `curl` command (with the real value for `[YOUR_ANON_KEY]`) and run it once against the default function implementation: console $ curl -i --location --request POST 'http://127.0.0.1:54321/functions/v1/hello-electric' \ --header 'Authorization: Bearer [YOUR_ANON_KEY]' \ --header 'Content-Type: application/json' \ --data '{"name":"Functions"}' ... {"message":"Hello Functions!"} Now, replace the contents of `supabase/functions/hello-electric/index.ts` with the following, replacing `[YOUR_ELECTRIC_URL]` with the URL of an Electric service, running against a Postgres database with an `items` table. (This can be `http://localhost:3000` if you're running the local docker command we [used above](#connect-electric) when connecting Electric to Supabase Postgres). ts import { Shape, ShapeStream } from 'npm:@electric-sql/client' Deno.serve(async (req) => { const stream = new ShapeStream({ url: '[YOUR_ELECTRIC_URL]/v1/shape', params: { table: 'items' } }) const shape = new Shape(stream) const items = [...await shape.value] return new Response( JSON.stringify(items), { headers: { "Content-Type": "application/json" } }, ) }) Save it, wait a second and then run the same `curl` command you just ran before to make a request to the edge function. You should see the data from your `items` table in the HTTP response, e.g.: console $ curl -i --location --request POST 'http://127.0.0.1:54321/functions/v1/hello-electric' \ --header 'Authorization: Bearer [YOUR_ANON_KEY]' \ --header 'Content-Type: application/json' \ --data '{"name":"Functions"}' ... [["\"public\".\"items\"/\"69ad0c7c-7a84-48e8-84fc-d92e5bd5e2f4\"", ...]\ \ PGlite [​](#pglite)\ \ --------------------\ \ Electric and Supabase are also collaborating to develop [PGlite](/product/pglite)\ , which Supabase sponsor, contribute to and have developed [database.build](https://database.build)\ on. --- # Alternatives | ElectricSQL Return to top Alternatives [​](#alternatives) ================================ There are many systems for syncing data and building local-first applications. Add a project to this page If you'd like to suggest a project to add to this page, please feel free to [submit a pull‑request](https://github.com/electric-sql/electric/edit/main/website/docs/reference/alternatives.md) . Alternative projects [​](#alternative-projects) ------------------------------------------------ We list a selection of projects below, including for example: * [InstantDB](https://www.instantdb.com) * [Jazz](https://jazz.tools) * [PowerSync](https://www.powersync.co) * [Turso](https://turso.tech) * [Zero](https://zerosync.dev) ### Real-time and sync [​](#real-time-and-sync) [Electric](/product/electric) is a sync-engine. It syncs subsets of data from Postgres, in real-time, into local apps and services. Other real-time streaming systems and sync engines include: * [Ably](https://ably.com) * [Ampli-sync](https://ampliapps.com/sqlite-sync) * [Convex](https://www.convex.dev) * [Debezium](https://debezium.io/) * [EDB BDR](https://www.enterprisedb.com/docs/pgd/4/bdr) * [Fanout](https://www.fastly.com/products/fanout) * [Feldera](https://www.feldera.com) * [Firebase](https://firebase.google.com) * [Hevo](https://hevodata.com/) * [Litestream](https://litestream.io) * [Liveblocks](https://liveblocks.io) * [Materialize](https://materialize.com) * [Mycelial](https://mycelial.com) * [PartyKit](https://partykit.io) * [pgEdge](https://www.pgedge.com) * [PowerSync](https://www.powersync.co) * [Pusher](https://pusher.com) * [Prisma Pulse](https://www.prisma.io/data-platform/pulse) * [RisingWave](https://risingwave.com) * [Qlik](https://www.qlik.com/us/products/qlik-data-streaming-cdc) * [Sequin](https://sequinstream.com) * [SKDB](https://skdb.io) * [SQLedge](https://github.com/zknill/sqledge) * [SQLSync](https://github.com/orbitinghail/sqlsync) * [Supabase Realtime](https://supabase.com/docs/guides/realtime) * [Turso](https://turso.tech) * [Y-Sweet by Jamsocket](https://jamsocket.com/y-sweet) ### Embedded databases [​](#embedded-databases) [PGlite](/product/pglite) is an embedded database. Other embedded databases include: * [CozoDB](https://www.cozodb.org) * [DuckDB](https://duckdb.org) * [libSQL](https://turso.tech/libsql) * [SQlite](https://www.sqlite.org) * [Tonbo](https://github.com/tonbo-io/tonbo) ### Local-first [​](#local-first) Electric and PGlite are _components_ that can be composed, with other tools, into a local-first stack. Other local-first libraries and frameworks include: * [Automerge](https://automerge.org) * [Ditto](https://ditto.live) * [DXOS](https://dxos.org) * [Evolu](https://github.com/evoluhq/evolu) * [Fireproof](https://fireproof.storage) * [Homebase](https://homebase.io) * [InstantDB](https://www.instantdb.com) * [Jazz](https://jazz.tools) * [Kinto](https://kinto-storage.org) * [LiveStore](https://github.com/livestorejs) * [Pocketbase](https://pocketbase.io) * [Pouch](https://pouchdb.com) * [remoteStorage.js](https://remotestorage.io) * [Replicache](https://replicache.dev) * [RxDB](https://rxdb.info) * [Socket](https://socketsupply.co) * [sqlite\_crdt](https://github.com/cachapa/sqlite_crdt) * [Synql](https://github.com/coast-team/synql) * [TinyBase](https://tinybase.org) * [Triplit](https://www.triplit.dev) * [Verdant](https://github.com/a-type/verdant) * [Vlcn](https://vlcn.io) / [cr-sqlite](https://github.com/vlcn-io/cr-sqlite) * [Watermelon](https://nozbe.github.io/WatermelonDB) * [Yjs](https://yjs.dev) * [Y-Sweet by Jamsocket](https://jamsocket.com/y-sweet) * [Zero](https://zerosync.dev) You can also find out more about local-first software development and discover other projects from a range of communities, including: * [lofi.software](https://lofi.software) * [localfirst.fm](https://www.localfirst.fm) * [Local-first Conf](https://www.localfirstconf.com) (_disclaimer: we're a co-organiser_) * [crdt.tech](https://crdt.tech) ### State transfer [​](#state-transfer) Other systems for managing state transfer: * [Apollo (GraphQL)](https://www.apollographql.com) * [Relay (GraphQL)](https://relay.dev) * [tRPC](https://trpc.io) ### Postgres APIs [​](#postgres-apis) Other tools for exposing data from Postgres: * [Hasura](https://hasura.io) * [PostGraphile](https://www.graphile.org/postgraphile) * [PostgREST](https://postgrest.org/en/stable) ### Interesting projects [​](#interesting-projects) Other interesting projects include: * [AntidoteDB](https://www.antidotedb.eu) * [Beehive](https://www.inkandswitch.com/beehive) * [Cambria](https://www.inkandswitch.com/cambria) * [CockroachDB](https://www.cockroachlabs.com/product) * [Concordant](https://github.com/concordant) * [Dat-ecosystem](https://blog.dat-ecosystem.org/staying-connected) * [Declarative Dataflow](https://github.com/comnik/declarative-dataflow) * [Dqlite](https://dqlite.io) * [Electric Clojure](https://github.com/hyperfiddle/electric) * [Fauna](https://fauna.com) * [Hydro Project](https://hydro.run/research) * [Ink & Switch](https://www.inkandswitch.com) * [IPFS](https://ipfs.tech) * [Macrometa](https://www.macrometa.com) * [MotherDuck](https://motherduck.com) * [RainbowFS](https://rainbowfs.lip6.fr) * [rqlite](https://github.com/rqlite/rqlite) * [SOLID](https://solidproject.org) * [Source](https://source.network) * [Spanner](https://cloud.google.com/spanner/) * [SyncFree](https://pages.lip6.fr/syncfree/index.php/crdt-resources.html) --- # Benchmarks - Reference | ElectricSQL Return to top Benchmarks [​](#benchmarks) ============================ We run benchmarks for [cloud](#cloud) , [core Electric](#electric) and [PGlite](#pglite) . Understanding the benchmarks [​](#understanding-the-benchmarks) ---------------------------------------------------------------- Electric is designed to be simple, scalable and low-cost to run. This means that it should handle a high number of concurrent users, with high read and write workloads, corresponding to high throughput of data from Postgres out to many clients, through various shape topologies, for a variety of workloads. Our goals are to support millions of concurrent users, hundreds of thousands of shapes and high read and write throughput from a single Electric instance, with minimal impact on the source Postgres. With low, stable and predictable compute and memory usage. In order to work towards this, we use benchmarks internally to measure and inform work to improve performance. This page lists a selection of these benchmarks, to give you an indication of: 1. the types of benchmarks that the Electric team is running 2. the performance levels that we're seeing for our internal benchmark runs Benchmarks are always highly workload, version and hardware dependent. These benchmarks are **not in any way guaranteed to be representative of the performance numbers that you will see when running Electric on your own hardware**. You **must** test Electric yourself, with a representative workload, on your own infrastructure. ### Running yourself [​](#running-yourself) We are in the process of open sourcing our [electric-sql/benchmarking-fleet](https://github.com/electric-sql/benchmarking-fleet) repo. When public, you will be able to follow the instructions there to run the benchmarks yourself. ### Continuous integration [​](#continuous-integration) We are working to set up benchmarks to run on every release (patch, minor and major). When this is done, we will document how to see the release benchmarks and how to track improvements and/or regression in performance. Cloud [​](#cloud) ------------------ Electric is designed to run behind a CDN, using the CDN's [request collapsing](/docs/api/http#request-collapsing) capability to scale out data delivery to lots of concurrent users. The graph below shows the latency and compute resource of a single Electric server using this technique to handle between 100k and 1 million concurrent users, with a write workload of 960 transactions per minute: These statistics were generated using our [client load benchmarking](https://github.com/electric-sql/client-load-benchmarking) suite that allows for measuring (a) client latencies and (b) sync service resource use for any combination of concurrent connected clients and database workload. Electric [​](#electric) ------------------------ The first two benchmarks measure a client's initial sync time: 1. [many concurrent clients syncing a small shape](#_1-many-concurrent-clients-syncing-a-small-shape) 2. [a single client syncing a large shape](#_2-a-single-client-syncing-a-large-shape) The next four measure how long it takes for clients to recieve an update after a write: 3. [many independent shapes](#_3-many-independent-shapes) 4. [one shape with many clients](#_4-one-shape-with-many-clients) 5. [many overlapping shapes, each with a single client](#_5-many-overlapping-shapes-each-with-a-single-client) 6. [many overlapping shapes, one client](#_6-many-overlapping-shapes-one-client) The last two benchmarks measure how long it takes Electric to process a write: 7. [write throughput with optimised where clauses](#_7-write-throughput-with-optimised-where-clauses) 8. [write throughput with non-optimised where clauses](#_8-write-throughput-with-non-optimised-where-clauses) ### Initial sync [​](#initial-sync) #### 1\. Many concurrent clients syncing a small shape [​](#_1-many-concurrent-clients-syncing-a-small-shape) [![Benchmark measuring many concurrent clients syncing a small shape](/assets/concurrent-shape-creation.x63ApwIH.png)](/assets/concurrent-shape-creation.x63ApwIH.png) This measures the memory use and the time to sync all the data into all the clients for an increasing number of concurrent clients performing an initial sync of a 500 row single shape. The results show stable memory use with time to sync all data rising roughly linearly up to 2,000 concurrent clients. #### 2\. A single client syncing a large shape [​](#_2-a-single-client-syncing-a-large-shape) [![Benchmark measuring a single client syncing an increasingly large shape](/assets/single-shape-single-client.DetANT0S.png)](/assets/single-shape-single-client.DetANT0S.png) This measures a single client syncing a single large shape of up-to 1M rows. The sync time is linear, the memory is stable. ### Live updates [​](#live-updates) #### 3\. Many independent shapes [​](#_3-many-independent-shapes) [![Benchmark measuring how long a write that affects a single shape takes to reach a client](/assets/unrelated-shapes-one-client-latency.y69acmMm.png)](/assets/unrelated-shapes-one-client-latency.y69acmMm.png) This benchmark evaluates the time it takes for a write operation to reach a client subscribed to the relevant shape. On the x-axis, the number of active shapes is shown. Each shape in this benchmark is independent, ensuring that a write operation affects only one shape at a time. The two graphs differ based on the type of where clause used for the shapes: * **Top Graph:** The where clause is in the form `field = constant`, where each shape is assigned a unique constant. These types of where clause, along with [other patterns](/docs/guides/shapes#optimised-where-clauses) , are optimised for high performance regardless of the number of shapes — analogous to having an index on the field. As shown in the graph, the latency remains consistently flat at 6ms as the number of shapes increases. This 6ms latency includes 3ms for PostgreSQL to process the write operation and 3ms for Electric to propagate it. We are actively working to optimise additional where clause types in the future. * **Bottom Graph:** The where clause is in the form `field ILIKE constant`, an example of a non-optimised query type. In this case, the latency increases linearly with the number of shapes because Electric must evaluate each shape individually to determine if it is affected by the write. Despite this, the response times remain low, a tenth of a second for 10,000 shapes. #### 4\. One shape with many clients [​](#_4-one-shape-with-many-clients) [![Benchmark measuring write fanout into to one shape with many clients](/assets/write-fanout.BG7cFjdb.png)](/assets/write-fanout.BG7cFjdb.png) Measures write latency (i.e.: time for the client to see the write) for a transaction of increasing size written to one shape log, streamed to an increasing number of clients. Below is the memory use for the same benchmark. [![Benchmark measuring memory use for write fanout into one shape with many clients](/assets/write-fanout-memory.DUjYFv-Y.png)](/assets/write-fanout-memory.DUjYFv-Y.png) #### 5\. Many overlapping shapes, each with a single client [​](#_5-many-overlapping-shapes-each-with-a-single-client) [![Benchmark measuring write fanout into many shapes, each with a single client](/assets/diverse-shape-fanout.BfkkPpIj.png)](/assets/diverse-shape-fanout.BfkkPpIj.png) In this benchmark there are a varying number of shapes with each shape having a single client subscribed to it. It shows the average length of time it takes for a single write that affects all the shapes to reach each client. Latency and memory use rises linearly. #### 6\. Many overlapping shapes, one client [​](#_6-many-overlapping-shapes-one-client) [![Benchmark measuring write fanout into many shapes, all streamed to the same client](/assets/many-shapes-one-client.FU0fUiBT.png)](/assets/many-shapes-one-client.FU0fUiBT.png) In this benchmark there are a varying number of shapes with just one client subscribed to one of the shapes. It shows the length of time it takes for a single write that affects all the shapes to reach the client. Latency and peak memory use rises linearly. Average memory use is flat. #### 7\. Write throughput with optimised where clauses [​](#_7-write-throughput-with-optimised-where-clauses) [![Benchmark measuring how many writes per second Electric can process](/assets/replication-throughput-optimised.DRH6Xics.png)](/assets/replication-throughput-optimised.DRH6Xics.png) This benchmark measures how long each write takes to process with a varying number of shapes. Each shape in this benchmark is using an optimised where clause, specifically `field = constant`. Optimised where clauses When you create a shape, you can specify a where clause that filters the rows that the shape is interested in. In Electric, we filter the changes we receive from Postgres so that each shape only receives changes that affect the rows it is interested in. If there are lots of shapes, this could mean we have to evaluate lots of where clauses for each write, however we have optimised this process so that we can evaluate millions of where clauses at once, providing the where clauses follow various patterns, which we call optimised where clauses. `field = constant` is one of the patterns we optimise, we can evaluate millions of these where clauses at once by indexing the shapes based on the constant value for each shape. This index is internal to Electric, and nothing to do with Postgres indexes. It's a hashmap if you're interested. `field = const AND another_condition` is another pattern we optimise. We aim to optimise a large subset of Postgres where clauses in the future. Optimised where clauses mean that we can process writes in a quarter of a millisecond, regardless of how many shapes there are. For more information on optimised where clauses, see the [shape API](/docs/guides/shapes#optimised-where-clauses) . The top graph shows throughput for Postgres 14, the bottom graph for Postgres 15. The green line shows how fast we process writes that affect shapes. You can see in both graphs that throughput is flat at 0.17-0.27 milliseconds per change (4000 - 6000 row changes per second) regardless of how many shapes there are. The purple line shows how fast we ignore writes that don't affect any shapes. For Postgres 14 (top graph) this is flat at 0.02 milliseconds per change (50,000 row changes per second).For Postgres 15 (bottom graph) the throughput scales linearly with the number of shapes. This is because Postgres 15 has the ability to filter the replication stream based on a where clause, so we use this to filter out writes that don't affect any shapes. However as you can see in the graph, in this situation this is not a good optimisation! We're working on improving this, but at the moment it's kept as it's beneficial when using non-optimised where clauses (see benchmark 8). #### 8\. Write throughput with non-optimised where clauses [​](#_8-write-throughput-with-non-optimised-where-clauses) [![Benchmark measuring how many writes per second Electric can process](/assets/replication-throughput-non-optimised.XLxbPb_H.png)](/assets/replication-throughput-non-optimised.XLxbPb_H.png) This benchmark also measures how long each write takes to process with a varying number of shapes, but in this benchmark each shape is using an non-optimised where clause, specifically `field ILIKE constant`. You can see in both graphs that throughput scales linearly with the number of shapes. This is because, for non-optimised where clauses, Electric has to evaluate each shape individually to determine if it is affected by the write. The top graph shows throughput for Postgres 14. You can see throughput is the roughly the same regardless of whether the write affects shapes (green) or not (purple), 140k row changes per second per shape. The bottom graph shows throughput for Postgres 15. Postgres 15 has the ability to filter the replication stream based on a where clause, so we use this to filter out writes that don't affect any shapes. So for writes that affect shapes, we get the same 140k row changes per second per shape as Postgres 14, but for writes that don't affect shapes, we get 1400k row changes per second per shape. PGlite [​](#pglite) -------------------- PGlite benchmarks are documented at [pglite.dev/benchmarks](https://pglite.dev/benchmarks) . --- # Telemetry | ElectricSQL Return to top Telemetry [​](#telemetry) ========================== Electric provides telemetry data — such as traces, logs, and metrics — for real-time system monitoring. Self-hosted Electric instances are also configured by default to send aggregated, anonymous usage data to ElectricSQL to help us understand how our software is being used. You can opt-out of this reporting by setting an environment variable. See the [Anonymous usage data](#anonymous-usage-data) section below for more details. Metrics [​](#metrics) ---------------------- Metrics are reported in StatsD and Prometheus formats. To configure Electric to expose metric information in those formats use the following environment variables. | VARIABLE | Description | | --- | --- | | ELECTRIC\_STATSD\_HOST | The address of the StatsD server | | ELECTRIC\_PROMETHEUS\_PORT | The scrape port for Prometheus | You can get the current status of the service by calling the `http://electric-hostname:PROMETHUES_PORT/metrics` endpoint. OpenTelemetry [​](#opentelemetry) ---------------------------------- Traces are exported using the OpenTelemetry Protocol (OTLP). You can configure the OpenTelemetry Exporter for Electric using the following environment variables. | VARIABLE | Type | Description | | --- | --- | --- | | ELECTRIC\_OTLP\_ENDPOINT | `URL` | An OpenTelemetry collector endpoint url. | | ELECTRIC\_HNY\_API\_KEY | `string` | API key for exporting to Honeycomb.io. | | ELECTRIC\_HNY\_DATASET | `string` | Dataset name for Honeycomb.io. | | ELECTRIC\_OTEL\_DEBUG | `boolean` | Enable or disable debug logging of telemetry data to stdout. | Electric enables export of telemetry data when it is configured with an `ELECTRIC_OTLP_ENDPOINT`. There is builtin support for [Honeycomb.io](https://www.honeycomb.io/) : telemetry data can be exported directly to it by specifying `ELECTRIC_OTLP_ENDPOINT=https://api.honeycomb.io` and adding at least the `ELECTRIC_HNY_API_KEY` configuration option. In order to use other telemetry data collectors, you'll need to run the [OpenTelemetry Collector](https://opentelemetry.io/docs/collector/) and include the exporter of choice in its configuration file along with any required credentials, then use Collector's URL as the value for `ELECTRIC_OTLP_ENDPOINT`. Electric always adds the following resource attributes to events: elixir %{service: %{name: service_name, version: version}, instance: %{id: instance_id}} Attributes `service_name` and `instance_id` can be overridden with `ELECTRIC_SERVICE_NAME` and `ELECTRIC_INSTANCE_ID` respectively. By default, `instance_id` is an uuid. Electric will also load additional resource attributes from `OTEL_RESOURCE_ATTRIBUTES`. Learn more about resource attributes in the [OpenTelemetry documentation](https://opentelemetry.io/docs/languages/js/resources/) . Example [​](#example) ---------------------- You can find an example of a docker compose that runs Electric with an OpenTelemetry Collector agent that sends telemetry data to Honeycomb under `packages/sync-service/dev`. Set `ELECTRIC_HNY_DATASET` and `ELECTRIC_HNY_API_KEY` environment variables in a terminal session and run docker compose in it like so: shell docker compose -f docker-compose-otel.yml up Anonymous usage data [​](#anonymous-usage-data) ------------------------------------------------ Electric instances are configured by default to send anonymized usage data to checkpoint.electric-sql.com to help us understand how the software is being used. Absolutely no information about transaction contents is sent. I.e.: none of your data that you're using Electric to replicate is captured in the telemetry information or shared with the Electric checkpoint service. Captured data includes the disk usage by the shape cache, CPU/memory information about the running Electric instance, Postgres version, number of shapes, amount of distinct shape requests, and numerical information about processed transactions: byte size, amount of operations, and percentiles of response times. Essentially, what kind of load Electric was under, and how did it cope. Aggregated statistics are sent every 30 minutes. To disable anonymous usage data, set the `ELECTRIC_USAGE_REPORTING` environment variable to `false`. We encourage everyone to keep this enabled so we can get a better understanding of how Electric is used. If you have any further questions about what we collect, feel free to ask on our [open community Discord](https://discord.electric-sql.com) or reach out to us via email at [info@electric-sql.com](mailto:info@electric-sql.com) . --- # Literature | ElectricSQL Return to top Literature [​](#literature) ============================ ElectricSQL builds on decades of research into distributed database technology. Some of which was authored by [our team and advisors](/about/team) . Edit this page If you'd like to suggest a paper or an edit to make to this page, please [submit a pull‑request](https://github.com/electric-sql/electric/edit/main/website/docs/reference/alternatives.md) . Research papers [​](#research-papers) -------------------------------------- This page lists a non-exhaustive selection of papers that chart the development of some of the concepts and algorithms that ElectricSQL and [other systems](./alternatives) are based on. ### 2011 [​](#2011) * [Conflict-free Replicated Data Types (CRDTs)](https://hal.inria.fr/hal-00932836) by [Nuno Preguiça](/about/team#nuno) , Carlos Baquero and [Marc Shapiro](/about/team#marc) * [Don’t Settle for Eventual: Scalable Causal Consistency for Wide-Area Storage with COPS](https://www.cs.cmu.edu/~dga/papers/cops-sosp2011.pdf) by Wyatt Lloyd, Michael J. Freedman, Michael Kaminsky and David G. Andersen ### 2012 [​](#2012) * [Making Geo-Replicated Systems Fast as Possible, Consistent when Necessary](https://doi.org/10.5555/2387880.2387906) by Cheng Li, Daniel Porto, Allen Clement, Johannes Gehrke, [Nuno Preguiça](/about/team#nuno) and Rodrigo Rodrigues * [Calvin: Fast Distributed Transactions for Partitioned Database Systems](https://doi.org/10.1145/2213836.2213838) by Alexander Thomson, Thaddeus Diamond, Shu-Chun Weng, Kun Ren, Philip Shao and Daniel J. Abadi ### 2013 [​](#2013) * [Spanner: Google’s Globally-Distributed Database](https://doi.org/10.1145/2491245) by James C. Corbett, Jeffrey Dean, Michael Epstein, Andrew Fikes, Christopher Frost, J. J. Furman, Sanjay Ghemawat, Andrey Gubarev, Christopher Heiser, Peter Hochschild, Wilson Hsieh, Sebastian Kanthak, Eugene Kogan, Hongyi Li, Alexander Lloyd, Sergey Melnik, David Mwaura, David Nagle, Sean Quinlan, Rajesh Rao, Lindsay Rolig, Yasushi Saito, Michal Szymaniak, Christopher Taylor, Ruth Wang and Dale Woodford * [Highly Available Transactions: Virtues and Limitations](https://doi.org/10.14778/2732232.2732237) by Peter Bailis, Aaron Davidson, Alan Fekete, Ali Ghodsi, Joseph M. Hellerstein and Ion Stoica * [SwiftCloud: Fault-Tolerant Geo-Replication Integrated all the Way to the Client Machine](https://arxiv.org/abs/1310.3107) by Marek Zawirski, [Annette Bieniusa](/about/team#annette) , [Valter Balegas](/about/team#valter) , Sérgio Duarte, Carlos Baquero, [Marc Shapiro](/about/team#marc) and [Nuno Preguiça](/about/team#nuno) ### 2014 [​](#2014) * [Coordination Avoidance in Database Systems](https://arxiv.org/abs/1402.2237) by Peter Bailis, Alan Fekete†, Michael J. Franklin, Ali Ghodsi, Joseph M. Hellerstein and Ion Stoica * [Scalable Atomic Visibility with RAMP Transactions](https://doi.org/10.1145/2588555.2588562) by Peter Bailis, Alan Fekete, Ali Ghodsi, Joseph M. Hellerstein and Ion Stoica ### 2015 [​](#2015) * [Extending Eventually Consistent Cloud Databases for Enforcing Numeric Invariants](https://doi.org/10.1109/SRDS.2015.32) by [Valter Balegas](/about/team#valter) , Sérgio Duarte, Carla Ferreira, Mahsa Najafzadeh, [Nuno Preguiça](/about/team#nuno) , Rodrigo Rodrigues, [Marc Shapiro](/about/team#marc) and Diogo Serra * [Feral Concurrency Control: An Empirical Investigation of Modern Application Integrity](https://doi.org/10.1145/2723372.2737784) by Peter Bailis, Alan Fekete, Michael J. Franklin, Ali Ghodsi, Joseph M. Hellerstein and Ion Stoica ### 2016 [​](#2016) * [Cause I’m Strong Enough: Reasoning about Consistency Choices in Distributed Systems](https://doi.org/10.1145/2837614.2837625) by Alexey Gotsman, Hongseok Yang, Mahsa Najafzadeh, Carla Ferreira and [Marc Shapiro](/about/team#marc) * [The CISE Tool: Proving Weakly-Consistent Applications Correct](https://doi.org/10.1145/2911151.2911160) by Mahsa Najafzadeh, Alexey Gotsman, Hongseok Yang, Carla Ferreira and [Marc Shapiro](/about/team#marc) * [Cure: strong semantics meets high availability and low latency](https://doi.org/10.1109/ICDCS.2016.98) by Deepthi Devaki Akkoorath, Alejandro Z. Tomsic, Manuel Bravo, Zhongmiao Li, Tyler Crain, [Annette Bieniusa](/about/team#annette) , [Nuno Preguiça](/about/team#nuno) and [Marc Shapiro](/about/team#marc) * [Antidote: the highly-available geo-replicated database with strongest guarantees](https://pages.lip6.fr/syncfree/attachments/article/59/antidote-white-paper.pdf) by Deepthi Devaki Akkoorath and [Annette Bieniusa](/about/team#annette) * [BigSets: Scaling CRDTs to large sizes in Riak](https://pages.lip6.fr/syncfree/index.php/2-uncategorised/53-big-sets.html) by Russell Brown and Torben Hoffmann ### 2017 [​](#2017) * [Bringing Hybrid Consistency Closer to Programmers](https://doi.org/10.1145/3064889.3064896) by Gonçalo Marcelino, [Valter Balegas](/about/team#valter) and Carla Ferreira * [Pure Operation-Based Replicated Data Types](https://arxiv.org/abs/1710.04469) by Carlos Baquero, Paulo Sérgio Almeida and Ali Shoker ### 2018 [​](#2018) * [Just-Right Consistency: reconciling availability and safety](https://arxiv.org/abs/1801.06340) by [Marc Shapiro](/about/team#marc) , [Annette Bieniusa](/about/team#annette) , [Nuno Preguiça](/about/team#nuno) , [Valter Balegas](/about/team#valter) and Christopher Meiklejohn * [IPA: invariant-preserving applications for weakly consistent replicated databases](https://doi.org/10.14778/3297753.3297760) by [Valter Balegas](/about/team#valter) , Sérgio Duarte, Carla Ferreira, Rodrigo Rodrigues and [Nuno Preguiça](/about/team#nuno) * [Delta State Replicated Data Types](https://doi.org/10.1016/j.jpdc.2017.08.003) by Paulo Sérgio Almeida, Ali Shoker and Carlos Baquero * [Anna: A KVS For Any Scale](https://dl.acm.org/doi/10.1109/TKDE.2019.2898401) by Chenggang Wu, Jose M. Faleiro, Yihan Lin and Joseph M. Hellerstein * [Interactive Checks for Coordination Avoidance](https://doi.org/10.14778/3275536.3275538) by Michael Whittaker and Joseph M. Hellerstein * [ACGreGate: A Framework for Practical Access Control for Applications using Weakly Consistent Databases](https://arxiv.org/abs/1801.07005) by Mathias Weber and [Annette Bieniusa](/about/team#annette) ### 2019 [​](#2019) * [CAnDoR: Consistency Aware Dynamic data Replication](https://hal.inria.fr/hal-02274165) by Etienne Mauffret, Flavien Vernier and Sébastien Monnet * [A Generic Replicated Data Type for Strong Eventual Consistency](https://doi.org/10.1145/3301419.3323974) by [Kevin De Porre](/about/team#kevin) , Florian Myter, Christophe De Troyer, Christophe Scholliers, Wolfgang De Meuter and Elisa Gonzalez Boix * [Keeping CALM: When Distributed Consistency is Easy](https://arxiv.org/abs/1901.01930) by Joseph M. Hellerstein and Peter Alvaro * [Invariant Safety for Distributed Applications"](https://doi.org/10.1145/3301419.3323970) by Sreeja Nair, Gustavo Petri and [Marc Shapiro](/about/team#marc) * [LightKone Reference Architecture (LiRA)](https://www.lightkone.eu) by Ali Shoker, Paulo Sergio Almeida, Carlos Baquero, [Annette Bieniusa](/about/team#annette) , Roger Pueyo Centelles, Pedro Akos Costa, Vitor Enes, Carla Ferreira, Pedro Fouto, Felix Freitag, Bradley King, Igor Kopestenski, Giorgos Kostopoulos, João Leitão, Adam Lindberg, Albert van der Linde, Sreeja Nair, [Nuno Preguiça](/about/team#nuno) , Mennan Selimi, [Marc Shapiro](/about/team#marc) , Peer Stritzinger, Ilyas Toumlilt, Peter Van Roy, Dimitrios Vasilas, Georges Younes, Igor Zavalyshyn and Peter Zeller * [CDB: Geo-Replicated, Conflict-Free Document Database with Session Guarantees](https://www.macrometa.com/global-data-network) by Chetan Venkatesh, Durga Gokina and Christopher S. Meiklejohn * [A Tour of Gallifrey, a Language for Geodistributed Programming](https://doi.org/10.4230/LIPIcs.SNAPL.2019.11) by Mae Milano, Rolph Recto, Tom Magrino and Andrew C. Myers * [Local-First Software: You Own Your Data, in spite of the Cloud](https://www.inkandswitch.com/local-first/static/local-first.pdf) by Martin Kleppmann, [Adam Wiggins](/about/team#adam) , [Peter van Hardenberg](/about/team#pvh) and Mark McGranaghan ### 2020 [​](#2020) * [Specification of a Transactionally and Causally-Consistent (TCC) database](https://hal.inria.fr/hal-02902474v1) by Saalik Hatia and [Marc Shapiro](/about/team#marc) * [CScript: A distributed programming language for building mixed-consistency applications](https://soft.vub.ac.be/~kdeporre/) by [Kevin De Porre](/about/team#kevin) , Florian Myter, Christophe Scholliers and Elisa Gonzalez Boix * [Cloudburst: Stateful Functions-as-a-Service](https://doi.org/10.14778/3407790.3407836) by Vikram Sreekanti, Chenggang Wu, Xiayue Charles Lin, Johann Schleier-Smith, Joseph E. Gonzalez, Joseph M. Hellerstein and Alexey Tumanov * [Transactional Causal Consistency for Serverless Computing](https://doi.org/10.1145/3318464.3389710) by Chenggang Wu, Vikram Sreekanti and Joseph M. Hellerstein * [Conflict-Free Replicated Relations for Multi-Synchronous Database Management at Edge](https://hal.inria.fr/hal-02983557) by Weihai Yu and Claudia-Lavinia Ignat ### 2021 [​](#2021) * [Advanced Domain-Driven Design for Consistency in Distributed Data-Intensive Systems](https://doi.org/10.1145/3447865.3457969) by Susanne Braun, [Annette Bieniusa](/about/team#annette) and Frank Elberzhager * [Tackling Consistency-related Design Challenges of Distributed Data-Intensive Systems - An Action Research Study](https://doi.org/10.1145/3475716.3475771) by Susanne Braun, Stefan Deßloch, Eberhard Wolff, Frank Elberzhager and Andreas Jedlitschka * [ECROs: Building Global Scale Systems from Sequential Code](https://doi.org/10.1145/3485484) by [Kevin De Porre](/about/team#kevin) , Carla Ferreira, [Nuno Preguiça](/about/team#nuno) and Elisa Gonzalez * [It’s about Thyme: On the design and implementation of a time-aware reactive storage system for pervasive edge computing](https://doi.org/10.1016/j.future.2020.12.008) by João A. Silva, Filipe Cerqueira, Hervé Paulino, João M. Lourenço, João Leitão and [Nuno Preguiça](/about/team#nuno) * [Thespis: Causally-consistent OLTP](https://doi.org/0.15439/2021F34) by Joseph G. Vella and Vitezslav Nezval * [AUTOGR: automated geo-replication with fast system performance and preserved application semantics](https://doi.org/10.14778/3461535.3461541) by Jiawei Wang, Cheng Li, Kai Ma, Jingze Huo, Feng Yan, Xinyu Feng and Yinlong Xu * [New Directions in Cloud Programming](https://arxiv.org/abs/2101.01159) by Alvin Cheung, Natacha Crooks, Joseph M. Hellerstein and Mae Milano * [Highly-Available and Consistent Group Collaboration at the Edge with Colony](https://dl.acm.org/doi/abs/10.1145/3464298.3493405) by Ilyas Toumlilt, Pierre Sutra and [Marc Shapiro](/about/team#marc) * [Towards a General Database Management System of Conflict-Free Repli- cated Relations](https://munin.uit.no/bitstream/handle/10037/22344/thesis.pdf) by Iver Toft Tomter ### 2022 [​](#2022) * [Building data-centric apps with a reactive relational database](https://riffle.systems/essays/prelude) by Nicholas Schiefer, Geoffrey Litt, [Johannes Schickling](/about/team#johannes) and Daniel Jackson * [VeriFx: Correct Replicated Data Types for the Masses](https://arxiv.org/pdf/2207.02502.pdf) by [Kevin De Porre](/about/team#kevin) , Carla Ferreira and Elisa Gonzalez Boix ---