# Table of Contents - [Overview - Pipecat](#overview-pipecat) - [Guides - Pipecat](#guides-pipecat) - [Server API Reference - Pipecat](#server-api-reference-pipecat) - [Client SDKs - Pipecat](#client-sdks-pipecat) - [Use Cases - Pipecat](#use-cases-pipecat) - [Core Concepts - Pipecat](#core-concepts-pipecat) - [Installation & Setup - Pipecat](#installation-setup-pipecat) - [Quickstart - Pipecat](#quickstart-pipecat) - [Supported Services - Pipecat](#supported-services-pipecat) - [Building With OpenAI Audio Models and APIs - Pipecat](#building-with-openai-audio-models-and-apis-pipecat) - [Building with Gemini Multimodal Live - Pipecat](#building-with-gemini-multimodal-live-pipecat) - [SDK Introduction - Pipecat](#sdk-introduction-pipecat) - [Metrics - Pipecat](#metrics-pipecat) - [SDK Introduction - Pipecat](#sdk-introduction-pipecat) - [SDK Introduction - Pipecat](#sdk-introduction-pipecat) - [UserIdleProcessor - Pipecat](#useridleprocessor-pipecat) - [TranscriptProcessor - Pipecat](#transcriptprocessor-pipecat) - [Function Calling - Pipecat](#function-calling-pipecat) - [API Reference - Pipecat](#api-reference-pipecat) - [Noise cancellation with Krisp - Pipecat](#noise-cancellation-with-krisp-pipecat) - [SDK Introduction - Pipecat](#sdk-introduction-pipecat) - [SDK Introduction - Pipecat](#sdk-introduction-pipecat) - [Overview - Pipecat](#overview-pipecat) - [Pipecat Flows - Pipecat](#pipecat-flows-pipecat) - [SDK Introduction - Pipecat](#sdk-introduction-pipecat) - [Dial-in: WebRTC (Daily) - Pipecat](#dial-in-webrtc-daily-pipecat) - [PipelineParams - Pipecat](#pipelineparams-pipecat) - [PipelineTask - Pipecat](#pipelinetask-pipecat) - [Pipeline Idle Detection - Pipecat](#pipeline-idle-detection-pipecat) - [Speech Service - Pipecat](#speech-service-pipecat) --- # Overview - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Get Started Overview [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) Pipecat is an open source Python framework that handles the complex orchestration of AI services, network transport, audio processing, and multimodal interactions. “Multimodal” means you can use any combination of audio, video, images, and/or text in your interactions. And “real-time” means that things are happening quickly enough that it feels conversational—a “back-and-forth” with a bot, not submitting a query and waiting for results. [​](#what-you-can-build) What You Can Build ---------------------------------------------- [Voice Assistants\ ----------------\ \ Natural, real-time conversations with AI using speech recognition and synthesis](https://github.com/pipecat-ai/pipecat/tree/main/examples/simple-chatbot) [Interactive Agents\ ------------------\ \ Personal coaches and meeting assistants that can understand context and provide guidance](https://github.com/pipecat-ai/pipecat/tree/main/examples/studypal) [Multimodal Apps\ ---------------\ \ Applications that combine voice, video, images, and text for rich interactions](https://github.com/pipecat-ai/pipecat/tree/main/examples/moondream-chatbot) [Creative Tools\ --------------\ \ Storytelling experiences and social companions that engage users](https://github.com/pipecat-ai/pipecat/tree/main/examples/storytelling-chatbot) [Business Solutions\ ------------------\ \ Customer intake flows and support bots for automated business processes](https://github.com/pipecat-ai/pipecat/tree/main/examples/patient-intake) [Complex Flows\ -------------\ \ Structured conversations using Pipecat Flows for managing complex interactions](https://github.com/pipecat-ai/pipecat-flows) [​](#how-it-works) How It Works ---------------------------------- The flow of interactions in a Pipecat application is typically straightforward: 1. The bot says something 2. The user says something 3. The bot says something 4. The user says something This continues until the conversation naturally ends. While this flow seems simple, making it feel natural requires sophisticated real-time processing. ### [​](#real-time-processing) Real-time Processing Pipecat’s pipeline architecture handles both simple voice interactions and complex multimodal processing. Let’s look at how data flows through the system: * Voice app * Multimodal app 1 Send Audio Transmit and capture streamed audio from the user 2 Transcribe Speech Convert speech to text as the user is talking 3 Process with LLM Generate responses using a large language model 4 Convert to Speech Transform text responses into natural speech 5 Play Audio Stream the audio response back to the user In both cases, Pipecat: * Processes responses as they stream in * Handles multiple input/output modalities concurrently * Manages resource allocation and synchronization * Coordinates parallel processing tasks This architecture creates fluid, natural interactions without noticeable delays, whether you’re building a simple voice assistant or a complex multimodal application. Pipecat’s pipeline architecture is particularly valuable for managing the complexity of real-time, multimodal interactions, ensuring smooth data flow and proper synchronization regardless of the input/output types involved. Pipecat handles all this complexity for you, letting you focus on building your application rather than managing the underlying infrastructure. [​](#next-steps) Next Steps ------------------------------ Ready to build your first Pipecat application? [Installation & Setup\ --------------------\ \ Prepare your environment and install required dependencies](/getting-started/installation) [Quickstart\ ----------\ \ Build and run your first Pipecat application](/getting-started/quickstart) [Core Concepts\ -------------\ \ Learn about pipelines, frames, and real-time processing](/getting-started/core-concepts) [Use Cases\ ---------\ \ Explore example implementations and patterns](/getting-started/use-cases) [​](#join-our-community) Join Our Community ---------------------------------------------- Need help or want to share your project? Join our [Discord community](https://discord.gg/pipecat) where you can connect with other developers and get support from the Pipecat team. [Installation & Setup](/getting-started/installation) On this page * [What You Can Build](#what-you-can-build) * [How It Works](#how-it-works) * [Real-time Processing](#real-time-processing) * [Next Steps](#next-steps) * [Join Our Community](#join-our-community) --- # Guides - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Guides [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) These guides cover key aspects of building and deploying Pipecat applications. Choose a guide based on what you want to accomplish: [​](#features) Features -------------------------- [OpenAI Audio Models and APIs\ ----------------------------\ \ Build voice agents with OpenAI audio models](/guides/features/openai-audio-models-and-apis) [Gemini Multimodal Live\ ----------------------\ \ Build real-time AI chatbots with Gemini](/guides/features/gemini-multimodal-live) [Function Calling\ ----------------\ \ Implement custom functions in your bot](/guides/features/function-calling) [Metrics & Monitoring\ --------------------\ \ Track and monitor your application](/guides/features/metrics) [Noise Reduction\ ---------------\ \ Improve audio quality with Krisp](/guides/features/krisp) [Pipecat Flows\ -------------\ \ Build structured conversation flows](/guides/features/pipecat-flows) [​](#telephony) Telephony ---------------------------- [Overview\ --------\ \ Introduction to voice and telephony features](/guides/telephony/overview) [WebRTC with Daily\ -----------------\ \ Implement dial-in using Daily’s WebRTC](/guides/telephony/daily-webrtc) [Twilio + Daily Integration\ --------------------------\ \ Combine Twilio and Daily for advanced telephony](/guides/telephony/twilio-daily-webrtc) [WebSockets with Twilio\ ----------------------\ \ Using WebSockets for Twilio integration](/guides/telephony/twilio-websockets) [Dialout Capabilities\ --------------------\ \ Enable outbound calling with Daily](/guides/telephony/dialout) [​](#deployment) Deployment ------------------------------ [Overview\ --------\ \ Learn the basics of deploying Pipecat applications](/guides/deployment/overview) [Deployment Patterns\ -------------------\ \ Common architectures and deployment strategies](/guides/deployment/pattern) [Deploying to Pipecat Cloud\ --------------------------\ \ Step-by-step guide for deploying to Pipecat Cloud](/guides/deployment/pipecat-cloud) [Deploying to Fly.io\ -------------------\ \ Step-by-step guide for deploying to Fly.io](/guides/deployment/fly) [Deploying to Cerebrium\ ----------------------\ \ Deploy your application on Cerebrium](/guides/deployment/cerebrium) [​](#new-to-pipecat%3F) New to Pipecat? ------------------------------------------ [Get started\ -----------\ \ Check out our Getting Started guide to build your first application](/getting-started/installation) [OpenAI Audio Models and APIs](/guides/features/openai-audio-models-and-apis) On this page * [Features](#features) * [Telephony](#telephony) * [Deployment](#deployment) * [New to Pipecat?](#new-to-pipecat%3F) --- # Server API Reference - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Server API Reference [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) [​](#overview) Overview -------------------------- Pipecat’s server-side functionality is organized into three main categories: [Services\ --------\ \ AI service integrations for speech, language, vision, and more](/server/services/supported-services) [Frameworks\ ----------\ \ > User RTVI for client/server communication or Pipecat Flows for building structured conversations](/server/frameworks) [Utilities\ ---------\ \ Helper functions and tools for audio, filtering, and flows](/server/utilities) [​](#service-categories) Service Categories ---------------------------------------------- Pipecat integrates with various AI services across different categories: Transport --------- WebRTC and WebSocket implementations for real-time communication Speech Processing ----------------- Speech-to-text, text-to-speech, and speech-to-speech services Language Models --------------- Integration with various LLM providers Vision & Media -------------- Image generation, video processing, and computer vision [​](#getting-started) Getting Started ---------------------------------------- 1. Browse our [Supported Services](/server/services/supported-services) to see available integrations 2. Install required dependencies for your chosen services 3. Reference individual service docs for detailed configuration options [​](#example-usage) Example Usage ------------------------------------ from pipecat.audio.vad.silero import SileroVADAnalyzer from pipecat.pipeline import Pipeline from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext from pipecat.services.cartesia import CartesiaTTSService from pipecat.services.deepgram import DeepgramSTTService from pipecat.services.openai import OpenAILLMService from pipecat.transports.services.daily import DailyParams, DailyTransport transport = DailyTransport( room_url, token, "Respond bot", DailyParams( audio_out_enabled=True, vad_enabled=True, vad_analyzer=SileroVADAnalyzer(), vad_audio_passthrough=True, ), ) # Configure services stt = DeepgramSTTService(api_key=KEY) llm = OpenAILLMService(api_key=KEY, model="gpt-4o") tts = CartesiaTTSService(api_key=KEY, voice_id=ID) # Create context and aggregators context = OpenAILLMContext( messages=[{"role": "system", "content": "You are a helpful assistant."}] ) context_aggregator = llm.create_context_aggregator(context) # Create pipeline pipeline = Pipeline([\ transport.input(),\ stt,\ context_aggregator.user(),\ llm,\ tts,\ transport.output(),\ context_aggregator,assistant()\ ]) [Reference docs](/server/links/server-reference) On this page * [Overview](#overview) * [Service Categories](#service-categories) * [Getting Started](#getting-started) * [Example Usage](#example-usage) --- # Client SDKs - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Client SDKs [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) Pipecat provides client SDKs for multiple platforms, all implementing the RTVI (Real-Time Voice and Video Inference) standard. These SDKs make it easy to build real-time AI applications that can handle voice, video, and text interactions. [Javascript\ ----------\ \ Pipecat JS SDK](/client/js/introduction) [React\ -----\ \ Pipecat React SDK](/client/react/introduction) [React Native\ ------------\ \ Pipecat React Native SDK](/client/react-native/introduction) [Swift\ -----\ \ Pipecat iOS SDK](/client/ios/introduction) [Kotlin\ ------\ \ Pipecat Android SDK](/client/android/introduction) [C++\ ---\ \ Pipecat C++ SDK](/client/c++/introduction) [​](#core-functionality) Core Functionality ---------------------------------------------- All Pipecat client SDKs provide: Media Management ---------------- Handle device inputs and media streams for audio and video Bot Integration --------------- Configure and communicate with your Pipecat bot Action Handling --------------- Send actions and process bot responses Session Management ------------------ Manage connection state and error handling [​](#interaction-modes) Interaction Modes -------------------------------------------- * Real-time Streaming * Single-turn Actions Establish ongoing connections via WebSocket or WebRTC for: * Live voice conversations * Real-time video processing * Continuous interactions // Example: Establishing a real-time connection import { RTVIEvent, RTVIMessage, RTVIClient } from "realtime-ai"; import { DailyTransport } from "@pipecat-ai/daily-transport"; const rtviClient = new RTVIClient({ transport: new DailyTransport(), params: { baseUrl: "https://your-connect-end-point-here", endpoint: { connect: "/connect", } } enableMic: true, enableCam: false, enableScreenShare: false, timeout: 15 * 1000, callbacks: { onBotConnected: () => { console.log("[CALLBACK] Bot connected"); }, onBotDisconnected: () => { console.log("[CALLBACK] Bot disconnected"); }, onBotReady: () => { console.log("[CALLBACK] Bot ready to chat!"); }, }, }); try { await rtviClient.connect(); } catch (e) { console.error(e.message); } // Events rtviClient.on(RTVIEvent.Connected, () => { console.log("[EVENT] User connected"); }); rtviClient.on(RTVIEvent.Disconnected, () => { console.log("[EVENT] User disconnected"); }); [​](#about-rtvi) About RTVI ------------------------------ Pipecat’s client SDKs implement the RTVI (Real-Time Voice and Video Inference) standard, an open specification for real-time AI inference. This means: * Your code can work with any RTVI-compatible inference service * You get battle-tested tooling for real-time multimedia handling * You can easily set up development and testing environments For production deployments, we recommend setting up your own authentication endpoints rather than exposing service credentials directly in client applications. [​](#next-steps) Next Steps ------------------------------ Get started by trying out examples: [Simple Chatbot Example\ ----------------------\ \ Complete client-server example with both bot backend (Python) and frontend implementation (JS, React).](https://github.com/pipecat-ai/pipecat/tree/main/examples/simple-chatbot) [More Examples\ -------------\ \ Explore our full collection of example applications and implementations across different platforms and use cases.](https://github.com/pipecat-ai/pipecat/tree/main/examples) [SDK Introduction](/client/js/introduction) On this page * [Core Functionality](#core-functionality) * [Interaction Modes](#interaction-modes) * [About RTVI](#about-rtvi) * [Next Steps](#next-steps) --- # Use Cases - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Get Started Use Cases [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) Pipecat is designed for building real-time AI applications that interact through voice, text, images, or video. Here are some common ways developers use Pipecat: [​](#voice-assistants) Voice Assistants ------------------------------------------ The most straightforward use of Pipecat is building voice-enabled AI agents that can: * Have natural conversations with users * Maintain context across multiple exchanges * Process speech in real-time * Generate voice responses * Execute function calls The [Simple Chatbot](https://github.com/pipecat-ai/pipecat/tree/main/examples/simple-chatbot) is a great example project. This example shows how to build a basic bot and how to connect to it through a few different types of client interfaces. [​](#multimodal-applications) Multimodal Applications -------------------------------------------------------- Pipecat can handle multiple types of input and output simultaneously: * Process both voice and images * Generate text and image responses * Handle video streams * Combine different AI models Check out our [Moondream chatbot example](https://github.com/pipecat-ai/pipecat/tree/main/examples/moondream-chatbot) to see multimodal processing in action. [​](#complex-conversational-flows) Complex Conversational Flows ------------------------------------------------------------------ For applications that need structured conversations, like customer service or form filling, you can use [Pipecat Flows](https://github.com/pipecat-ai/pipecat-flows) to: * Create predefined conversation paths * Handle dynamic branching based on user input * Manage conversation state * Design flows visually using the [Flow Editor](https://flows.pipecat.ai) Get started by running one of the [Flows examples](https://github.com/pipecat-ai/pipecat-flows/tree/main/examples) . [​](#real-world-examples) Real-world Examples ------------------------------------------------ Here are some applications built with Pipecat: * [Storytelling bot](https://storytelling-chatbot.fly.dev/) that generates and narrates stories * [Customer intake system](https://www.youtube.com/watch?v=lDevgsp9vn0) for automated form filling * [Real-time AI conversations](https://demo.dailybots.ai/) using WebRTC [​](#next-steps) Next Steps ------------------------------ Ready to build your own application? 1. Follow our [Installation Guide](/getting-started/installation) to set up Pipecat 2. Build your first bot with the [Quickstart](/getting-started/quickstart) 3. Browse our [example applications](https://github.com/pipecat-ai/pipecat/tree/main/examples) for inspiration [Core Concepts](/getting-started/core-concepts) On this page * [Voice Assistants](#voice-assistants) * [Multimodal Applications](#multimodal-applications) * [Complex Conversational Flows](#complex-conversational-flows) * [Real-world Examples](#real-world-examples) * [Next Steps](#next-steps) --- # Core Concepts - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Get Started Core Concepts [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) Pipecat uses a pipeline-based architecture to handle real-time AI processing. Let’s look at how this works in practice, then break down the key components. [​](#real-time-processing-in-action) Real-time Processing in Action ---------------------------------------------------------------------- Consider how a voice assistant processes a user’s question and generates a response: Instead of waiting for complete responses at each step, Pipecat processes data in small units called frames that flow through the pipeline: 1. Speech is transcribed in real-time as the user speaks 2. Transcription is sent to the LLM as it becomes available 3. LLM responses are processed as they stream in 4. Text-to-speech begins generating audio for early sentences while later ones are still being generated 5. Audio playback starts as soon as the first sentence is ready 6. LLM context is aggregated and updated continuously and in real-time This streaming approach creates natural, responsive interactions. [​](#architecture-overview) Architecture Overview ---------------------------------------------------- Here’s how Pipecat organizes these processes: The architecture consists of three key components: ### [​](#1-frames) 1\. Frames Frames are containers for data moving through your application. Think of them like packages on a conveyor belt - each contains a specific type of cargo. For example: * Audio data from a microphone * Text from transcription * LLM responses * Generated speech audio * Images or video * Control signals and system messages Frames can flow in two directions: * Downstream (normal processing flow) * Upstream (for errors and control signals) ### [​](#2-processors-services) 2\. Processors (Services) Processors are workers along our conveyor belt. Each one: * Receives frames as inputs * Processes specific frame types * Passes through frames it doesn’t handle * Generates new frames as output Frame processors can do anything, but for real-time, multimodal AI applications, they commonly include: * A speech-to-text processor that receives raw audio input frames and outputs transcription frames * An LLM processor takes context frames and produces text frames * A text-to-speech processor that receives text frames and generates raw audio output frames * An image generation processor that takes in text frames and outputs an image URL frame * A logging processor might watch all frames but not modify them ### [​](#3-pipelines) 3\. Pipelines Pipelines connect processors together, creating a path for frames to flow through your application. They can be: # Simple linear pipeline pipeline = Pipeline([\ transport.input() # Speech -> Audio\ stt, # Audio -> Text\ llm, # Text -> Response\ tts, # Response -> Audio\ transport.output() # Audio -> Playback\ ]) # Complex parallel pipeline pipeline = Pipeline([\ input_source,\ ParallelPipeline([\ [image_processor, image_output], # Handle images\ [audio_processor, audio_output] # Handle audio\ ])\ ]) The pipeline also contains the transport, which is the connection to the real world (e.g., microphone, speakers). [​](#how-it-all-works-together) How It All Works Together ------------------------------------------------------------ Let’s see how these components handle a simple voice interaction: 1. **Input** * User speaks into their microphone * Transport converts audio into frames * Frames enter the pipeline 2. **Processing** * Transcription processor converts speech to text frames * LLM processor takes text frames, generates response frames * TTS processor converts response frames to audio frames * Error frames flow upstream if issues occur * System frames can bypass normal processing for immediate handling 3. **Output** * Audio frames reach the transport * Transport plays the audio for the user This happens continuously and in parallel, creating smooth, real-time interactions. [​](#next-steps) Next Steps ------------------------------ Now that you understand the big picture, the next step is to install and run your first Pipecat application. Check out the [Installation & Setup](/getting-started/installation) guide to get started. Need help? Join our [Discord community](https://discord.gg/pipecat) for support and discussions. [Quickstart](/getting-started/quickstart) [Use Cases](/getting-started/use-cases) On this page * [Real-time Processing in Action](#real-time-processing-in-action) * [Architecture Overview](#architecture-overview) * [1\. Frames](#1-frames) * [2\. Processors (Services)](#2-processors-services) * [3\. Pipelines](#3-pipelines) * [How It All Works Together](#how-it-all-works-together) * [Next Steps](#next-steps) --- # Installation & Setup - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Get Started Installation & Setup [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) [​](#accounts-you%E2%80%99ll-need) Accounts you’ll need ---------------------------------------------------------- While you can use Pipecat without them, there are a few third-party services you’ll probably want to use with Pipecat. We recommend these to start, since they’re easy to sign up and get started for free or low cost: * [Daily](https://dashboard.daily.co/u/signup?pipecat=true) for your transport layer—handling real-time audio, video, and data exchange between the bot and user through WebRTC. * [Cartesia](https://play.cartesia.ai/) for text-to-speech. * [OpenAI](https://platform.openai.com/signup) gets you ChatGPT for an LLM, and DALL-E for image generation. If you’re going to run the quickstart, you’ll need to set up Daily and Cartesia accounts. Learn about other [supported services](/server/services/supported-services) you can use with Pipecat. [​](#prerequisites) Prerequisites ------------------------------------ Pipecat requires Python 3.10 or higher. To check your Python version: python --version [​](#installing-pipecat) Installing Pipecat ---------------------------------------------- To start, we recommend using a virtual environment, or `venv`, inside a new directory: mkdir pipecat-test cd pipecat-test python3 -m venv env Activate the virtual environment based on your operating system: * macOS/Linux * Windows source env/bin/activate The `pipecat-ai` Python module has a lot of optional dependencies, including some pretty big AI libraries. This allows you to only install what you need. For example, to install `pipecat-ai` along with support for the services above, run this command (or add it to your `requirements.txt`): * Daily, Cartesia, and OpenAI pip install "pipecat-ai[daily,cartesia,openai]" You can always install other dependencies later. For the full list, look for the `[project.optional-dependencies]` section in [pyproject.toml](https://github.com/pipecat-ai/pipecat/blob/main/pyproject.toml) . [​](#service-configuration) Service Configuration ---------------------------------------------------- ### [​](#daily-setup) Daily Setup Head over to [dashboard.daily.co/rooms](https://dashboard.daily.co/rooms) and create a new room. You can leave everything as-is in the room creation screen, or change anything you want. (Emoji reactions and advanced chat are fun.) Make note of: * Your room URL: `https://YOURDOMAIN.daily.co/YOURROOM` * Your API key from [dashboard.daily.co/developers](https://dashboard.daily.co/developers) ### [​](#cartesia-text-to-speech-setup) Cartesia Text-to-Speech Setup From [Cartesia’s Voice library](https://play.cartesia.ai/voices) , find a voice that you like. Then click the three dots at the end of the row and select `Copy ID`. Then, head over to [Cartesia API keys](https://play.cartesia.ai/keys) and create your API key. [​](#environment-setup) Environment Setup -------------------------------------------- Create a `.env` file in your project directory with your service credentials: DAILY_SAMPLE_ROOM_URL=https://YOURDOMAIN.daily.co/YOURROOM DAILY_API_KEY=your_daily_api_key CARTESIA_API_KEY=your_cartesia_api_key CARTESIA_VOICE_ID=your_voice_id [​](#next-steps) Next Steps ------------------------------ Now that you’ve got everything set up, head to the [Quickstart](/getting-started/quickstart) to build your first Pipecat application. [Overview](/getting-started/overview) [Quickstart](/getting-started/quickstart) On this page * [Accounts you’ll need](#accounts-you%E2%80%99ll-need) * [Prerequisites](#prerequisites) * [Installing Pipecat](#installing-pipecat) * [Service Configuration](#service-configuration) * [Daily Setup](#daily-setup) * [Cartesia Text-to-Speech Setup](#cartesia-text-to-speech-setup) * [Environment Setup](#environment-setup) * [Next Steps](#next-steps) --- # Quickstart - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Get Started Quickstart [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) Now that you’ve got Pipecat installed, let’s run your first example application. We’ll use the “Say One Thing” example, which creates a simple bot that greets users when they join a room and exits when they leave. [​](#getting-the-example) Getting the Example ------------------------------------------------ Download the [“Say One Thing”](https://github.com/pipecat-ai/pipecat/raw/main/examples/foundational/01-say-one-thing.py) example from the `pipecat` repo and save it to your project directory. [​](#configuring-the-example) Configuring the Example -------------------------------------------------------- The example uses Cartesia’s text-to-speech service. Make sure you have your `.env` file set up with: DAILY_SAMPLE_ROOM_URL=https://YOURDOMAIN.daily.co/YOURROOM DAILY_API_KEY=your_daily_api_key CARTESIA_API_KEY=your_cartesia_api_key The example uses a pre-configured voice. You can change this by modifying the `voice_id` parameter in the `CartesiaTTSService` configuration. [​](#running-the-example) Running the Example ------------------------------------------------ 1. Open your Daily room URL in a browser tab 2. Join the room 3. Run the example: python 01-say-one-thing.py If all goes well, you should see another participant (named “Say One Thing”) join the room, greet you, and disconnect when you leave the room. [​](#understanding-how-it-works) Understanding How It Works -------------------------------------------------------------- Let’s look at the key components of this example: # Configure Daily transport for WebRTC communication # - room_url: Where to connect # - None: No authentication token needed # - "Say One Thing": Bot's display name # - Enable audio output for text-to-speech playback transport = DailyTransport( room_url, None, "Say One Thing", DailyParams(audio_out_enabled=True) ) # Initialize Cartesia's text-to-speech service # Using a pre-selected British female voice # You can find other voices at https://play.cartesia.ai/voices tts = CartesiaTTSService( api_key=os.getenv("CARTESIA_API_KEY"), voice_id="79a125e8-cd45-4c13-8a67-188112f4dd22", # British Lady ) # Create a pipeline runner to manage the processing pipeline runner = PipelineRunner() # Create pipeline to process text into speech and play it # 1. tts: Converts text messages into audio # 2. transport.output(): Sends audio to the Daily room pipeline = Pipeline([\ tts,\ transport.output()\ ]) # Create PipelineTask task = PipelineTask(pipeline) # When the first person joins the room: # - Extract their username from the participant info # - Generate a greeting message # - Queue it for text-to-speech conversion # - Queue an EndFrame for the bot to leave (and shutdown the pipeline) @transport.event_handler("on_first_participant_joined") async def on_first_participant_joined(transport, participant): participant_name = participant.get("info", {}).get("userName", "") await task.queue_frames( [TTSSpeakFrame(f"Hello there, {participant_name}!"), EndFrame()] ) # Run the pipeline task await runner.run(task) This example demonstrates several core Pipecat concepts: * Using a transport for audio output * Setting up a TTS service * Creating a simple processing pipeline * Handling events and generating responses * Managing application lifecycle [​](#next-steps) Next Steps ------------------------------ Now that you’ve got your first Pipecat app running, you can: * Discover Pipecat’s [supported services](/server/services/supported-services) * Explore more [Examples](https://github.com/pipecat-ai/pipecat/tree/main/examples) in our GitHub repo * Join our [Discord community](https://discord.gg/pipecat) to get help and share your projects [Installation & Setup](/getting-started/installation) [Core Concepts](/getting-started/core-concepts) On this page * [Getting the Example](#getting-the-example) * [Configuring the Example](#configuring-the-example) * [Running the Example](#running-the-example) * [Understanding How It Works](#understanding-how-it-works) * [Next Steps](#next-steps) --- # Supported Services - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Services Supported Services [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) [​](#transports) Transports ------------------------------ | Service | Setup | | --- | --- | | [Daily](/server/services/transport/daily) | `pip install "pipecat-ai[daily]"` | | [SmallWebRTCTransport](/server/services/transport/small-webrtc) | `pip install "pipecat-ai[webrtc]"` | | [FastAPI WebSocket](/server/services/transport/fastapi-websocket) | `pip install "pipecat-ai[websocket]"` | | [WebSocket Server](/server/services/transport/websocket-server) | `pip install "pipecat-ai[websocket]"` | [​](#speech-to-text) Speech-to-Text -------------------------------------- | Service | Setup | | --- | --- | | [AssemblyAI](/server/services/stt/assemblyai) | `pip install "pipecat-ai[assemblyai]"` | | [Azure](/server/services/stt/azure) | `pip install "pipecat-ai[azure]"` | | [Deepgram](/server/services/stt/deepgram) | `pip install "pipecat-ai[deepgram]"` | | [Fal Wizper](/server/services/stt/fal) | `pip install "pipecat-ai[fal]"` | | [Gladia](/server/services/stt/gladia) | `pip install "pipecat-ai[gladia]"` | | [Google](/server/services/stt/google) | `pip install "pipecat-ai[google]"` | | [Groq (Whisper)](/server/services/stt/groq) | `pip install "pipecat-ai[groq]"` | | [NVIDIA Parakeet](/server/services/stt/parakeet) | `pip install "pipecat-ai[riva]"` | | [OpenAI (Whisper)](/server/services/stt/openai) | `pip install "pipecat-ai[openai]"` | | [Ultravox](/server/services/stt/ultravox) | `pip install "pipecat-ai[ultravox]"` | | [Whisper](/server/services/stt/whisper) | `pip install "pipecat-ai[whisper]"` | [​](#large-language-models) Large Language Models ---------------------------------------------------- | Service | Setup | | --- | --- | | [Anthropic](/server/services/llm/anthropic) | `pip install "pipecat-ai[anthropic]"` | | [Azure](/server/services/llm/azure) | `pip install "pipecat-ai[azure]"` | | [Cerebras](/server/services/llm/cerebras) | `pip install "pipecat-ai[cerebras]"` | | [DeepSeek](/server/services/llm/deepseek) | `pip install "pipecat-ai[deepseek]"` | | [Fireworks AI](/server/services/llm/fireworks) | `pip install "pipecat-ai[fireworks]"` | | [Google Gemini](/server/services/llm/gemini) | `pip install "pipecat-ai[google]"` | | [Grok](/server/services/llm/grok) | `pip install "pipecat-ai[grok]"` | | [Groq](/server/services/llm/groq) | `pip install "pipecat-ai[groq]"` | | [NVIDIA NIM](/server/services/llm/nim) | `pip install "pipecat-ai[nim]"` | | [Ollama](/server/services/llm/ollama) | `pip install "pipecat-ai[ollama]"` | | [OpenAI](/server/services/llm/openai) | `pip install "pipecat-ai[openai]"` | | [OpenPipe](/server/services/llm/openpipe) | `pip install "pipecat-ai[openpipe]"` | | [OpenRouter](/server/services/llm/openrouter) | `pip install "pipecat-ai[openrouter]"` | | [Perplexity](/server/services/llm/perplexity) | `pip install "pipecat-ai[perplexity]"` | | [Qwen](/server/services/llm/qwen) | `pip install "pipecat-ai[qwen]"` | | [Together AI](/server/services/llm/together) | `pip install "pipecat-ai[together]"` | [​](#text-to-speech) Text-to-Speech -------------------------------------- | Service | Setup | | --- | --- | | [Amazon Polly](/server/services/tts/aws) | `pip install "pipecat-ai[aws]"` | | [Azure](/server/services/tts/azure) | `pip install "pipecat-ai[azure]"` | | [Cartesia](/server/services/tts/cartesia) | `pip install "pipecat-ai[cartesia]"` | | [Deepgram](/server/services/tts/deepgram) | `pip install "pipecat-ai[deepgram]"` | | [ElevenLabs](/server/services/tts/elevenlabs) | `pip install "pipecat-ai[elevenlabs]"` | | [Fish](/server/services/tts/fish) | `pip install "pipecat-ai[fish]"` | | [Google](/server/services/tts/google) | `pip install "pipecat-ai[google]"` | | [Groq](/server/services/tts/groq) | `pip install "pipecat-ai[groq]"` | | [LMNT](/server/services/tts/lmnt) | `pip install "pipecat-ai[lmnt]"` | | [Neuphonic](/server/services/tts/neuphonic) | `pip install "pipecat-ai[neuphonic]"` | | [NVIDIA FastPitch](/server/services/tts/fastpitch) | `pip install "pipecat-ai[riva]"` | | [OpenAI](/server/services/tts/openai) | `pip install "pipecat-ai[openai]"` | | [Piper](/server/services/tts/piper) | No dependencies required | | [PlayHT](/server/services/tts/playht) | `pip install "pipecat-ai[playht]"` | | [Rime](/server/services/tts/rime) | `pip install "pipecat-ai[rime]"` | | [XTTS](/server/services/tts/xtts) | `pip install "pipecat-ai[xtts]"` | [​](#speech-to-speech) Speech-to-Speech ------------------------------------------ | Service | Setup | | --- | --- | | [Gemini Multimodal Live](/server/services/s2s/gemini) | `pip install "pipecat-ai[google]"` | | [OpenAI Realtime](/server/services/s2s/openai) | `pip install "pipecat-ai[openai]"` | [​](#image-generation) Image Generation ------------------------------------------ | Service | Setup | | --- | --- | | [fal](/server/services/image-generation/fal) | `pip install "pipecat-ai[fal]"` | | [Google Imagen](/server/services/image-generation/google-imagen) | `pip install "pipecat-ai[google]"` | [​](#video) Video -------------------- | Service | Setup | | --- | --- | | [Simli](/server/services/video/simli) | `pip install "pipecat-ai[simli]"` | | [Tavus](/server/services/video/tavus) | `pip install "pipecat-ai[tavus]"` | [​](#memory) Memory ---------------------- | Service | Setup | | --- | --- | | [mem0](/server/services/memory/mem0) | `pip install "pipecat-ai[mem0]"` | [​](#vision) Vision ---------------------- | Service | Setup | | --- | --- | | [Moondream](/server/services/vision/moondream) | `pip install "pipecat-ai[moondream]"` | [​](#analytics-%26-monitoring) Analytics & Monitoring -------------------------------------------------------- | Service | Setup | | --- | --- | | [Canonical AI](/server/services/analytics/canonical) | `pip install "pipecat-ai[canonical]"` | | [Sentry](/server/services/analytics/sentry) | `pip install "pipecat-ai[sentry]"` | For detailed configuration and usage examples, see each service’s individual documentation page. [Reference docs](/server/links/server-reference) [Daily WebRTC](/server/services/transport/daily) On this page * [Transports](#transports) * [Speech-to-Text](#speech-to-text) * [Large Language Models](#large-language-models) * [Text-to-Speech](#text-to-speech) * [Speech-to-Speech](#speech-to-speech) * [Image Generation](#image-generation) * [Video](#video) * [Memory](#memory) * [Vision](#vision) * [Analytics & Monitoring](#analytics-%26-monitoring) --- # Building With OpenAI Audio Models and APIs - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Features Building With OpenAI Audio Models and APIs [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) This guide provides an overview of the audio capabilities OpenAI offers via their APIs. We’ll also link to Pipecat sample code. [​](#two-ways-to-build-voice-to-voice) Two Ways To Build Voice-to-voice -------------------------------------------------------------------------- You can build voice-to-voice applications in two ways: 1. The cascaded models approach, using separate models for transcription, the LLM, and voice generation. A cascaded pipeline looks like this, in Pipecat code. Here’s a [single-file example that uses a cascaded pipeline](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07g-interruptible-openai.py) . (See below for an overview of Pipecat core concepts.) pipeline = Pipeline( [\ transport.input(),\ speech_to_text,\ context_aggregator.user(),\ llm,\ text_to_speech,\ context_aggregator.assistant(),\ transport.output(),\ ] ) 2. Using a single, speech-to-speech model. This is conceptually much simpler. Though note that most applications also need to implement things like function calling, retrieval-augmented search, context management, and integration with existing systems. So the core pipeline is only part of an app’s complexity. Here’s a speech-to-speech pipeline in Pipecat code. And here’s a [single-file example that uses the OpenAI Realtime API](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/19-openai-realtime-beta.py) . pipeline = Pipeline( [\ transport.input(),\ context_aggregator.user(),\ speech_to_speech_llm,\ context_aggregator.assistant(),\ transport.output(),\ ] ) Which approach should you choose? * The cascaded models approach is preferable if you are implementing a complex workflow and need the best possible instruction following performance and function calling reliability. The `gpt-4o` model operating in text-to-text mode has the strongest instruction following and function calling performance. * The speech-to-speech approach offers better audio understanding and human-like voice output. If your application is primarily free-form, open-ended conversation, these attributes might be more important than instruction following and function calling performance. Note also that `gpt-4o-audio-preview` and the OpenAI Realtime API are currently beta products. [​](#openai-audio-models-and-apis) OpenAI Audio Models and APIs ------------------------------------------------------------------ ### [​](#transcription-api) Transcription API * Models: `gpt-4o-transcribe`, `gpt-4o-mini-transcribe` * Pipecat service: `OpenAISTTService` ([reference docs](https://docs.pipecat.ai/server/services/stt/openai) ) * OpenAI endpoint: `/v1/audio/transcriptions` ([docs](https://platform.openai.com/docs/api-reference/audio/createTranscription) ) ### [​](#chat-completions-api) Chat Completions API * Models: `gpt-4o`, `gpt-4o-mini`, `gpt-4o-audio-preview` * Pipecat service: `OpenAILLMService` ([reference docs](https://docs.pipecat.ai/server/services/llm/openai) ) * OpenAI endpoint: `/v1/chat/completions` ([docs](https://platform.openai.com/docs/api-reference/chat) ) ### [​](#realtime-api) Realtime API * Models: `gpt-4o-realtime-preview`, `gpt-4o-mini-realtime-preview` * Pipecat service: `OpenAIRealtimeBetaLLMService` ([reference docs](https://docs.pipecat.ai/server/services/s2s/openai) ) * OpenAI docs ([overview](https://platform.openai.com/docs/guides/realtime) ) ### [​](#speech-api) Speech API * Models: `gpt-4o-mini-tts` * Pipecat service: `OpenAITTSService` ([reference docs](https://docs.pipecat.ai/server/services/tts/openai) ) * OpenAI endpoint: `/v1/audio/speech` ([docs](https://platform.openai.com/docs/api-reference/audio/createSpeech) ) [​](#sample-code-and-starter-kits) Sample code and starter kits ------------------------------------------------------------------ _If you have a code example or starter kit you would like this doc to link to, please let us know. We can add examples that help people get started with the OpenAI audio models and APIs._ ### [​](#single-file-examples) Single-file examples [OpenAI STT → LLM → TTS\ ----------------------\ \ A complete implementation demonstrating the cascaded approach with OpenAI services](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07g-interruptible-openai.py) [OpenAI Realtime API\ -------------------\ \ A speech-to-speech implementation using OpenAI’s Realtime API](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/19-openai-realtime-beta.py) ### [​](#openai-%2B-twilio-%2B-pipecat-cloud) OpenAI + Twilio + Pipecat Cloud [This starter kit](https://github.com/daily-co/pcc-openai-twilio/) is a complete telephone voice agent that can talk about the NCAA March Madness basketball tournaments and look up realtime game information using function calls. The starter kit includes two bot configurations: cascaded model and speech-to-speech. The code can be packaged for deployment to Pipecat Cloud, a commercial platform for Pipecat agent hosting. [Guides](/guides/introduction) [Gemini Multimodal Live](/guides/features/gemini-multimodal-live) On this page * [Two Ways To Build Voice-to-voice](#two-ways-to-build-voice-to-voice) * [OpenAI Audio Models and APIs](#openai-audio-models-and-apis) * [Transcription API](#transcription-api) * [Chat Completions API](#chat-completions-api) * [Realtime API](#realtime-api) * [Speech API](#speech-api) * [Sample code and starter kits](#sample-code-and-starter-kits) * [Single-file examples](#single-file-examples) * [OpenAI + Twilio + Pipecat Cloud](#openai-%2B-twilio-%2B-pipecat-cloud) --- # Building with Gemini Multimodal Live - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Features Building with Gemini Multimodal Live [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) This guide will walk you through building a real-time AI chatbot using Gemini Multimodal Live and Pipecat. We’ll create a complete application with a Pipecat server and a Pipecat React client that enables natural conversations with an AI assistant. [API Reference\ -------------\ \ Gemini Multimodal Live API documentation](/server/services/s2s/gemini) [Example Code\ ------------\ \ Find the complete client and server code in Github](https://github.com/pipecat-ai/pipecat/tree/main/examples/simple-chatbot) [Client SDK\ ----------\ \ Pipecat React SDK documentation](/client/react/introduction) [​](#what-we%E2%80%99ll-build) What We’ll Build -------------------------------------------------- In this guide, you’ll create: * A FastAPI server that manages bot instances * A Gemini-powered conversational AI bot * A React client with real-time audio/video * A complete pipeline for speech-to-speech interaction [​](#key-concepts) Key Concepts ---------------------------------- Before we dive into implementation, let’s cover some important concepts that will help you understand how Pipecat and Gemini work together. ### [​](#understanding-pipelines) Understanding Pipelines At the heart of Pipecat is the pipeline system. A pipeline is a sequence of processors that handle different aspects of the conversation flow. Think of it like an assembly line where each station (processor) performs a specific task. For our chatbot, the pipeline looks like this: pipeline = Pipeline([\ transport.input(), # Receives audio/video from the user via WebRTC\ context_aggregator.user(), # Manages user message history\ llm, # Processes speech through Gemini\ rtvi_speaking, # Tracks speaking states\ rtvi_user_transcription, # Handles user speech transcription\ rtvi_bot_transcription, # Handles bot speech transcription\ talking_animation, # Controls bot's avatar\ transport.output(), # Sends audio/video back to the user via WebRTC\ ]) ### [​](#processors) Processors Each processor in the pipeline handles a specific task: Transport --------- `transport.input()` and `transport.output()` handle media streaming with Daily Context ------- `context_aggregator` maintains conversation history for natural dialogue Speech Processing ----------------- `rtvi_user_transcription` and `rtvi_bot_transcription` handle speech-to-text Animation --------- `talking_animation` controls the bot’s visual state based on speaking activity The order of processors matters! Data flows through the pipeline in sequence, so each processor should receive the data it needs from previous processors. Learn more about the [Core Concepts](/getting-started/core-concepts) to Pipecat server. ### [​](#gemini-integration) Gemini Integration The `GeminiMultimodalLiveLLMService` is a speech-to-speech LLM service that interfaces with the Gemini Multimodal Live API. It provides: * Real-time speech-to-speech conversation * Context management * Voice activity detection * Tool use Pipecat manages two types of connections: 1. A WebRTC connection between the Pipecat client and server for reliable audio/video streaming 2. A WebSocket connection between the Pipecat server and Gemini for real-time AI processing This architecture ensures stable media streaming while maintaining responsive AI interactions. [​](#prerequisites) Prerequisites ------------------------------------ Before we begin, you’ll need: * Python 3.10 or higher * Node.js 16 or higher * A [Daily API key](https://dashboard.daily.co/u/signup?pipecat=1) * A Google API key with Gemini Multimodal Live access * Clone the Pipecat repo: git clone git@github.com:pipecat-ai/pipecat.git [​](#server-implementation) Server Implementation ---------------------------------------------------- Let’s start by setting up the server components. Our server will handle bot management, room creation, and client connections. ### [​](#environment-setup) Environment Setup 1. Navigate to the simple-chatbot’s server directory: cd examples/simple-chatbot/server 2. Set up a python virtual environment: python -m venv venv source venv/bin/activate # On Windows: venv\Scripts\activate 3. Install requirements: pip install -r requirements.txt 4. Copy env.example to .env and make a few changes: # Remove the hard-coded example room URL DAILY_SAMPLE_ROOM_URL= # Add your Daily and Gemini API keys DAILY_API_KEY=[your key here] GEMINI_API_KEY=[your key here] # Use Gemini implementation BOT_IMPLEMENTATION=gemini ### [​](#server-setup-server-py) Server Setup (server.py) `server.py` is a FastAPI server that creates the meeting room where clients and bots interact, manages bot instances, and handles client connections. It’s the orchestrator that brings everything on the server-side together. #### [​](#creating-meeting-room) Creating Meeting Room The server uses Daily’s API via a [REST API helper](/server/utilities/daily/rest-helpers#create-room) to create rooms where clients and bots can meet. Each room is a secure space for audio/video communication: server/server.py async def create_room_and_token(): """Create a Daily room and generate access credentials.""" room = await daily_helpers["rest"].create_room(DailyRoomParams()) token = await daily_helpers["rest"].get_token(room.url) return room.url, token #### [​](#managing-bot-instances) Managing Bot Instances When a client connects, the server starts a new bot instance configured specifically for that room. It keeps track of running bots and ensures there’s only one bot per room: server/server.py # Start the bot process for a specific room bot_file = "bot-gemini.py" proc = subprocess.Popen([f"python3 -m {bot_file} -u {room_url} -t {token}"]) bot_procs[proc.pid] = (proc, room_url) #### [​](#connection-endpoints) Connection Endpoints The server provides two ways to connect: Browser Access (/) ------------------ Creates a room, starts a bot, and redirects the browser to the Daily meeting URL. Perfect for quick testing and development. RTVI Client (/connect) ---------------------- Creates a room, starts a bot, and returns connection credentials. Used by RTVI clients for custom implementations. ### [​](#bot-implementation-bot-gemini-py) Bot Implementation (bot-gemini.py) The bot implementation connects all the pieces: Daily transport, Gemini service, conversation context, and processors. Let’s break down each component: #### [​](#transport-setup) Transport Setup First, we configure the Daily transport, which handles WebRTC communication between the client and server. server/bot-gemini.py transport = DailyTransport( room_url, token, "Chatbot", DailyParams( audio_out_enabled=True, # Enable audio output camera_out_enabled=True, # Enable video output vad_enabled=True, # Enable voice activity detection, 0.5 sec stop time vad_analyzer=SileroVADAnalyzer(params=VADParams(stop_secs=0.5)), ), ) Gemini Multimodal Live audio requirements: * Input: 16 kHz sample rate * Output: 24 kHz sample rate #### [​](#gemini-service-configuration) Gemini Service Configuration Next, we initialize the Gemini service which will provide speech-to-speech inference and communication: server/bot-gemini.py llm = GeminiMultimodalLiveLLMService( api_key=os.getenv("GEMINI_API_KEY"), voice_id="Puck", # Choose your bot's voice transcribe_user_audio=True, # Enable speech-to-text transcribe_model_audio=True, # Log bot responses params=InputParams(temperature=0.7) # Set model input params ) #### [​](#conversation-context) Conversation Context We give our bot its personality and initial instructions: server/bot-gemini.py messages = [{\ "role": "user",\ "content": """You are Chatbot, a friendly, helpful robot.\ Keep responses brief and avoid special characters\ since output will be converted to audio."""\ }] context = OpenAILLMContext(messages) context_aggregator = llm.create_context_aggregator(context) `OpenAILLMContext` is used as a common LLM base service for context management. In the future, we may add a specific context manager for Gemini. The context aggregator automatically maintains conversation history, helping the bot remember previous interactions. #### [​](#processor-setup) Processor Setup We initialize several processors to handle different aspects of the interaction: RTVI Processors --------------- `RTVISpeakingProcessor`: Manages speaking states `RTVITranscriptionProcessor`: Handles transcription events `RTVIMetricsProcessor`: Tracks performance metrics Animation --------- `TalkingAnimation`: Controls the bot’s visual state, switching between static and animated frames based on speaking status [Learn more](/server/frameworks/rtvi/introduction) about the RTVI framework and available processors. #### [​](#pipeline-assembly) Pipeline Assembly Finally, we bring everything together in a pipeline: server/bot-gemini.py pipeline = Pipeline([\ transport.input(), # Receive media\ context_aggregator.user(), # Process user context\ llm, # Gemini processing\ rtvi_speaking, # Speaking states\ rtvi_user_transcription, # User transcripts\ rtvi_bot_transcription, # Bot transcripts\ ta, # Animation\ rtvi_metrics, # Metrics\ transport.output(), # Send media\ context_aggregator.assistant() # Process bot context\ ]) The order of processors is crucial! For example, transcription processors should come after the LLM to capture the processed speech. Traditional STT, LLM, TTS pipelines have a different ordering, so be sure to tailor your processor ordering based on the elements in the pipeline. [​](#client-implementation) Client Implementation ---------------------------------------------------- Our React client uses the [Pipecat React SDK](/client/react/introduction) to communicate with the bot. Let’s explore how the client connects and interacts with our Pipecat server. ### [​](#connection-setup) Connection Setup The client needs to connect to our bot server using the same transport type (Daily WebRTC) that we configured on the server: examples/react/src/providers/RTVIProvider.tsx const transport = new DailyTransport(); const client = new RTVIClient({ transport, params: { baseUrl: "http://localhost:7860", // Your bot server address endpoints: { connect: "/connect", // Matches server.py endpoint }, }, enableMic: true, // Enable audio input enableCam: false, // Disable video input enableScreenShare: false, // Disable screen sharing }); The connection configuration must match your server: * `DailyTransport`: Matches the WebRTC transport used in `bot-gemini.py` * `connect` endpoint: Matches the `/connect` route in `server.py` * Media settings: Controls which devices are enabled on join ### [​](#media-handling) Media Handling Pipecat’s React components handle all the complex media stream management for you: function App() { return (
{/* Bot's video feed */} {/* Audio input/output */}
); } The `RTVIClientProvider` is the root component for providing RTVI client context to your application. By wrapping your `RTVIClientAudio` and `RTVIClientVideo` components in this provider, they can access the client instance and receive and process the streams received from the Pipecat server. ### [​](#real-time-events) Real-time Events The RTVI processors we configured in the pipeline emit events that we can handle in our client: // Listen for transcription events useRTVIClientEvent(RTVIEvent.UserTranscript, (data: TranscriptData) => { if (data.final) { console.log(`User said: ${data.text}`); } }); // Listen for bot responses useRTVIClientEvent(RTVIEvent.BotTranscript, (data: BotLLMTextData) => { console.log(`Bot responded: ${data.text}`); }); Available Events ---------------- * Speaking state changes * Transcription updates * Bot responses * Connection status * Performance metrics Event Usage ----------- Use these events to: * Show speaking indicators * Display transcripts * Update UI state * Monitor performance Optionally, uses callbacks to handle events in your application. [Learn more](/client/js/api-reference/callbacks#callbacks) in the Pipecat client docs. ### [​](#complete-example) Complete Example Here’s a basic client implementation with connection status and transcription display: function ChatApp() { return (
{/* Connection UI */} {/* Media Components */} {/* Debug/Transcript Display */}
); } Check out the [example repository](https://github.com/pipecat-ai/pipecat/tree/main/examples/simple-chatbot) for a complete client implementation with styling and error handling. [​](#running-the-application) Running the Application -------------------------------------------------------- From the `simple-chatbot` directory, start the server and client to test the chatbot: ### [​](#1-start-the-server) 1\. Start the Server In one terminal: python server/server.py ### [​](#2-start-the-client) 2\. Start the Client In another terminal: cd examples/react npm install npm run dev ### [​](#3-testing-the-connection) 3\. Testing the Connection 1. Open `http://localhost:5173` in your browser 2. Click “Connect” to join a room 3. Allow microphone access when prompted 4. Start talking with your AI assistant Troubleshooting: * Check that all API keys are properly configured in .env * Grant your browser access to your microphone, so it can receive your audio input * Verify WebRTC ports aren’t blocked by firewalls [​](#next-steps) Next Steps ------------------------------ Now that you have a working chatbot, consider these enhancements: * Add custom avatar animations * Implement [function calling](/guides/features/function-calling) for external integrations * Add support for multiple languages * Enhance error recovery and reconnection logic ### [​](#examples) Examples [Foundational Example\ --------------------\ \ A basic implementation demonstrating core Gemini Multimodal Live features and transcription capabilities](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/26a-gemini-multimodal-live-transcription.py) [Simple Chatbot\ --------------\ \ A complete client/server implementation showing how to build a Pipecat JS or React client that connects to a Gemini Live Pipecat bot](https://github.com/pipecat-ai/pipecat/tree/main/examples/simple-chatbot) ### [​](#learn-more) Learn More * [Gemini Multimodal Live API Reference](/server/services/s2s/gemini) * [RTVI React SDK Documentation](https://docs.pipecat.ai/client/react/introduction) [OpenAI Audio Models and APIs](/guides/features/openai-audio-models-and-apis) [Function Calling](/guides/features/function-calling) On this page * [What We’ll Build](#what-we%E2%80%99ll-build) * [Key Concepts](#key-concepts) * [Understanding Pipelines](#understanding-pipelines) * [Processors](#processors) * [Gemini Integration](#gemini-integration) * [Prerequisites](#prerequisites) * [Server Implementation](#server-implementation) * [Environment Setup](#environment-setup) * [Server Setup (server.py)](#server-setup-server-py) * [Creating Meeting Room](#creating-meeting-room) * [Managing Bot Instances](#managing-bot-instances) * [Connection Endpoints](#connection-endpoints) * [Bot Implementation (bot-gemini.py)](#bot-implementation-bot-gemini-py) * [Transport Setup](#transport-setup) * [Gemini Service Configuration](#gemini-service-configuration) * [Conversation Context](#conversation-context) * [Processor Setup](#processor-setup) * [Pipeline Assembly](#pipeline-assembly) * [Client Implementation](#client-implementation) * [Connection Setup](#connection-setup) * [Media Handling](#media-handling) * [Real-time Events](#real-time-events) * [Complete Example](#complete-example) * [Running the Application](#running-the-application) * [1\. Start the Server](#1-start-the-server) * [2\. Start the Client](#2-start-the-client) * [3\. Testing the Connection](#3-testing-the-connection) * [Next Steps](#next-steps) * [Examples](#examples) * [Learn More](#learn-more) --- # SDK Introduction - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation React Native SDK SDK Introduction [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) The Pipecat React Native SDK leverages the [Pipecat JavaScript SDK](/client/js/introduction) to provide seamless integration for React Native applications. Since the JavaScript SDK is designed to work across both web and React Native platforms, the core functionalities remain the same: * Device and media stream management * Managing bot configuration * Sending actions to the bot * Handling bot messages and responses * Managing session state and errors The primary difference lies in the transport layer, which is tailored to support the unique requirements of the React Native environment. For example, when using the SDK with React Native, you would install `RNDailyTransport` instead of `DailyTransport`. [​](#installation) Installation ---------------------------------- Install the SDK and a transport implementation (e.g. Daily for WebRTC): npm i @pipecat-ai/react-native-daily-transport npm i @daily-co/react-native-daily-js@^0.70.0 npm i @daily-co/react-native-webrtc@^118.0.3-daily.2 npm i @react-native-async-storage/async-storage@^1.23.1 npm i react-native-background-timer@^2.4.1 npm i react-native-get-random-values@^1.11.0 Installing `@pipecat-ai/react-native-daily-transport` automatically includes the corresponding version of the JavaScript SDK. If you are using Expo, you will also need to add the following dependencies: npm i @config-plugins/react-native-webrtc@^10.0.0 npm i @daily-co/config-plugin-rn-daily-js@0.0.7 [​](#requirements) Requirements ---------------------------------- This package introduces some constraints on what OS/SDK versions your project can support: * iOS: Deployment target >= 13 * Android: `minSdkVersion` >= 24 [​](#quick-start) Quick start -------------------------------- Here’s a simple example using Daily as the transport layer: import { RNDailyTransport } from '@pipecat-ai/react-native-daily-transport'; import { RTVIClient } from '@pipecat-ai/client-js'; // Create and configure the client let voiceClient = new RTVIClient({ params: { baseUrl: process.env.PIPECAT_API_URL || "/api", }, transport: new RNDailyTransport(), enableMic: true }); // Connect to your bot await voiceClient.connect(); > You can find a basic working example [here](https://github.com/pipecat-ai/pipecat-client-react-native-daily-transport/tree/main/example) > and a more comprehensive example [here](https://github.com/daily-demos/daily-bots-react-native-demo/) > . [​](#explore-the-sdk) Explore the SDK ---------------------------------------- The Pipecat React Native SDK leverages the Pipecat JavaScript SDK for seamless integration with React Native applications. For detailed information, refer to our JavaScript documentation. > Just ensure you use the appropriate transport layer for React Native. [Client Constructor\ ------------------\ \ Configure your client instance with transport and callbacks](/client/js/api-reference/client-constructor) [Client Methods\ --------------\ \ Core methods for interacting with your bot](/client/js/api-reference/client-methods) [API Reference\ -------------\ \ Detailed documentation of all available APIs](/client/js/api-reference) [Helpers\ -------\ \ Utility functions for common operations](/client/js/helpers) [Hooks](/client/react/hooks) [API Reference](/client/react-native/api-reference) On this page * [Installation](#installation) * [Requirements](#requirements) * [Quick start](#quick-start) * [Explore the SDK](#explore-the-sdk) --- # Metrics - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Features Metrics [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) When developing real-time, multimodal AI applications, monitoring two key factors is crucial: performance (latency) and LLM/TTS usage. Performance impacts user experience, while usage can affect operational costs. Pipecat offers built-in metrics for both, which can be enabled with straightforward configuration options. [​](#enabling-performance-metrics) Enabling performance metrics ------------------------------------------------------------------ Set `enable_metrics=True` in `PipelineParams` when creating a task: Example config task = PipelineTask( pipeline, params=PipelineParams( ... enable_metrics=True, ... ), ) Once enabled, Pipecat logs the following metrics: | Metric | Description | | --- | --- | | TTFB | Time To First Byte in seconds | | Processing Time | Time taken by the service to respond in seconds | Sample output AnthropicLLMService#0 TTFB: 0.8378312587738037 CartesiaTTSService#0 processing time: 0.0005071163177490234 CartesiaTTSService#0 TTFB: 0.17177796363830566 AnthropicLLMService#0 processing time: 2.4927797317504883 ### [​](#limiting-ttfb-responses) Limiting TTFB responses If you only want the **first** TTFB measurement for each service, you can optionally pass `report_only_initial_ttfb=True` in `PipelineParams`: Example config task = PipelineTask( pipeline, params=PipelineParams( ... enable_metrics=True, report_only_initial_ttfb=True, ... ), ) > **Note:** `enable_metrics=True` is required for this setting to have an effect. [​](#enabling-llm%2Ftts-usage-metrics) Enabling LLM/TTS Usage Metrics ======================================================================== Set `enable_usage_metrics=True` in PipelineParams when creating a task: Example config task = PipelineTask( pipeline, params=PipelineParams( ... enable_usage_metrics=True, ... ), ) Pipecat will log the following as applicable: | Metric | Description | | --- | --- | | LLM Usage | Number of prompt and completion tokens used | | TTS Usage | Number of characters processed | Sample output CartesiaTTSService#0 usage characters: 65 AnthropicLLMService#0 prompt tokens: 104, completion tokens: 53 > **Note:** Usage metrics are recorded per interaction and do not represent running totals. [Function Calling](/guides/features/function-calling) [Noise cancellation with Krisp](/guides/features/krisp) On this page * [Enabling performance metrics](#enabling-performance-metrics) * [Limiting TTFB responses](#limiting-ttfb-responses) * [Enabling LLM/TTS Usage Metrics](#enabling-llm%2Ftts-usage-metrics) --- # SDK Introduction - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation React SDK SDK Introduction [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) The Pipecat React SDK provides React-specific components and hooks for building voice and multimodal AI applications. It wraps the core JavaScript SDK functionality in an idiomatic React interface that handles: * React context for client state management * Components for audio and video rendering * Hooks for accessing client functionality * Media device management * Event handling through hooks [​](#installation) Installation ---------------------------------- Install the SDK, core client, and a transport implementation (e.g. Daily for WebRTC): npm install @pipecat-ai/client-js npm install @pipecat-ai/client-react npm install @pipecat-ai/daily-transport [​](#example) Example ------------------------ Here’s a simple example using Daily as the transport layer: import { RTVIClient } from "@pipecat-ai/client-js"; import { RTVIClientProvider, RTVIClientAudio, useRTVIClient, } from "@pipecat-ai/client-react"; import { DailyTransport } from "@pipecat-ai/daily-transport"; // Create the client instance const client = new RTVIClient({ params: { baseUrl: process.env.PIPECAT_API_URL || "/api", endpoint: { connect: "/connect", }, }, transport: new DailyTransport(), enableMic: true, }); // Root component wraps the app with the provider function App() { return ( ); } // Component using the client function VoiceBot() { const client = useRTVIClient(); return ( ; ); } [​](#explore-the-sdk) Explore the SDK ---------------------------------------- [Components\ ----------\ \ Ready-to-use components for audio, video, and visualization](/client/react/components) [Hooks\ -----\ \ React hooks for accessing client functionality](/client/react/hooks) The Pipecat React SDK builds on top of the [JavaScript SDK](/client/js/introduction) to provide an idiomatic React interface while maintaining compatibility with the RTVI standard. [OpenAIRealTimeWebRTCTransport](/client/js/transports/openai-webrtc) [Components](/client/react/components) On this page * [Installation](#installation) * [Example](#example) * [Explore the SDK](#explore-the-sdk) --- # SDK Introduction - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Javascript SDK SDK Introduction [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) The Pipecat JavaScript SDK provides a lightweight client implementation that handles: * Device and media stream management * Managing bot configuration * Sending actions to the bot * Handling bot messages and responses * Managing session state and errors [​](#installation) Installation ---------------------------------- Install the SDK and a transport implementation (e.g. Daily for WebRTC): npm install @pipecat-ai/client-js npm install @pipecat-ai/daily-transport [​](#example) Example ------------------------ Here’s a simple example using Daily as the transport layer: import { RTVIClient } from "@pipecat-ai/client-js"; import { DailyTransport } from "@pipecat-ai/daily-transport"; // Handle incoming audio from the bot function handleBotAudio(track, participant) { if (participant.local || track.kind !== "audio") return; const audioElement = document.createElement("audio"); audioElement.srcObject = new MediaStream([track]); document.body.appendChild(audioElement); audioElement.play(); } // Create and configure the client const rtviClient = new RTVIClient({ params: { baseUrl: process.env.PIPECAT_API_URL || "/api", }, transport: new DailyTransport(), enableMic: true, callbacks: { onTrackStart: handleBotAudio, }, }); // Connect to your bot rtviClient.connect(); [​](#explore-the-sdk) Explore the SDK ---------------------------------------- [Client Constructor\ ------------------\ \ Configure your client instance with transport and callbacks](/client/js/api-reference/client-constructor) [Client Methods\ --------------\ \ Core methods for interacting with your bot](/client/js/api-reference/client-methods) [API Reference\ -------------\ \ Detailed documentation of all available APIs](/client/js/api-reference) [Helpers\ -------\ \ Utility functions for common operations](/client/js/helpers) The Pipecat JavaScript SDK implements the [RTVI standard](/client/introduction#about-rtvi) for real-time AI inference, ensuring compatibility with any RTVI-compatible server and transport layer. [Client SDKs](/client/introduction) [Client Constructor](/client/js/api-reference/client-constructor) On this page * [Installation](#installation) * [Example](#example) * [Explore the SDK](#explore-the-sdk) --- # UserIdleProcessor - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Utilities UserIdleProcessor [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) The `UserIdleProcessor` is a specialized frame processor that monitors user activity in a conversation and executes callbacks when the user becomes idle. It’s particularly useful for maintaining engagement by detecting periods of user inactivity and providing escalating responses to inactivity. [​](#constructor-parameters) Constructor Parameters ------------------------------------------------------ [​](#param-callback) callback Union\[BasicCallback, RetryCallback\] required An async function that will be called when user inactivity is detected. Can be either: * Basic callback: `async def(processor: UserIdleProcessor) -> None` * Retry callback: `async def(processor: UserIdleProcessor, retry_count: int) -> bool` where returning `False` stops idle monitoring [​](#param-timeout) timeout float required The number of seconds to wait before considering the user idle. [​](#behavior) Behavior -------------------------- The processor starts monitoring for inactivity only after the first conversation activity (either `UserStartedSpeakingFrame` or `BotSpeakingFrame`). It manages idle state based on the following rules: * Resets idle timer when user starts or stops speaking * Pauses idle monitoring while user is speaking * Resets idle timer when bot is speaking * Stops monitoring on conversation end or cancellation * Manages a retry count for the retry callback * Stops monitoring when retry callback returns `False` [​](#properties) Properties ------------------------------ [​](#param-retry-count) retry\_count int The current number of retry attempts made to engage the user. [​](#example-implementations) Example Implementations -------------------------------------------------------- Here are two example showing how to use the `UserIdleProcessor`: one with the basic callback and one with the retry callback: * Basic Callback * Retry Callback async def handle_idle(user_idle: UserIdleProcessor) -> None: messages.append({ "role": "system", "content": "Ask the user if they are still there and try to prompt for some input." }) await user_idle.push_frame(LLMMessagesFrame(messages)) # Create the processor user_idle = UserIdleProcessor( callback=handle_idle, timeout=5.0 ) # Add to pipeline pipeline = Pipeline([\ transport.input(),\ user_idle, # Add the processor to monitor user activity\ context_aggregator.user(),\ # ... rest of pipeline\ ]) [​](#frame-handling) Frame Handling -------------------------------------- The processor handles the following frame types: * `UserStartedSpeakingFrame`: Marks user as active, resets idle timer and retry count * `UserStoppedSpeakingFrame`: Starts idle monitoring * `BotSpeakingFrame`: Resets idle timer * `EndFrame` / `CancelFrame`: Stops idle monitoring [​](#notes) Notes -------------------- * The idle callback won’t be triggered while the user or bot is actively speaking * The processor automatically cleans up its resources when the pipeline ends * Basic callbacks are supported for backward compatibility [TranscriptProcessor](/server/utilities/transcript-processor) [Overview](/server/frameworks/rtvi/introduction) On this page * [Constructor Parameters](#constructor-parameters) * [Behavior](#behavior) * [Properties](#properties) * [Example Implementations](#example-implementations) * [Frame Handling](#frame-handling) * [Notes](#notes) --- # TranscriptProcessor - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Utilities TranscriptProcessor [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) [​](#overview) Overview -------------------------- `TranscriptProcessor` is a factory that creates and manages processors for handling conversation transcripts from both users and assistants. It provides unified access to transcript processors with shared event handling, making it easy to track and respond to conversation updates in real-time. The processor normalizes messages from various LLM services (OpenAI, Anthropic, Google) into a consistent format and emits events when new messages are added to the conversation. Check out the transcript processor examples for [OpenAI](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/28a-transcription-processor-openai.py) , [Anthropic](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/28b-transcript-processor-anthropic.py) , and [Google Gemini](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/28c-transcription-processor-gemini.py) to see it in action. [​](#factory-methods) Factory Methods ---------------------------------------- [​](#param-user) user method Creates/returns a UserTranscriptProcessor instance that handles user messages [​](#param-assistant) assistant method Creates/returns an AssistantTranscriptProcessor instance that handles assistant messages [​](#param-event-handler) event\_handler decorator Registers event handlers that will be applied to both processors [​](#events) Events ---------------------- [​](#param-on-transcript-update) on\_transcript\_update event Emitted when new messages are added to the conversation transcript. Handler receives: * `processor`: The `TranscriptProcessor` instance * `frame`: `TranscriptionUpdateFrame` containing new messages [​](#transcription-messages) Transcription Messages ------------------------------------------------------ Transcription messages are normalized to this format: @dataclass class TranscriptionMessage: role: Literal["user", "assistant"] # Type of message sender content: str # Transcription message timestamp: str | None = None # Message timestamp [​](#pipeline-integration) Pipeline Integration -------------------------------------------------- The TranscriptProcessor is designed to be used in a pipeline to process conversation transcripts in real-time. In the pipeline: * The UserTranscriptProcessor (`transcript.user()`) handles `TranscriptionFrame`s from the STT service * The AssistantTranscriptProcessor (`transcript.assistant()`) handles `TTSTextFrame`s from the TTS service. Pipeline placement: * `transcript.user()`: Place after the STT Service * `transcript.assistant()`: Place after the transport.output() so that spoken words are transcribed See the Basic Usage example below for more details. [​](#usage-examples) Usage Examples -------------------------------------- ### [​](#basic-usage) Basic Usage This example shows the basic usage for the TranscriptProcessor factory. transcript = TranscriptProcessor() pipeline = Pipeline([\ transport.input(),\ stt,\ transcript.user(), # Process user messages\ context_aggregator.user(),\ llm,\ tts,\ transport.output(),\ transcript.assistant(), # Process assistant messages\ context_aggregator.assistant(),\ ]) # Register event handler for transcript updates @transcript.event_handler("on_transcript_update") async def handle_update(processor, frame): for msg in frame.messages: print(f"{msg.role}: {msg.content}") ### [​](#maintaining-conversation-history) Maintaining Conversation History This example extends the basic usage example by showing how to create a custom handler to maintain conversation history and log new messages with timestamps. class TranscriptHandler: def __init__(self): self.messages = [] async def on_transcript_update(self, processor, frame): self.messages.extend(frame.messages) # Log new messages with timestamps for msg in frame.messages: timestamp = f"[{msg.timestamp}] " if msg.timestamp else "" print(f"{timestamp}{msg.role}: {msg.content}") transcript = TranscriptProcessor() handler = TranscriptHandler() @transcript.event_handler("on_transcript_update") async def on_update(processor, frame): await handler.on_transcript_update(processor, frame) [​](#notes) Notes -------------------- * Supports multiple LLM services (OpenAI, Anthropic, Google) * Normalizes message formats from different services * Maintains conversation history with timestamps * Emits events for real-time transcript updates * Thread-safe for concurrent processing [PatternPairAggregator](/server/utilities/text/pattern-pair-aggregator) [UserIdleProcessor](/server/utilities/user-idle-processor) On this page * [Overview](#overview) * [Factory Methods](#factory-methods) * [Events](#events) * [Transcription Messages](#transcription-messages) * [Pipeline Integration](#pipeline-integration) * [Usage Examples](#usage-examples) * [Basic Usage](#basic-usage) * [Maintaining Conversation History](#maintaining-conversation-history) * [Notes](#notes) --- # Function Calling - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Features Function Calling [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) [​](#understanding-function-calling) Understanding Function Calling ---------------------------------------------------------------------- Function calling (also known as tool calling) allows LLMs to request information from external services and APIs. This enables your bot to access real-time data and perform actions that aren’t part of its training data. For example, you could give your bot the ability to: * Check current weather conditions * Look up stock prices * Query a database * Control smart home devices * Schedule appointments Here’s how it works: 1. You define functions the LLM can use and register them to the LLM service used in your pipeline 2. When needed, the LLM requests a function call 3. Your application executes any corresponding functions 4. The result is sent back to the LLM 5. The LLM uses this information in its response [​](#implementation) Implementation -------------------------------------- ### [​](#1-define-functions) 1\. Define Functions Pipecat provides a standardized `FunctionSchema` that works across all supported LLM providers. This makes it easy to define functions once and use them with any provider. #### [​](#using-the-standard-schema-recommended) Using the Standard Schema (Recommended) from pipecat.adapters.schemas.function_schema import FunctionSchema from pipecat.adapters.schemas.tools_schema import ToolsSchema # Define a function using the standard schema weather_function = FunctionSchema( name="get_current_weather", description="Get the current weather in a location", properties={ "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA", }, "format": { "type": "string", "enum": ["celsius", "fahrenheit"], "description": "The temperature unit to use.", }, }, required=["location", "format"] ) # Create a tools schema with your functions tools = ToolsSchema(standard_tools=[weather_function]) # Pass this to your LLM context context = OpenAILLMContext( messages=[{"role": "system", "content": "You are a helpful assistant."}], tools=tools ) The `ToolsSchema` will be automatically converted to the correct format for your LLM provider through adapters. #### [​](#using-provider-specific-formats-alternative) Using Provider-Specific Formats (Alternative) You can also define functions in the provider-specific format if needed: OpenAI Anthropic Gemini from openai.types.chat import ChatCompletionToolParam # OpenAI native format tools = [\ ChatCompletionToolParam(\ type="function",\ function={\ "name": "get_current_weather",\ "description": "Get the current weather",\ "parameters": {\ "type": "object",\ "properties": {\ "location": {\ "type": "string",\ "description": "The city and state, e.g. San Francisco, CA",\ },\ "format": {\ "type": "string",\ "enum": ["celsius", "fahrenheit"],\ "description": "The temperature unit to use.",\ },\ },\ "required": ["location", "format"],\ },\ },\ )\ ] #### [​](#provider-specific-custom-tools) Provider-Specific Custom Tools Some providers support unique tools that don’t fit the standard function schema. For these cases, you can add custom tools: from pipecat.adapters.schemas.tools_schema import AdapterType, ToolsSchema # Standard functions weather_function = FunctionSchema( name="get_current_weather", description="Get the current weather", properties={"location": {"type": "string"}}, required=["location"] ) # Custom Gemini search tool gemini_search_tool = { "web_search": { "description": "Search the web for information" } } # Create a tools schema with both standard and custom tools tools = ToolsSchema( standard_tools=[weather_function], custom_tools={ AdapterType.GEMINI: [gemini_search_tool] } ) See the provider-specific documentation for details on custom tools and their formats. ### [​](#2-register-function-handlers) 2\. Register Function Handlers Register handlers for your functions using the LLM service’s [`register_function` method](/server/base-classes/llm#methods) : llm = OpenAILLMService(api_key="your-api-key", model="gpt-4") # Main function handler - called to execute the function async def fetch_weather_from_api(function_name, tool_call_id, args, llm, context, result_callback): # Fetch weather data from your API weather_data = {"conditions": "sunny", "temperature": "75"} await result_callback(weather_data) # Register the function llm.register_function( "get_current_weather", fetch_weather_from_api, ) ### [​](#3-create-the-pipeline) 3\. Create the Pipeline Include your LLM service in your pipeline with the registered functions: # Initialize the LLM context with your function schemas context = OpenAILLMContext( messages=[{"role": "system", "content": "You are a helpful assistant."}], tools=tools ) # Create the context aggregator to collect the user and assistant context context_aggregator = llm.create_context_aggregator(context) # Create the pipeline pipeline = Pipeline([\ transport.input(), # Input from the transport\ stt, # STT processing\ context_aggregator.user(), # User context aggregation\ llm, # LLM processing\ tts, # TTS processing\ transport.output(), # Output to the transport\ context_aggregator.assistant(), # Assistant context aggregation\ ]) [​](#function-handler-details) Function Handler Details ---------------------------------------------------------- ### [​](#handler-parameters) Handler Parameters * `function_name`: Name of the called function * `tool_call_id`: Unique identifier for the function call * `args`: Arguments passed by the LLM * `llm`: Reference to the LLM service * `context`: Current conversation context * `result_callback`: Async function to return results ### [​](#return-values) Return Values * Return data through the `result_callback` * Return `None` to ignore the function call * Errors should be handled within your function [​](#controlling-function-call-behavior-advanced) Controlling Function Call Behavior (Advanced) -------------------------------------------------------------------------------------------------- When returning results from a function handler, you can control how the LLM processes those results using a `FunctionCallResultProperties` frame. It can be handy to skip a completion when you have back-to-back function calls. Note, if you skip a completion, then you must manually trigger one from the context. ### [​](#properties) Properties [​](#param-run-llm) run\_llm Optional\[bool\] Controls whether the LLM should generate a response after the function call: * `True`: Run LLM after function call (default if no other function calls in progress) * `False`: Don’t run LLM after function call * `None`: Use default behavior [​](#param-on-context-updated) on\_context\_updated Optional\[Callable\[\[\], Awaitable\[None\]\]\] Optional callback that runs after the function result is added to the context ### [​](#example-usage) Example Usage async def fetch_weather_from_api(function_name, tool_call_id, args, llm, context, result_callback): # Fetch weather data weather_data = {"conditions": "sunny", "temperature": "75"} # Don't run LLM after this function call properties = FunctionCallResultProperties(run_llm=False) await result_callback(weather_data, properties=properties) async def query_database(function_name, tool_call_id, args, llm, context, result_callback): # Query database results = await db.query(args["query"]) async def on_update(): await notify_system("Database query complete") # Run LLM after function call and notify when context is updated properties = FunctionCallResultProperties( run_llm=True, on_context_updated=on_update ) await result_callback(results, properties=properties) [​](#next-steps) Next steps ------------------------------ * Check out the [function calling examples](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/14-function-calling.py) to see a complete example for specific LLM providers. * Refer to your LLM providers documentation to learn more about their funciton calling capabilities. [Gemini Multimodal Live](/guides/features/gemini-multimodal-live) [Metrics](/guides/features/metrics) On this page * [Understanding Function Calling](#understanding-function-calling) * [Implementation](#implementation) * [1\. Define Functions](#1-define-functions) * [Using the Standard Schema (Recommended)](#using-the-standard-schema-recommended) * [Using Provider-Specific Formats (Alternative)](#using-provider-specific-formats-alternative) * [Provider-Specific Custom Tools](#provider-specific-custom-tools) * [2\. Register Function Handlers](#2-register-function-handlers) * [3\. Create the Pipeline](#3-create-the-pipeline) * [Function Handler Details](#function-handler-details) * [Handler Parameters](#handler-parameters) * [Return Values](#return-values) * [Controlling Function Call Behavior (Advanced)](#controlling-function-call-behavior-advanced) * [Properties](#properties) * [Example Usage](#example-usage) * [Next steps](#next-steps) --- # API Reference - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation React Native SDK API Reference [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) The Pipecat React Native SDK leverages the Pipecat JavaScript SDK for seamless integration with React Native applications. For detailed information, please reference to the [Javascript SDK docs](/client/js/api-reference/client-constructor) . **Just ensure you use the appropriate transport layer for React Native.** [SDK Introduction](/client/react-native/introduction) [SDK Introduction](/client/ios/introduction) --- # Noise cancellation with Krisp - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Features Noise cancellation with Krisp [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) [​](#overview) Overview -------------------------- This guide will walk you through setting up and using Krisp’s noise reduction capabilities in your Pipecat application. Krisp provides professional-grade noise cancellation that can significantly improve audio quality in real-time communications. To use Krisp’s noise cancellation, you’ll need to obtain their SDK and models through a Krisp developer account. Our Pipecat Krisp module simplifies the integration process, which we’ll cover in this guide. [​](#walkthrough) Walkthrough -------------------------------- ### [​](#get-access-to-krisp-sdk-and-models) Get Access to Krisp SDK and Models 1. Create a Krisp developers account at [krisp.ai/developers](https://krisp.ai/developers) 2. Download the [Krisp Desktop SDK (v7)](https://sdk.krisp.ai/sdk/desktop) that matches your platform: * Linux: `Desktop SDK v7.0.2: Linux` * macOS (ARM): `Desktop SDK v7.0.1: Mac ARM` * macOS (Intel): `Desktop SDK v7.0.1: Mac Intel` * Windows(ARM): `Desktop SDK v7.0.2: Windows ARM` * Windows (x64): `Desktop SDK v7.0.2: Windows` 3. Download the corresponding models. We recommend trying the `Background Voice Cancellation` model. Recommended model for each platform: * Linux (ARM): `hs.c6.f.s.de56df.kw` * Mac (ARM): `outbound-bvc-models-fp16/hs.c6.f.s.de56df.bucharest.kef` * Linux (x86\_64): `outbound-bvc-models-fp32/hs.c6.f.s.de56df.bucharest.kef` * Mac (x86\_64): `outbound-bvc-models-fp32/hs.c6.f.s.de56df.bucharest.kef` * Windows (x86\_64): `outbound-bvc-models-fp32/hs.c6.f.s.de56df.bucharest.kef` ### [​](#install-build-dependencies) Install build dependencies The `pipecat-ai-krisp` module is a python wrapper around Krisp’s C++ SDK. To build the module, you’ll need to install the following dependencies: * macOS * Ubuntu/Debian * Windows # Using Homebrew brew install cmake pybind11 For Windows users: Make sure you have Visual Studio installed with C++ development tools, or alternatively, have the Visual C++ Build Tools installed. ### [​](#install-the-pipecat-ai-krisp-module) Install the pipecat-ai-krisp module In your Pipecat repo, activate your virtual environment: * macOS/Linux * Windows python3 -m venv venv source venv/bin/activate Export the path to your Krisp SDK and model files: * macOS/Linux * Windows # Add to your .env file or shell configuration export KRISP_SDK_PATH=/PATH/TO/KRISP/SDK export KRISP_MODEL_PATH=/PATH/TO/KRISP/MODEL.kef When selecting a `KRISP_MODEL_PATH`, ensure that you’re selecting the actual model file, not just the directory. The path should look something like this: * ARM Linux * Mac ARM * Windows export KRISP_MODEL_PATH=./krisp/hs.c6.f.s.de56df.kw Next, install the `pipecat-ai[krisp]` module, which will automatically build the `pipecat-ai-krisp` python wrapper module: pip install "pipecat-ai[krisp]" ### [​](#test-the-integration) Test the integration You can now test the Krisp integration. The easiest way to do this is to run the foundational example: [07p-interruptible-krisp.py](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07p-interruptible-krisp.py) . You can run a foundational example by running the following command: python examples/foundational/07p-interruptible-krisp.py -u YOUR_DAILY_ROOM_URL #### [​](#important-for-macos-users) Important for macOS users If you’re running on macOS you may receive a security warning about running the script. This is expected. You can allow access by going to `System Settings > Privacy & Security` then click `Allow Anyway` to permit the example to run. After allowing and re-running, you may get a pop-up asking for permission. Select `Open Anyway` to run the script. [​](#usage-example) Usage Example ------------------------------------ Here’s a basic example of how to add Krisp noise reduction to your Pipecat pipeline: from pipecat.audio.filters.krisp_filter import KrispFilter from pipecat.transports.services.daily import DailyParams, DailyTransport # Add to transport configuration transport = DailyTransport( room_url, token, "Audio Bot", DailyParams( audio_in_filter=KrispFilter(), # Enable Krisp noise reduction audio_out_enabled=True, vad_enabled=True, vad_analyzer=SileroVADAnalyzer(), vad_audio_passthrough=True, ), ) [​](#troubleshooting) Troubleshooting ---------------------------------------- Common issues and solutions: 1. **Missing Dependencies** Error: Missing module: pipecat_ai_krisp Solution: Ensure you’ve installed with the krisp extra: `pip install "pipecat-ai[krisp]"` 2. **Model Path Not Found** Error: Model path for KrispAudioProcessor must be provided Solution: Set the `KRISP_MODEL_PATH` environment variable or provide it in the constructor 3. **SDK Path Issues** Error: Cannot find Krisp SDK Solution: Verify `KRISP_SDK_PATH` points to a valid Krisp SDK installation [​](#additional-resources) Additional Resources -------------------------------------------------- * [KrispFilter Reference Documentation](/api-reference/utilities/audio/krisp-filter) * [Example Application](https://github.com/pipecat-ai/pipecat/blob/main/examples/foundational/07p-interruptible-krisp.py) [Metrics](/guides/features/metrics) [Pipecat Flows](/guides/features/pipecat-flows) On this page * [Overview](#overview) * [Walkthrough](#walkthrough) * [Get Access to Krisp SDK and Models](#get-access-to-krisp-sdk-and-models) * [Install build dependencies](#install-build-dependencies) * [Install the pipecat-ai-krisp module](#install-the-pipecat-ai-krisp-module) * [Test the integration](#test-the-integration) * [Important for macOS users](#important-for-macos-users) * [Usage Example](#usage-example) * [Troubleshooting](#troubleshooting) * [Additional Resources](#additional-resources) --- # SDK Introduction - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation iOS SDK SDK Introduction [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) The Pipecat iOS SDK provides a Swift implementation for building voice and multimodal AI applications on iOS. It handles: * Real-time audio streaming * Bot communication and state management * Media device handling * Configuration management * Event handling [​](#installation) Installation ---------------------------------- Add the SDK to your project using Swift Package Manager: // Core SDK .package(url: "https://github.com/pipecat-ai/pipecat-client-ios.git", from: "0.3.0"), // Daily transport implementation .package(url: "https://github.com/pipecat-ai/pipecat-client-ios-daily.git", from: "0.3.0"), Then add the dependencies to your target: .target(name: "YourApp", dependencies: [\ .product(name: "PipecatClientIOS", package: "pipecat-client-ios")\ .product(name: "PipecatClientIOSDaily", package: "pipecat-client-ios-daily")\ ]), [​](#example) Example ------------------------ Here’s a simple example using Daily as the transport layer: import PipecatClientIOS import PipecatClientIOSDaily let clientConfig = [\ ServiceConfig(\ service: "llm",\ options: [\ Option(name: "model", value: .string("meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo")),\ Option(name: "messages", value: .array([\ .object([\ "role" : .string("system"),\ "content": .string("You are a helpful assistant.")\ ])\ ]))\ ]\ ),\ ServiceConfig(\ service: "tts",\ options: [\ Option(name: "voice", value: .string("79a125e8-cd45-4c13-8a67-188112f4dd22"))\ ]\ )\ ] let options = RTVIClientOptions.init( enableMic: true, params: RTVIClientParams( baseUrl: $PIPECAT_API_URL, config: clientConfig ) ) let client = RTVIClient.init( transport: DailyTransport.init(options: configOptions), options: configOptions ) try await client.start() [​](#documentation) Documentation ------------------------------------ [API Reference\ -------------\ \ Complete SDK API documentation](https://docs-ios.pipecat.ai/) [Source\ ------\ \ Pipecat Client iOS](https://github.com/pipecat-ai/pipecat-client-ios/) [Demo\ ----\ \ Simple Chatbot Demo](https://github.com/pipecat-ai/pipecat/tree/main/examples/simple-chatbot/client/ios) [Daily Transport\ ---------------\ \ WebRTC implementation using Daily](./transports/daily) [API Reference](/client/react-native/api-reference) [API Reference](/client/ios/api-reference) On this page * [Installation](#installation) * [Example](#example) * [Documentation](#documentation) --- # SDK Introduction - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Android SDK SDK Introduction [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) The Pipecat Android SDK provides a Kotlin implementation for building voice and multimodal AI applications on Android. It handles: * Real-time audio and video streaming * Bot communication and state management * Media device handling * Configuration management * Event handling [​](#installation) Installation ---------------------------------- Add the dependency for your chosen transport to your `build.gradle` file. For example, to use the Daily transport: implementation "ai.pipecat:daily-transport:0.3.3" [​](#example) Example ------------------------ Here’s a simple example using Daily as the transport layer. Note that the `clientConfig` is optional and depends on what is required by the bot backend. val clientConfig = listOf( ServiceConfig( service = "llm", options = listOf( Option("model", "meta-llama/Meta-Llama-3.1-8B-Instruct-Turbo"), Option("messages", Value.Array( Value.Object( "role" to Value.Str("system"), "content" to Value.Str("You are a helpful assistant.") ) )) ) ), ServiceConfig( service = "tts", options = listOf( Option("voice", "79a125e8-cd45-4c13-8a67-188112f4dd22") ) ) ) val callbacks = object : RTVIEventCallbacks() { override fun onBackendError(message: String) { Log.e(TAG, "Error from backend: $message") } } val options = RTVIClientOptions( services = listOf(ServiceRegistration("llm", "together"), ServiceRegistration("tts", "cartesia")), params = RTVIClientParams(baseUrl = "", config = clientConfig) ) val client = RTVIClient(DailyTransport.Factory(context), callbacks, options) client.connect().await() // Using Coroutines // Or using callbacks: // client.start().withCallback { /* handle completion */ } [​](#documentation) Documentation ------------------------------------ [API Reference\ -------------\ \ Complete SDK API documentation](https://docs-android.rtvi.ai/) [Daily Transport\ ---------------\ \ WebRTC implementation using Daily](./transports/daily) [OpenAIRealTimeWebRTCTransport](/client/ios/transports/openai-webrtc) [API Reference](/client/android/api-reference) On this page * [Installation](#installation) * [Example](#example) * [Documentation](#documentation) --- # Overview - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Telephony Overview [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) [​](#introduction) Introduction ---------------------------------- You can dial-in to your Pipecat bots, and have them dial-out too, across both PSTN and SIP. The technical implementation will depend on your chosen transport and phone number vendor; each will likely have their own methods and events to consider. [​](#which-transport-should-i-use%3F) Which transport should I use? ---------------------------------------------------------------------- This really depends on your project. We have examples that cover both WebRTC (Daily) and Twilio (WebSockets). Pipecat supports multiple different types of media transport: local, WebSockets, WebRTC etc. Given the end-user will very likely be using a mobile device with varying network conditions, we strongly recommend using a WebRTC transport for production-ready experiences. Another advantage of using a WebRTC provider is that they will very likely abstract some of the complexity for handling PSTN and SIP, such as forwarding calls etc. **Please note:** you can configure your Pipecat bots to handle multiple vendors. You could,for example, use both Daily and Twilio as phone number vendors concurrently. [​](#what-are-pstn-and-sip%3F-what-are-the-differences%3F) What are PSTN and SIP? What are the differences? -------------------------------------------------------------------------------------------------------------- PSTN is an abbreviation for traditional phone networks, consisting of the physical phone lines, cables and transmission links. One of the main differences to consider between these two forms of telephony is PSTN operates on a one user per line basis while SIP can have multiple users per line. Depending on your use-case, you may or may not want to have a single phone number that routes users to a specific bot session, something we’ll cover in the following guides. [Pipecat Flows](/guides/features/pipecat-flows) [Dial-in: WebRTC (Daily)](/guides/telephony/daily-webrtc) On this page * [Introduction](#introduction) * [Which transport should I use?](#which-transport-should-i-use%3F) * [What are PSTN and SIP? What are the differences?](#what-are-pstn-and-sip%3F-what-are-the-differences%3F) --- # Pipecat Flows - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Frameworks Pipecat Flows [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) New to building conversational flows? Check out our [Pipecat Flows guide](/guides/features/pipecat-flows) first. [​](#installation) Installation ---------------------------------- Existing Pipecat installation Fresh Pipecat installation pip install pipecat-ai-flows [​](#core-types) Core Types ------------------------------ ### [​](#flowargs) FlowArgs [​](#param-flow-args) FlowArgs Dict\[str, Any\] Type alias for function handler arguments. ### [​](#flowresult) FlowResult [​](#param-flow-result) FlowResult TypedDict Base type for function handler results. Additional fields can be included as needed. Show Fields [​](#param-status) status str Optional status field [​](#param-error) error str Optional error message ### [​](#flowconfig) FlowConfig [​](#param-flow-config) FlowConfig TypedDict Configuration for the entire conversation flow. Show Fields [​](#param-initial-node) initial\_node str required Starting node identifier [​](#param-nodes) nodes Dict\[str, NodeConfig\] required Map of node names to configurations ### [​](#nodeconfig) NodeConfig [​](#param-node-config) NodeConfig TypedDict Configuration for a single node in the flow. Show Fields [​](#param-role-messages) role\_messages List\[dict\] Defines the role or persona of the LLM. Required for the initial node and optional for subsequent nodes. # Example role messages "role_messages": [\ {\ "role": "system",\ "content": "You are a helpful assistant..."\ }\ ], [​](#param-task-messages) task\_messages List\[dict\] required Defines the task for a given node. Required for all nodes. # Example task messages "task_messages": [\ {\ "role": "system", # May be `user` depending on the LLM\ "content": "Ask the user for their name..."\ }\ ], [​](#param-context-strategy) context\_strategy ContextStrategyConfig Strategy for managing context during transitions to this node. # Example context strategy configuration "context_strategy": ContextStrategyConfig( strategy=ContextStrategy.RESET_WITH_SUMMARY, summary_prompt="Summarize the key points discussed so far." ) [​](#param-functions) functions List\[Union\[dict, FlowsFunctionSchema\]\] required LLM function / tool calls, defined either as dictionaries in the format for the specific LLM provider, or as FlowsFunctionSchema objects. In addition to the standard fields, the function call can also contain: * `handler`: a custom handler that executes when the function is called. * `transition_to`: for static flows, specifies the next node to transition to. * `transition_callback`: for dynamic flows, provides a handler to determine the next node. Note: A function cannot have both `transition_to` and `transition_callback`. # Using native format "functions": [\ {\ "type": "function",\ "function": {\ "name": "get_current_movies",\ "handler": get_movies, # Optional custom handler\ "description": "Fetch movies currently playing",\ "parameters": {...},\ "transition_to": "explore_movie" # Static transition\ },\ }\ ] # Using FlowsFunctionSchema "functions": [\ FlowsFunctionSchema(\ name="get_current_movies",\ description="Fetch movies currently playing",\ properties={...},\ required=[...],\ handler=get_movies,\ transition_to="explore_movie"\ )\ ] [​](#param-pre-actions) pre\_actions List\[dict\] Actions that execute before the LLM inference. For example, you can send a message to the TTS to speak a phrase (e.g. “Hold on a moment…”), which may be effective if an LLM function call takes time to execute. # Example pre_actions "pre_actions": [\ {\ "type": "tts_say",\ "text": "Hold on a moment..."\ }\ ], [​](#param-post-actions) post\_actions List\[dict\] Actions that execute after the LLM inference. For example, you can end the conversation. # Example post_actions "post_actions": [\ {\ "type": "end_conversation"\ }\ ] ### [​](#function-handler-types) Function Handler Types [​](#param-legacy-function-handler) LegacyFunctionHandler Callable\[\[FlowArgs\], Awaitable\[FlowResult\]\] Legacy function handler that only receives arguments. [​](#param-flow-function-handler) FlowFunctionHandler Callable\[\[FlowArgs, FlowManager\], Awaitable\[FlowResult\]\] Modern function handler that receives both arguments and FlowManager. ### [​](#contextstrategy) ContextStrategy [​](#param-context-strategy) ContextStrategy Enum Strategy for managing conversation context during node transitions. Show Values [​](#param-append) APPEND str Default strategy. Adds new messages to existing context. [​](#param-reset) RESET str Clears context and starts fresh with new messages. [​](#param-reset-with-summary) RESET\_WITH\_SUMMARY str Resets context but includes an AI-generated summary. ### [​](#contextstrategyconfig) ContextStrategyConfig [​](#param-context-strategy-config) ContextStrategyConfig dataclass Configuration for context management strategy. Show Fields [​](#param-strategy) strategy ContextStrategy required The strategy to use for context management [​](#param-summary-prompt) summary\_prompt Optional\[str\] Required when using RESET\_WITH\_SUMMARY. Prompt text for generating the conversation summary. # Example usage config = ContextStrategyConfig( strategy=ContextStrategy.RESET_WITH_SUMMARY, summary_prompt="Summarize the key points discussed so far." ) ### [​](#flowsfunctionschema) FlowsFunctionSchema [​](#param-flows-function-schema) FlowsFunctionSchema class A standardized schema for defining functions in Pipecat Flows with flow-specific properties. Show Constructor Parameters [​](#param-name) name str required Name of the function [​](#param-description) description str required Description of the function’s purpose [​](#param-properties) properties Dict\[str, Any\] required Dictionary defining properties types and descriptions [​](#param-required) required List\[str\] required List of required parameter names [​](#param-handler) handler Optional\[FunctionHandler\] Function handler to process the function call [​](#param-transition-to) transition\_to Optional\[str\] Target node to transition to after function execution [​](#param-transition-callback) transition\_callback Optional\[Callable\] Callback function for dynamic transitions You cannot specify both `transition_to` and `transition_callback` in the same function schema. Example usage: from pipecat_flows import FlowsFunctionSchema # Define a function schema collect_name_schema = FlowsFunctionSchema( name="collect_name", description="Record the user's name", properties={ "name": { "type": "string", "description": "The user's name" } }, required=["name"], handler=collect_name_handler, transition_to="next_node" ) # Use in node configuration node_config = { "task_messages": [\ {"role": "system", "content": "Ask the user for their name."}\ ], "functions": [collect_name_schema] } # Pass to flow manager await flow_manager.set_node("greeting", node_config) [​](#flowmanager) FlowManager -------------------------------- [​](#param-flow-manager) FlowManager class Main class for managing conversation flows, supporting both static (configuration-driven) and dynamic (runtime-determined) flows. Show Constructor Parameters [​](#param-task) task PipelineTask required Pipeline task for frame queueing [​](#param-llm) llm LLMService required LLM service instance (OpenAI, Anthropic, or Google). Must be initialized with the corresponding pipecat-ai provider dependency installed. [​](#param-context-aggregator) context\_aggregator Any required Context aggregator used for pushing messages to the LLM service [​](#param-tts) tts Optional\[Any\] **Deprecated**: Optional TTS service for voice actions [​](#param-flow-config) flow\_config Optional\[FlowConfig\] Optional static flow configuration [​](#param-context-strategy-1) context\_strategy Optional\[ContextStrategyConfig\] Optional configuration for how context should be managed during transitions. Defaults to APPEND strategy if not specified. ### [​](#methods) Methods [​](#param-initialize) initialize method Initialize the flow with starting messages. Show Raises [​](#param-flow-initialization-error) FlowInitializationError If initialization fails [​](#param-set-node) set\_node method Set up a new conversation node. * For dynamic flows, the application must advance the conversation using `set_node` to set up the next node. * For static flows, `set_node` is triggered by a function call that contains a `transition_to` field. Show Parameters [​](#param-node-id) node\_id str required Identifier for the new node [​](#param-node-config) node\_config NodeConfig required Node configuration including messages, functions, and actions Show Raises [​](#param-flow-error) FlowError If node setup fails [​](#param-register-action) register\_action method Register a handler for a custom action type. Show Parameters [​](#param-action-type) action\_type str required String identifier for the action [​](#param-handler-1) handler Callable required Async or sync function that handles the action [​](#param-get-current-context) get\_current\_context method Get the current conversation context. Returns a list of messages in the current context, including system messages, user messages, and assistant responses. Show Returns [​](#param-messages) messages List\[dict\] List of messages in the current context Show Raises [​](#param-flow-error-1) FlowError If context aggregator is not available Example usage: # Access current conversation context context = flow_manager.get_current_context() # Use in handlers async def process_response(args: FlowArgs) -> FlowResult: context = flow_manager.get_current_context() # Process conversation history return {"status": "success"} ### [​](#state-management) State Management The FlowManager provides a state dictionary for storing conversation data: Access state Access in transitions flow_manager.state: Dict[str, Any] # Store data flow_manager.state["user_age"] = 25 ### [​](#usage-examples) Usage Examples Static Flow Dynamic Flow flow_config: FlowConfig = { "initial_node": "greeting", "nodes": { "greeting": { "role_messages": [\ {\ "role": "system",\ "content": "You are a helpful assistant. Your responses will be converted to audio."\ }\ ], "task_messages": [\ {\ "role": "system",\ "content": "Start by greeting the user and asking for their name."\ }\ ], "functions": [{\ "type": "function",\ "function": {\ "name": "collect_name",\ "handler": collect_name_handler, # Specify handler\ "description": "Record user's name",\ "parameters": {...},\ "transition_to": "next_step" # Specify transition\ }\ }] } } } # Create and initialize the FlowManager flow_manager = FlowManager( task=task, llm=llm, context_aggregator=context_aggregator, flow_config=flow_config ) # Initialize the flow_manager to start the conversation await flow_manager.initialize() [​](#param-node-functions) Node Functions concept Functions that execute operations within a state. from pipecat_flows import FlowArgs, FlowResult async def process_data(args: FlowArgs) -> FlowResult: """Handle data processing within a node.""" data = args["data"] result = await process(data) return { "status": "success", "processed_data": result } # Node configuration with transition { "type": "function", "function": { "name": "process_data", "handler": process_data, "description": "Process user data", "parameters": { "type": "object", "properties": { "data": {"type": "string"} } }, } } [​](#param-edge-functions) Edge Functions concept Functions that create transitions between nodes. Use `transition_to` (static flow) or `transition_callback` (dynamic flow) to specify the target node. # Edge function configuration { "type": "function", "function": { "name": "next_step", "description": "Transition to next node", "parameters": {"type": "object", "properties": {}}, "transition_to": "target_node" # Required: Specify target node (or use transition_callback for dynamic flows) } } ### [​](#function-properties) Function Properties [​](#param-handler-2) handler Optional\[Callable\] Async function that processes data within a node. Can be specified as: * Direct function reference * Either a Callable function or a string with `__function__:` prefix (e.g., `"__function__:process_data"`) to reference a function in the main script Direct Reference Function Token { "type": "function", "function": { "name": "process_data", "handler": process_data, # Callable function "parameters": {...} } } [​](#param-transition-callback-1) transition\_callback Optional\[Callable\] Handler for dynamic flow transitions. Must be an async function with one of these signatures: # New style (recommended) async def handle_transition( args: Dict[str, Any], result: FlowResult, flow_manager: FlowManager ) -> None: """Handle transition to next node.""" if result.available: # Type-safe access to result await flow_manager.set_node("confirm", create_confirmation_node()) else: await flow_manager.set_node( "no_availability", create_no_availability_node(result.alternative_times) ) # Legacy style (supported for backwards compatibility) async def handle_transition( args: Dict[str, Any], flow_manager: FlowManager ) -> None: """Handle transition to next node.""" await flow_manager.set_node("next", create_next_node()) The callback receives: * `args`: Arguments from the function call * `result`: Typed result from the function handler (new style only) * `flow_manager`: Reference to the FlowManager instance Example usage: async def handle_availability_check( args: Dict, result: TimeResult, # Typed result flow_manager: FlowManager ): """Handle availability check and transition based on result.""" if result.available: await flow_manager.set_node("confirm", create_confirmation_node()) else: await flow_manager.set_node( "no_availability", create_no_availability_node(result.alternative_times) ) # Use in function configuration { "type": "function", "function": { "name": "check_availability", "handler": check_availability, "parameters": {...}, "transition_callback": handle_availability_check } } Note: A function cannot have both `transition_to` and `transition_callback`. ### [​](#handler-signatures) Handler Signatures Function handlers can be defined with three different signatures: Modern (Args + FlowManager) Legacy (Args Only) No Arguments async def handler_with_flow_manager(args: FlowArgs, flow_manager: FlowManager) -> FlowResult: """Modern handler that receives both arguments and FlowManager access.""" # Access state previous_data = flow_manager.state.get("stored_data") # Access pipeline resources await flow_manager.task.queue_frame(TTSSpeakFrame("Processing your request...")) # Store data in state for later flow_manager.state["new_data"] = args["input"] return { "status": "success", "result": "Processed with flow access" } The framework automatically detects which signature your handler is using and calls it appropriately. ### [​](#return-types) Return Types Success Response Error Response { "status": "success", "data": "some data" # Optional additional data } ### [​](#provider-specific-formats) Provider-Specific Formats You don’t need to handle these format differences manually - use the standard format and the FlowManager will adapt it for your chosen provider. OpenAI Anthropic Google (Gemini) { "type": "function", "function": { "name": "function_name", "handler": handler, "description": "Description", "parameters": {...} } } [​](#actions) Actions ------------------------ `pre_actions` and `post_actions` are used to manage conversation flow. They are included in the `NodeConfig` and executed before and after the LLM completion, respectively. Three kinds of actions are available: * Pre-canned actions: These actions perform common tasks and require little configuration. * Function actions: These actions run developer-defined functions at the appropriate time. * Custom actions: These are fully developer-defined actions, providing flexibility at the expense of complexity. ### [​](#pre-canned-actions) Pre-canned Actions Common actions shipped with Flows for managing conversation flow. To use them, just add them to your `NodeConfig`. [​](#param-tts-say) tts\_say action Speaks text immediately using the TTS service. "pre_actions": [\ {\ "type": "tts_say",\ "text": "Processing your request..." # Required\ }\ ] [​](#param-end-conversation) end\_conversation action Ends the conversation and closes the connection. "post_actions": [\ {\ "type": "end_conversation",\ "text": "Goodbye!" # Optional farewell message\ }\ ] ### [​](#function-actions) Function Actions Actions that run developer-defined functions at the appropriate time. For example, if used in `post_actions`, they’ll run after the bot has finished talking and after any previous `post_actions` have finished. [​](#param-function) function action Calls the developer-defined function at the appropriate time. "post_actions": [\ {\ "type": "function",\ "handler": bot_turn_ended # Required\ }\ ] ### [​](#custom-actions) Custom Actions Fully developer-defined actions, providing flexibility at the expense of complexity. Here’s the complexity: because these actions aren’t queued in the Pipecat pipeline, they may execute seemingly early if used in `post_actions`; they’ll run immediately after the LLM completion is triggered but won’t wait around for the bot to finish talking. Why would you want this behavior? You might be writing an action that: * Itself just queues another `Frame` into the Pipecat pipeline (meaning there would no benefit to waiting around for sequencing purposes) * Does work that can be done a bit sooner, like logging that the LLM was updated Custom actions are composed of at least: [​](#param-type) type str required String identifier for the action [​](#param-handler-3) handler Callable required Async or sync function that handles the action Example: # Define custom action handler async def custom_notification(action: dict, flow_manager: FlowManager): """Custom action handler.""" message = action.get("message", "") await notify_user(message) # Use in node configuration "pre_actions": [\ {\ "type": "notify",\ "handler": send_notification,\ "message": "Attention!",\ }\ ] [​](#exceptions) Exceptions ------------------------------ [​](#param-flow-error-2) FlowError exception Base exception for all flow-related errors. from pipecat_flows import FlowError try: await flow_manager.set_node("invalid", config) except FlowError as e: print(f"Flow error: {e}") [​](#param-flow-initialization-error-1) FlowInitializationError exception Raised when flow initialization fails. from pipecat_flows import FlowInitializationError try: await flow_manager.initialize() except FlowInitializationError as e: print(f"Initialization failed: {e}") [​](#param-flow-transition-error) FlowTransitionError exception Raised when a state transition fails. from pipecat_flows import FlowTransitionError try: await flow_manager.set_node("next", node_config) except FlowTransitionError as e: print(f"Transition failed: {e}") [​](#param-invalid-function-error) InvalidFunctionError exception Raised when an invalid or unavailable function is called. from pipecat_flows import InvalidFunctionError try: await flow_manager.set_node("node", { "functions": [{\ "type": "function",\ "function": {\ "name": "invalid_function"\ }\ }] }) except InvalidFunctionError as e: print(f"Invalid function: {e}") [RTVI Frame Processors](/server/frameworks/rtvi/frame-processors) [PipelineParams](/server/pipeline/pipeline-params) On this page * [Installation](#installation) * [Core Types](#core-types) * [FlowArgs](#flowargs) * [FlowResult](#flowresult) * [FlowConfig](#flowconfig) * [NodeConfig](#nodeconfig) * [Function Handler Types](#function-handler-types) * [ContextStrategy](#contextstrategy) * [ContextStrategyConfig](#contextstrategyconfig) * [FlowsFunctionSchema](#flowsfunctionschema) * [FlowManager](#flowmanager) * [Methods](#methods) * [State Management](#state-management) * [Usage Examples](#usage-examples) * [Function Properties](#function-properties) * [Handler Signatures](#handler-signatures) * [Return Types](#return-types) * [Provider-Specific Formats](#provider-specific-formats) * [Actions](#actions) * [Pre-canned Actions](#pre-canned-actions) * [Function Actions](#function-actions) * [Custom Actions](#custom-actions) * [Exceptions](#exceptions) --- # SDK Introduction - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation SDK Introduction [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) The Pipecat C++ SDK provides a native implementation for building voice and multimodal AI applications. It supports: * Linux (`x86_64` and `aarch64`) * macOS (`aarch64`) * Windows (`x86_64`) [​](#dependencies) Dependencies ---------------------------------- ### [​](#libcurl) libcurl The SDK uses [libcurl](https://curl.se/libcurl/) for HTTP requests. * Linux * macOS * Windows sudo apt-get install libcurl4-openssl-dev [​](#installation) Installation ---------------------------------- Build the SDK using CMake: * Linux/macOS * Windows cmake . -G Ninja -Bbuild -DCMAKE_BUILD_TYPE=Release ninja -C build ### [​](#cross-compilation) Cross-compilation For Linux aarch64: cmake . -G Ninja -Bbuild -DCMAKE_TOOLCHAIN_FILE=aarch64-linux-toolchain.cmake -DCMAKE_BUILD_TYPE=Release ninja -C build [​](#documentation) Documentation ------------------------------------ [API Reference\ -------------\ \ Complete SDK API documentation](https://github.com/pipecat-ai/pipecat-client-cxx) [Daily Transport\ ---------------\ \ WebRTC implementation using Daily](./transport) On this page * [Dependencies](#dependencies) * [libcurl](#libcurl) * [Installation](#installation) * [Cross-compilation](#cross-compilation) * [Documentation](#documentation) --- # Dial-in: WebRTC (Daily) - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Telephony Dial-in: WebRTC (Daily) [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) [​](#things-you%E2%80%99ll-need) Things you’ll need ------------------------------------------------------ * An active [Daily](https://www.daily.co) developer key. * One or more Daily provisioned phone numbers (covered below). [Prefer to look at code? See the example project!\ ------------------------------------------------\ \ We have a complete dialin-ready project using Daily as both a transport and PSTN/SIP provider in the Pipecat repo. This guide referencse the project and steps through the important parts that make dial-in work.](https://github.com/pipecat-ai/pipecat/tree/main/examples/phone-chatbot) ### [​](#do-i-need-to-provision-my-phone-numbers-through-daily%3F) Do I need to provision my phone numbers through Daily? You can use Daily solely as a transport if you prefer. This is particularly useful if you already have Twilio-provisioned numbers and workflows. In that case, you can configure Twilio to forward calls to your Pipecat agents and join a Daily WebRTC call. More details on using Twilio with Daily as a transport can be found [here](/telephony/twilio-daily-webrtc) . If you’re starting from scratch, using everything on one platform offers some convenience. By provisioning your phone numbers through Daily and using Daily as the transport layer, you won’t need to worry about initial call routing. [​](#purchasing-a-phone-number) Purchasing a phone number ------------------------------------------------------------ You can purchase a number via the Daily REST API Purchase a random number Purchase specific number List numbers curl --request POST \ --url 'https://api.daily.co/v1/buy-phone-number' \ --header 'Authorization: Bearer [YOUR_DAILY_API_KEY]' \ --header 'Content-Type: application/json' [​](#configuring-your-bot-runner) Configuring your bot runner ---------------------------------------------------------------- You’ll need a HTTP service that can receive incoming call hooks and trigger a new agent session. We discussed the concept of a [bot runner](/deployment/pattern) in the deployment section, which we’ll build on here to add support for incoming phone calls. Within the `start_bot` method, we’ll need to grab both `callId` and `callDomain` from the incoming web request that is triggered by Daily when someone dials the number: bot\_daily.py # Get the dial-in properties from the request try: data = await request.json() callId = data.get("callId") callDomain = data.get("callDomain") except Exception: raise HTTPException( status_code=500, detail="Missing properties 'callId' or 'callDomain'") Full bot source code [here](https://github.com/pipecat-ai/pipecat/blob/main/examples/phone-chatbot/bot_daily.py) ### [​](#orchestrating-incoming-calls) Orchestrating incoming calls Daily needs a URL / webhook endpoint it can trigger when a user dials the phone number. We can configure this by assigning the number to an endpoint via their REST API. Here is an example: curl --location 'https://api.daily.co/v1' \ --header 'Content-Type: application/json' \ --header 'Authorization: Bearer [DAILY API TOKEN HERE]' \ --data '{ "properties": { "pinless_dialin": [\ {\ "phone_number": "[DAILY PROVISIONED NUMBER HERE]",\ "room_creation_api": "[BOT RUNNER URL]/start_bot"\ }\ ] } }' If you want to test locally, you can expose your web method using a service such as [ngrok](https://ngrok.com/) . Example ngrok tunnel python bot_runner.py --host localhost --port 7860 --reload ngrok http localhost:7860 # E.g: https://123.ngrok.app/start_bot ### [​](#creating-a-new-sip-enabled-room) Creating a new SIP-enabled room We’ll need to configure the Daily room to be setup to receive SIP connections. `daily-helpers.py` included in Pipecat has some useful imports that make this easy. We just need to pass through new SIP parameters as part of room creation: bot\_runner.py from pipecat.transports.services.helpers.daily_rest import DailyRoomParams, DailyRoomProperties, DailyRoomSipParams params = DailyRoomParams( properties=DailyRoomProperties( sip=DailyRoomSipParams( display_name = "sip-dialin" video = False sip_mode = "dial-in" num_endpoints = 1 ) ) ) # Create sip-enabled Daily room via REST try: room: DailyRoomObject = daily_rest_helper.create_room(params=params) except Exception as e: raise HTTPException( status_code=500, detail=f"Unable to provision room {e}") print (f"Daily room returned {room.url} {room.config.sip_endpoint}") Incoming calls will include both `callId` and `callDomain` properties in the body of the request; we’ll need to pass to the Pipecat agent. For simplicity, our agents are spawned as sub-processes of the bot runner, so we’ll pass the callId and callDomain through as command line arguments: bot\_runner.py proc = subprocess.Popen( [\ f"python3 -m bot_daily -u {room.url} -t {token} -i {callId} -d {callDomain}"\ ], shell=True, bufsize=1, cwd=os.path.dirname(os.path.abspath(__file__)) ) That’s all the configuration we need in our `bot_runner.py`. [​](#configuring-your-pipecat-bot) Configuring your Pipecat bot ------------------------------------------------------------------ Let’s take a look at `bot_daily.py` and step through the differences from other examples. First, it’s setup to receive additional command line parameters which are passed through to the `DailyTransport` object: bot\_daily.py # ... async def main(room_url: str, token: str, callId: str, callDomain: str): async with aiohttp.ClientSession() as session: diallin_settings = DailyDialinSettings( call_id=callId, call_domain=callDomain ) transport = DailyTransport( room_url, token, "Chatbot", DailyParams( api_url=daily_api_url, api_key=daily_api_key, dialin_settings=diallin_settings, audio_in_enabled=True, audio_out_enabled=True, camera_out_enabled=False, vad_enabled=True, vad_analyzer=SileroVADAnalyzer(), transcription_enabled=True, ) ) # ... your bot code if __name__ == "__main__": parser = argparse.ArgumentParser(description="Pipecat Simple ChatBot") parser.add_argument("-u", type=str, help="Room URL") parser.add_argument("-t", type=str, help="Token") parser.add_argument("-i", type=str, help="Call ID") parser.add_argument("-d", type=str, help="Call Domain") config = parser.parse_args() asyncio.run(main(config.u, config.t, config.i, config.d)) Optionally, we can listen and respond to the `on_dialin_ready` event manually. This is useful if you have specific scenarios in whih you want to indicates that the SIP worker and is ready to be forwarded to the call. This would stop any hold music and connect the end-user to our Pipecat bot. @transport.event_handler("on_dialin_ready") async def on_dialin_ready(transport, cdata): print(f"on_dialin_ready", cdata) Since we’re using Daily as a phone vendor, this method is handled internally by the Pipecat Daily service. It can, however, be useful to override this default behaviour if you want to configure your bot in a certain way as soon as the call is ready. Typically, however, initial setup is done in the `on_first_participant_joined` event after the user has joined the session. @transport.event_handler("on_first_participant_joined") async def on_first_participant_joined(transport, participant): transport.capture_participant_transcription(participant["id"]) await task.queue_frames([LLMMessagesFrame(messages)]) [Overview](/guides/telephony/overview) [Dial-in: WebRTC (Twilio + Daily)](/guides/telephony/twilio-daily-webrtc) On this page * [Things you’ll need](#things-you%E2%80%99ll-need) * [Do I need to provision my phone numbers through Daily?](#do-i-need-to-provision-my-phone-numbers-through-daily%3F) * [Purchasing a phone number](#purchasing-a-phone-number) * [Configuring your bot runner](#configuring-your-bot-runner) * [Orchestrating incoming calls](#orchestrating-incoming-calls) * [Creating a new SIP-enabled room](#creating-a-new-sip-enabled-room) * [Configuring your Pipecat bot](#configuring-your-pipecat-bot) --- # PipelineParams - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Pipeline PipelineParams [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) [​](#overview) Overview -------------------------- The `PipelineParams` class provides a structured way to configure various aspects of pipeline execution. These parameters control behaviors like audio settings, metrics collection, heartbeat monitoring, and interruption handling. [​](#basic-usage) Basic Usage -------------------------------- from pipecat.pipeline.task import PipelineParams, PipelineTask # Create with default parameters params = PipelineParams() # Or customize specific parameters params = PipelineParams( allow_interruptions=True, audio_in_sample_rate=16000, enable_metrics=True ) # Pass to PipelineTask pipeline = Pipeline([...]) task = PipelineTask(pipeline, params=params) [​](#available-parameters) Available Parameters -------------------------------------------------- [​](#param-allow-interruptions) allow\_interruptions bool default:"False" Whether to allow pipeline interruptions. When enabled, a user’s speech will immediately interrupt the bot’s response. [​](#param-audio-in-sample-rate) audio\_in\_sample\_rate int default:"16000" Input audio sample rate in Hz. Setting the `audio_in_sample_rate` as a `PipelineParam` sets the input sample rate for all corresponding services in the pipeline. [​](#param-audio-out-sample-rate) audio\_out\_sample\_rate int default:"24000" Output audio sample rate in Hz. Setting the `audio_out_sample_rate` as a `PipelineParam` sets the output sample rate for all corresponding services in the pipeline. [​](#param-enable-heartbeats) enable\_heartbeats bool default:"False" Whether to enable heartbeat monitoring to detect pipeline stalls. See [Heartbeats](/server/pipeline/heartbeats) for details. [​](#param-heartbeats-period-secs) heartbeats\_period\_secs float default:"1.0" Period between heartbeats in seconds (when heartbeats are enabled). [​](#param-enable-metrics) enable\_metrics bool default:"False" Whether to enable metrics collection for pipeline performance. [​](#param-enable-usage-metrics) enable\_usage\_metrics bool default:"False" Whether to enable usage metrics tracking. [​](#param-report-only-initial-ttfb) report\_only\_initial\_ttfb bool default:"False" Whether to report only initial time to first byte metric. [​](#param-send-initial-empty-metrics) send\_initial\_empty\_metrics bool default:"True" Whether to send initial empty metrics frame at pipeline start. [​](#param-start-metadata) start\_metadata Dict\[str, Any\] default:"{}" Additional metadata to include in the StartFrame. [​](#common-configurations) Common Configurations ---------------------------------------------------- ### [​](#audio-processing-configuration) Audio Processing Configuration You can set the audio input and output sample rates in the `PipelineParams` to set the sample rate for all input and output services in the pipeline. This acts as a convenience to avoid setting the sample rate for each service individually. Note, if services are set individually, they will supersede the values set in `PipelineParams`. params = PipelineParams( audio_in_sample_rate=8000, # Lower quality input audio audio_out_sample_rate=8000 # High quality output audio ) ### [​](#performance-monitoring-configuration) Performance Monitoring Configuration Pipeline heartbeats provide a way to monitor the health of your pipeline by sending periodic heartbeat frames through the system. When enabled, the pipeline will send heartbeat frames every second and monitor their progress through the pipeline. params = PipelineParams( enable_heartbeats=True, heartbeats_period_secs=2.0, # Send heartbeats every 2 seconds enable_metrics=True ) [​](#how-parameters-are-used) How Parameters Are Used -------------------------------------------------------- The parameters you set in `PipelineParams` are passed to various components of the pipeline: 1. **StartFrame**: Many parameters are included in the StartFrame that initializes the pipeline 2. **Metrics Collection**: Metrics settings configure what performance data is gathered 3. **Heartbeat Monitoring**: Controls the pipeline’s health monitoring system 4. **Audio Processing**: Sample rates affect how audio is processed throughout the pipeline [​](#complete-example) Complete Example ------------------------------------------ from pipecat.frames.frames import TTSSpeakFrame from pipecat.observers.file_observer import FileObserver from pipecat.pipeline.task import PipelineParams, PipelineTask from pipecat.pipeline.runner import PipelineRunner # Create comprehensive parameters params = PipelineParams( allow_interruptions=True, audio_in_sample_rate=8000, audio_out_sample_rate=8000, enable_heartbeats=True, enable_metrics=True, enable_usage_metrics=True, heartbeats_period_secs=1.0, report_only_initial_ttfb=False, start_metadata={ "conversation_id": "conv-123", "session_data": { "user_id": "user-456", "start_time": "2023-10-25T14:30:00Z" } } ) # Create pipeline and task pipeline = Pipeline([...]) task = PipelineTask( pipeline, params=params, observers=[FileObserver("pipeline_logs.jsonl")] ) # Run the pipeline runner = PipelineRunner() await runner.run(task) [​](#additional-information) Additional Information ------------------------------------------------------ * Parameters are immutable once the pipeline starts * The `start_metadata` dictionary can contain any serializable data * For metrics collection to work properly, `enable_metrics` must be set to `True` [Pipecat Flows](/server/frameworks/flows/pipecat-flows) [PipelineTask](/server/pipeline/pipeline-task) On this page * [Overview](#overview) * [Basic Usage](#basic-usage) * [Available Parameters](#available-parameters) * [Common Configurations](#common-configurations) * [Audio Processing Configuration](#audio-processing-configuration) * [Performance Monitoring Configuration](#performance-monitoring-configuration) * [How Parameters Are Used](#how-parameters-are-used) * [Complete Example](#complete-example) * [Additional Information](#additional-information) --- # PipelineTask - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Pipeline PipelineTask [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) [​](#overview) Overview -------------------------- `PipelineTask` is the central class for managing pipeline execution. It handles the lifecycle of the pipeline, processes frames in both directions, manages task cancellation, and provides event handlers for monitoring pipeline activity. [​](#basic-usage) Basic Usage -------------------------------- from pipecat.pipeline.pipeline import Pipeline from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask # Create a pipeline pipeline = Pipeline([...]) # Create a task with the pipeline task = PipelineTask(pipeline) # Queue frames for processing await task.queue_frame(TTSSpeakFrame("Hello, how can I help you today?")) # Run the pipeline runner = PipelineRunner() await runner.run(task) [​](#constructor-parameters) Constructor Parameters ------------------------------------------------------ [​](#param-pipeline) pipeline BasePipeline required The pipeline to execute. [​](#param-params) params PipelineParams default:"PipelineParams()" Configuration parameters for the pipeline. See [PipelineParams](/server/pipeline/pipeline-params) for details. [​](#param-observers) observers List\[BaseObserver\] default:"\[\]" List of observers for monitoring pipeline execution. See [Observers](/server/utilities/observers/observer-pattern) for details. [​](#param-clock) clock BaseClock default:"SystemClock()" Clock implementation for timing operations. [​](#param-task-manager) task\_manager Optional\[BaseTaskManager\] default:"None" Custom task manager for handling asyncio tasks. If None, a default TaskManager is used. [​](#param-check-dangling-tasks) check\_dangling\_tasks bool default:"True" Whether to check for processors’ tasks finishing properly. [​](#param-idle-timeout-secs) idle\_timeout\_secs Optional\[float\] default:"300" Timeout in seconds before considering the pipeline idle. Set to None to disable idle detection. See [Pipeline Idle Detection](/server/pipeline/pipeline-idle-detection) for details. [​](#param-idle-timeout-frames) idle\_timeout\_frames Tuple\[Type\[Frame\], ...\] default:"(BotSpeakingFrame, LLMFullResponseEndFrame)" Frame types that should prevent the pipeline from being considered idle. See [Pipeline Idle Detection](/server/pipeline/pipeline-idle-detection) for details. [​](#param-cancel-on-idle-timeout) cancel\_on\_idle\_timeout bool default:"True" Whether to automatically cancel the pipeline task when idle timeout is reached. See [Pipeline Idle Detection](/server/pipeline/pipeline-idle-detection) for details. [​](#methods) Methods ------------------------ ### [​](#task-lifecycle-management) Task Lifecycle Management [​](#param-run) run() async Starts and manages the pipeline execution until completion or cancellation. await task.run() [​](#param-stop-when-done) stop\_when\_done() async Sends an EndFrame to the pipeline to gracefully stop the task after all queued frames have been processed. await task.stop_when_done() [​](#param-cancel) cancel() async Stops the running pipeline immediately by sending a CancelFrame. await task.cancel() [​](#param-has-finished) has\_finished() bool Returns whether the task has finished (all processors have stopped). if task.has_finished(): print("Task is complete") ### [​](#frame-management) Frame Management [​](#param-queue-frame) queue\_frame() async Queues a single frame to be pushed down the pipeline. await task.queue_frame(TTSSpeakFrame("Hello!")) [​](#param-queue-frames) queue\_frames() async Queues multiple frames to be pushed down the pipeline. frames = [TTSSpeakFrame("Hello!"), TTSSpeakFrame("How are you?")] await task.queue_frames(frames) [​](#event-handlers) Event Handlers -------------------------------------- PipelineTask provides an event handler that can be registered using the `event_handler` decorator: ### [​](#on-idle-timeout) on\_idle\_timeout Triggered when no activity frames (as specified by `idle_timeout_frames`) have been received within the idle timeout period. @task.event_handler("on_idle_timeout") async def on_idle_timeout(task): print("Pipeline has been idle too long") await task.queue_frame(TTSSpeakFrame("Are you still there?")) [PipelineParams](/server/pipeline/pipeline-params) [Pipeline Idle Detection](/server/pipeline/pipeline-idle-detection) On this page * [Overview](#overview) * [Basic Usage](#basic-usage) * [Constructor Parameters](#constructor-parameters) * [Methods](#methods) * [Task Lifecycle Management](#task-lifecycle-management) * [Frame Management](#frame-management) * [Event Handlers](#event-handlers) * [on\_idle\_timeout](#on-idle-timeout) --- # Pipeline Idle Detection - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Pipeline Pipeline Idle Detection [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) [​](#overview) Overview -------------------------- Pipeline idle detection is a feature that monitors activity in your pipeline and can automatically cancel tasks when no meaningful bot interactions are occurring. This helps prevent pipelines from running indefinitely when a conversation has naturally ended but wasn’t properly terminated. [​](#how-it-works) How It Works ---------------------------------- The system monitors specific “activity frames” that indicate the bot is actively engaged in the conversation. By default, these are: * `BotSpeakingFrame` - When the bot is speaking * `LLMFullResponseEndFrame` - When the LLM has completed a response If no activity frames are detected within the configured timeout period (5 minutes by default), the system considers the pipeline idle and can automatically terminate it. Idle detection only starts after the pipeline has begun processing frames. The idle timer resets whenever an activity frame (as specified in `idle_timeout_frames`) is received. [​](#configuration) Configuration ------------------------------------ You can configure idle detection behavior when creating a `PipelineTask`: from pipecat.pipeline.task import PipelineParams, PipelineTask # Default configuration - cancel after 5 minutes of inactivity task = PipelineTask(pipeline) # Custom configuration task = PipelineTask( pipeline, params=PipelineParams(allow_interruptions=True), idle_timeout_secs=600, # 10 minute timeout idle_timeout_frames=(BotSpeakingFrame,), # Only monitor bot speaking cancel_on_idle_timeout=False, # Don't auto-cancel, just notify ) [​](#configuration-parameters) Configuration Parameters ---------------------------------------------------------- [​](#param-idle-timeout-secs) idle\_timeout\_secs Optional\[float\] default:"300" Timeout in seconds before considering the pipeline idle. Set to `None` to disable idle detection. [​](#param-idle-timeout-frames) idle\_timeout\_frames Tuple\[Type\[Frame\], ...\] default:"(BotSpeakingFrame, LLMFullResponseEndFrame)" Frame types that should prevent the pipeline from being considered idle. [​](#param-cancel-on-idle-timeout) cancel\_on\_idle\_timeout bool default:"True" Whether to automatically cancel the pipeline task when idle timeout is reached. [​](#handling-idle-timeouts) Handling Idle Timeouts ------------------------------------------------------ You can respond to idle timeout events by adding an event handler: @task.event_handler("on_idle_timeout") async def on_idle_timeout(task): logger.info("Pipeline has been idle for too long") # Perform any custom cleanup or logging # Note: If cancel_on_idle_timeout=True, the pipeline will be cancelled after this handler runs [​](#example-implementation) Example Implementation ------------------------------------------------------ Here’s a complete example showing how to configure idle detection with custom handling: from pipecat.frames.frames import BotSpeakingFrame, LLMFullResponseEndFrame, TTSSpeakFrame from pipecat.pipeline.runner import PipelineRunner from pipecat.pipeline.task import PipelineParams, PipelineTask # Create pipeline pipeline = Pipeline([...]) # Configure task with custom idle settings task = PipelineTask( pipeline, params=PipelineParams(allow_interruptions=True), idle_timeout_secs=180, # 3 minutes cancel_on_idle_timeout=False # Don't auto-cancel ) # Add event handler for idle timeout @task.event_handler("on_idle_timeout") async def on_idle_timeout(task): logger.info("Conversation has been idle for 3 minutes") # Add a farewell message await task.queue_frame(TTSSpeakFrame("I haven't heard from you in a while. Goodbye!")) # Then end the conversation gracefully await task.stop_when_done() runner = PipelineRunner() await runner.run(task) [PipelineTask](/server/pipeline/pipeline-task) [Pipeline Heartbeats](/server/pipeline/heartbeats) On this page * [Overview](#overview) * [How It Works](#how-it-works) * [Configuration](#configuration) * [Configuration Parameters](#configuration-parameters) * [Handling Idle Timeouts](#handling-idle-timeouts) * [Example Implementation](#example-implementation) --- # Speech Service - Pipecat [Pipecat home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/light.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/daily/logo/dark.svg)](/) Search or ask... Search... Navigation Base Service Classes Speech Service [Getting Started](/getting-started/overview) [Guides](/guides/introduction) [Server APIs](/server/introduction) [Client SDKs](/client/introduction) [​](#sttservice) STTService ------------------------------ [​](#param-stt-service) STTService base class Base class for [Speech-to-Text services](/server/services/supported-services#speech-to-text) . ### [​](#constructor-parameters) Constructor Parameters [​](#param-audio-passthrough) audio\_passthrough bool default:"False" Whether to pass through audio frames ### [​](#methods) Methods [​](#param-set-model) set\_model async method Sets the STT model. @abstractmethod async def set_model(self, model: str) [​](#param-set-language) set\_language async method Sets the recognition language. @abstractmethod async def set_language(self, language: Language) [​](#ttsservice) TTSService ------------------------------ [​](#param-tts-service) TTSService base class Base class for [Text-to-Speech services](/server/services/supported-services#text-to-speech) . ### [​](#constructor-parameters-2) Constructor Parameters [​](#param-sample-rate) sample\_rate int default:"None" Output audio sample rate in Hz [​](#param-text-filter) text\_filter Optional\[BaseTextFilter\] default:"None" Text filter for preprocessing ### [​](#methods-2) Methods [​](#param-set-model-1) set\_model async method Sets the TTS model. @abstractmethod async def set_model(self, model: str) [​](#param-set-voice) set\_voice method Sets the voice identifier. @abstractmethod def set_voice(self, voice: str) [​](#param-say) say async method Converts text to speech immediately. async def say(self, text: str) [Transport](/server/base-classes/transport) [LLM Service](/server/base-classes/llm) On this page * [STTService](#sttservice) * [Constructor Parameters](#constructor-parameters) * [Methods](#methods) * [TTSService](#ttsservice) * [Constructor Parameters](#constructor-parameters-2) * [Methods](#methods-2) ---