# Table of Contents - [Entity Extraction - DSPy](#entity-extraction-dspy) - [FAQ - DSPy](#faq-dspy) - [Community Resources - DSPy](#community-resources-dspy) - [Learning DSPy - DSPy](#learning-dspy-dspy) - [DSPy](#dspy) - [Tutorials Overview - DSPy](#tutorials-overview-dspy) - [Adapter - DSPy](#adapter-dspy) - [Programming Overview - DSPy](#programming-overview-dspy) - [Language Models - DSPy](#language-models-dspy) - [Modules - DSPy](#modules-dspy) - [Signatures - DSPy](#signatures-dspy) - [Evaluation Overview - DSPy](#evaluation-overview-dspy) - [Data Handling - DSPy](#data-handling-dspy) - [Metrics - DSPy](#metrics-dspy) - [Optimization Overview - DSPy](#optimization-overview-dspy) - [Optimizers - DSPy](#optimizers-dspy) - [ChromadbRM - DSPy](#chromadbrm-dspy) - [DatabricksRM - DSPy](#databricksrm-dspy) - [Azure - DSPy](#azure-dspy) - [FalkordbRM - DSPy](#falkordbrm-dspy) - [ClarifaiRM - DSPy](#clarifairm-dspy) - [ColBERTv2 - DSPy](#colbertv2-dspy) - [FaissRM - DSPy](#faissrm-dspy) - [Custom RM Client - DSPy](#custom-rm-client-dspy) - [LancedbRM - DSPy](#lancedbrm-dspy) - [MyScaleRM - DSPy](#myscalerm-dspy) - [MilvusRM - DSPy](#milvusrm-dspy) - [RAGatouilleRM - DSPy](#ragatouillerm-dspy) - [Neo4jRM - DSPy](#neo4jrm-dspy) - [QdrantRM - DSPy](#qdrantrm-dspy) - [WatsonDiscovery - DSPy](#watsondiscovery-dspy) - [SnowflakeRM - DSPy](#snowflakerm-dspy) - [WeaviateRM - DSPy](#weaviaterm-dspy) - [Retrieval-Augmented Generation - DSPy](#retrieval-augmented-generation-dspy) - [YouRM - DSPy](#yourm-dspy) - [Saving and Loading - DSPy](#saving-and-loading-dspy) - [Agents - DSPy](#agents-dspy) - [Multi-Hop Search - DSPy](#multi-hop-search-dspy) - [Finetuning Agents - DSPy](#finetuning-agents-dspy) - [Privacy-Conscious Delegation - DSPy](#privacy-conscious-delegation-dspy) - [Classification - DSPy](#classification-dspy) - [Use Cases - DSPy](#use-cases-dspy) - [Deployment - DSPy](#deployment-dspy) - [Debugging & Observability - DSPy](#debugging-observability-dspy) - [Reasoning - DSPy](#reasoning-dspy) - [Roadmap - DSPy](#roadmap-dspy) - [Contributing - DSPy](#contributing-dspy) --- # Entity Extraction - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/tutorials/entity_extraction/index.ipynb "Edit this page") Tutorial: Entity Extraction[¶](#tutorial-entity-extraction) ============================================================ This tutorial demonstrates how to perform **entity extraction** using the CoNLL-2003 dataset with DSPy. The focus is on extracting entities referring to people. We will: * Extract and label entities from the CoNLL-2003 dataset that refer to people * Define a DSPy program for extracting entities that refer to people * Optimize and evaluate the program on a subset of the CoNLL-2003 dataset By the end of this tutorial, you'll understand how to structure tasks in DSPy using signatures and modules, evaluate your system's performance, and improve its quality with optimizers. Install the latest version of DSPy and follow along. If you're looking instead for a conceptual overview of DSPy, this [recent lecture](https://www.youtube.com/live/JEMYuzrKLUw) is a good place to start. In \[1\]: Copied! \# Install the latest version of DSPy %pip install \-U dspy\-ai \# Install the Hugging Face datasets library to load the CoNLL-2003 dataset %pip install datasets \# Install the latest version of DSPy %pip install -U dspy-ai # Install the Hugging Face datasets library to load the CoNLL-2003 dataset %pip install datasets Load and Prepare the Dataset[¶](#load-and-prepare-the-dataset) --------------------------------------------------------------- In this section, we prepare the CoNLL-2003 dataset, which is commonly used for entity extraction tasks. The dataset includes tokens annotated with entity labels such as persons, organizations, and locations. We will: 1. Load the dataset using the Hugging Face `datasets` library. 2. Define a function to extract tokens referring to people. 3. Slice the dataset to create smaller subsets for training and testing. DSPy expects examples in a structured format, so we'll also transform the dataset into DSPy `Examples` for easy integration. In \[2\]: Copied! import os import tempfile from datasets import load\_dataset from typing import Dict, Any, List import dspy def load\_conll\_dataset() \-> dict: """ Loads the CoNLL-2003 dataset into train, validation, and test splits. Returns: dict: Dataset splits with keys 'train', 'validation', and 'test'. """ with tempfile.TemporaryDirectory() as temp\_dir: \# Use a temporary Hugging Face cache directory for compatibility with certain hosted notebook \# environments that don't support the default Hugging Face cache directory os.environ\["HF\_DATASETS\_CACHE"\] \= temp\_dir return load\_dataset("conll2003", trust\_remote\_code\=True) def extract\_people\_entities(data\_row: Dict\[str, Any\]) \-> List\[str\]: """ Extracts entities referring to people from a row of the CoNLL-2003 dataset. Args: data\_row (Dict\[str, Any\]): A row from the dataset containing tokens and NER tags. Returns: List\[str\]: List of tokens tagged as people. """ return \[\ token\ for token, ner\_tag in zip(data\_row\["tokens"\], data\_row\["ner\_tags"\])\ if ner\_tag in (1, 2) \# CoNLL entity codes 1 and 2 refer to people\ \] def prepare\_dataset(data\_split, start: int, end: int) \-> List\[dspy.Example\]: """ Prepares a sliced dataset split for use with DSPy. Args: data\_split: The dataset split (e.g., train or test). start (int): Starting index of the slice. end (int): Ending index of the slice. Returns: List\[dspy.Example\]: List of DSPy Examples with tokens and expected labels. """ return \[\ dspy.Example(\ tokens\=row\["tokens"\],\ expected\_extracted\_people\=extract\_people\_entities(row)\ ).with\_inputs("tokens")\ for row in data\_split.select(range(start, end))\ \] \# Load the dataset dataset \= load\_conll\_dataset() \# Prepare the training and test sets train\_set \= prepare\_dataset(dataset\["train"\], 0, 50) test\_set \= prepare\_dataset(dataset\["test"\], 0, 200) import os import tempfile from datasets import load\_dataset from typing import Dict, Any, List import dspy def load\_conll\_dataset() -> dict: """ Loads the CoNLL-2003 dataset into train, validation, and test splits. Returns: dict: Dataset splits with keys 'train', 'validation', and 'test'. """ with tempfile.TemporaryDirectory() as temp\_dir: # Use a temporary Hugging Face cache directory for compatibility with certain hosted notebook # environments that don't support the default Hugging Face cache directory os.environ\["HF\_DATASETS\_CACHE"\] = temp\_dir return load\_dataset("conll2003", trust\_remote\_code=True) def extract\_people\_entities(data\_row: Dict\[str, Any\]) -> List\[str\]: """ Extracts entities referring to people from a row of the CoNLL-2003 dataset. Args: data\_row (Dict\[str, Any\]): A row from the dataset containing tokens and NER tags. Returns: List\[str\]: List of tokens tagged as people. """ return \[ token for token, ner\_tag in zip(data\_row\["tokens"\], data\_row\["ner\_tags"\]) if ner\_tag in (1, 2) # CoNLL entity codes 1 and 2 refer to people \] def prepare\_dataset(data\_split, start: int, end: int) -> List\[dspy.Example\]: """ Prepares a sliced dataset split for use with DSPy. Args: data\_split: The dataset split (e.g., train or test). start (int): Starting index of the slice. end (int): Ending index of the slice. Returns: List\[dspy.Example\]: List of DSPy Examples with tokens and expected labels. """ return \[ dspy.Example( tokens=row\["tokens"\], expected\_extracted\_people=extract\_people\_entities(row) ).with\_inputs("tokens") for row in data\_split.select(range(start, end)) \] # Load the dataset dataset = load\_conll\_dataset() # Prepare the training and test sets train\_set = prepare\_dataset(dataset\["train"\], 0, 50) test\_set = prepare\_dataset(dataset\["test"\], 0, 200) Configure DSPy and create an Entity Extraction Program[¶](#configure-dspy-and-create-an-entity-extraction-program) ------------------------------------------------------------------------------------------------------------------- Here, we define a DSPy program for extracting entities referring to people from tokenized text. Then, we configure DSPy to use a particular language model (`gpt-4o-mini`) for all invocations of the program. **Key DSPy Concepts Introduced:** * **Signatures:** Define structured input/output schemas for your program. * **Modules:** Encapsulate program logic in reusable, composable units. Specifically, we'll: * Create a `PeopleExtraction` DSPy Signature to specify the input (`tokens`) and output (`extracted_people`) fields. * Define a `people_extractor` program that uses DSPy's built-in `dspy.ChainOfThought` module to implement the `PeopleExtraction` signature. The program extracts entities referring to people from a list of input tokens using language model (LM) prompting. * Use the `dspy.LM` class and `dspy.settings.configure()` method to configure the language model that DSPy will use when invoking the program. In \[3\]: Copied! from typing import List class PeopleExtraction(dspy.Signature): """ Extract contiguous tokens referring to specific people, if any, from a list of string tokens. Output a list of tokens. In other words, do not combine multiple tokens into a single value. """ tokens: list\[str\] \= dspy.InputField(desc\="tokenized text") extracted\_people: list\[str\] \= dspy.OutputField(desc\="all tokens referring to specific people extracted from the tokenized text") people\_extractor \= dspy.ChainOfThought(PeopleExtraction) from typing import List class PeopleExtraction(dspy.Signature): """ Extract contiguous tokens referring to specific people, if any, from a list of string tokens. Output a list of tokens. In other words, do not combine multiple tokens into a single value. """ tokens: list\[str\] = dspy.InputField(desc="tokenized text") extracted\_people: list\[str\] = dspy.OutputField(desc="all tokens referring to specific people extracted from the tokenized text") people\_extractor = dspy.ChainOfThought(PeopleExtraction) Here, we tell DSPy to use OpenAI's `gpt-4o-mini` model in our program. To authenticate, DSPy reads your `OPENAI_API_KEY`. You can easily swap this out for [other providers or local models](https://github.com/stanfordnlp/dspy/blob/main/examples/migration.ipynb) . In \[4\]: Copied! lm \= dspy.LM(model\="openai/gpt-4o-mini") dspy.settings.configure(lm\=lm) lm = dspy.LM(model="openai/gpt-4o-mini") dspy.settings.configure(lm=lm) Define Metric and Evaluation Functions[¶](#define-metric-and-evaluation-functions) ----------------------------------------------------------------------------------- In DSPy, evaluating a program's performance is critical for iterative development. A good evaluation framework allows us to: * Measure the quality of our program's outputs. * Compare outputs against ground-truth labels. * Identify areas for improvement. **What We'll Do:** * Define a custom metric (`extraction_correctness_metric`) to evaluate whether the extracted entities match the ground truth. * Create an evaluation function (`evaluate_correctness`) to apply this metric to a training or test dataset and compute the overall accuracy. The evaluation function uses DSPy's `Evaluate` utility to handle parallelism and visualization of results. In \[5\]: Copied! def extraction\_correctness\_metric(example: dspy.Example, prediction: dspy.Prediction, trace\=None) \-> bool: """ Computes correctness of entity extraction predictions. Args: example (dspy.Example): The dataset example containing expected people entities. prediction (dspy.Prediction): The prediction from the DSPy people extraction program. trace: Optional trace object for debugging. Returns: bool: True if predictions match expectations, False otherwise. """ return prediction.extracted\_people \== example.expected\_extracted\_people evaluate\_correctness \= dspy.Evaluate( devset\=test\_set, metric\=extraction\_correctness\_metric, num\_threads\=24, display\_progress\=True, display\_table\=True ) def extraction\_correctness\_metric(example: dspy.Example, prediction: dspy.Prediction, trace=None) -> bool: """ Computes correctness of entity extraction predictions. Args: example (dspy.Example): The dataset example containing expected people entities. prediction (dspy.Prediction): The prediction from the DSPy people extraction program. trace: Optional trace object for debugging. Returns: bool: True if predictions match expectations, False otherwise. """ return prediction.extracted\_people == example.expected\_extracted\_people evaluate\_correctness = dspy.Evaluate( devset=test\_set, metric=extraction\_correctness\_metric, num\_threads=24, display\_progress=True, display\_table=True ) Evaluate Initial Extractor[¶](#evaluate-initial-extractor) ----------------------------------------------------------- Before optimizing our program, we need a baseline evaluation to understand its current performance. This helps us: * Establish a reference point for comparison after optimization. * Identify potential weaknesses in the initial implementation. In this step, we'll run our `people_extractor` program on the test set and measure its accuracy using the evaluation framework defined earlier. In \[6\]: Copied! evaluate\_correctness(people\_extractor, devset\=test\_set) evaluate\_correctness(people\_extractor, devset=test\_set) Average Metric: 172.00 / 200 (86.0%): 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 \[00:16<00:00, 11.94it/s\] 2024/11/18 21:08:04 INFO dspy.evaluate.evaluate: Average Metric: 172 / 200 (86.0%) | | tokens | expected\_extracted\_people | rationale | extracted\_people | extraction\_correctness\_metric | | --- | --- | --- | --- | --- | --- | | 0 | \[SOCCER, -, JAPAN, GET, LUCKY, WIN, ,, CHINA, IN, SURPRISE, DEFEAT... | \[CHINA\] | We extracted "JAPAN" and "CHINA" as they refer to specific countri... | \[JAPAN, CHINA\] | |\ | 1 | \[Nadim, Ladki\] | \[Nadim, Ladki\] | We extracted the tokens "Nadim" and "Ladki" as they refer to speci... | \[Nadim, Ladki\] | ✔️ \[True\] |\ | 2 | \[AL-AIN, ,, United, Arab, Emirates, 1996-12-06\] | \[\] | There are no tokens referring to specific people in the provided l... | \[\] | ✔️ \[True\] |\ | 3 | \[Japan, began, the, defence, of, their, Asian, Cup, title, with, a... | \[\] | We did not find any tokens referring to specific people in the pro... | \[\] | ✔️ \[True\] |\ | 4 | \[But, China, saw, their, luck, desert, them, in, the, second, matc... | \[\] | The extracted tokens referring to specific people are "China" and ... | \[China, Uzbekistan\] | |\ | ... | ... | ... | ... | ... | ... |\ | 195 | \['The', 'Wallabies', 'have', 'their', 'sights', 'set', 'on', 'a', ... | \[David, Campese\] | The extracted\_people includes "David Campese" as it refers to a sp... | \[David, Campese\] | ✔️ \[True\] |\ | 196 | \['The', 'Wallabies', 'currently', 'have', 'no', 'plans', 'to', 'ma... | \[\] | The extracted\_people includes "Wallabies" as it refers to a specif... | \[\] | ✔️ \[True\] |\ | 197 | \['Campese', 'will', 'be', 'up', 'against', 'a', 'familiar', 'foe',... | \[Campese, Rob, Andrew\] | The extracted tokens refer to specific people mentioned in the tex... | \[Campese, Rob, Andrew\] | ✔️ \[True\] |\ | 198 | \['"', 'Campo', 'has', 'a', 'massive', 'following', 'in', 'this', '... | \[Campo, Andrew\] | The extracted tokens referring to specific people include "Campo" ... | \[Campo, Andrew\] | ✔️ \[True\] |\ | 199 | \['On', 'tour', ',', 'Australia', 'have', 'won', 'all', 'four', 'te... | \[\] | We extracted the names of specific people from the tokenized text.... | \[\] | ✔️ \[True\] |\ \ 200 rows × 5 columns\ \ Out\[6\]:\ \ 86.0\ \ Optimize the Model[¶](#optimize-the-model)\ \ -------------------------------------------\ \ DSPy includes powerful optimizers that can improve the quality of your system.\ \ Here, we use DSPy's `MIPROv2` optimizer to:\ \ * Automatically tune the program's language model (LM) prompt by 1. using the LM to adjust the prompt's instructions and 2. building few-shot examples from the training dataset that are augmented with reasoning generated from `dspy.ChainOfThought`.\ * Maximize correctness on the training set.\ \ This optimization process is automated, saving time and effort while improving accuracy.\ \ In \[ \]:\ \ Copied!\ \ mipro\_optimizer \= dspy.MIPROv2(\ metric\=extraction\_correctness\_metric,\ auto\="medium",\ )\ optimized\_people\_extractor \= mipro\_optimizer.compile(\ people\_extractor,\ trainset\=train\_set,\ max\_bootstrapped\_demos\=4,\ requires\_permission\_to\_run\=False,\ minibatch\=False\ )\ \ mipro\_optimizer = dspy.MIPROv2( metric=extraction\_correctness\_metric, auto="medium", ) optimized\_people\_extractor = mipro\_optimizer.compile( people\_extractor, trainset=train\_set, max\_bootstrapped\_demos=4, requires\_permission\_to\_run=False, minibatch=False )\ \ Evaluate Optimized Program[¶](#evaluate-optimized-program)\ \ -----------------------------------------------------------\ \ After optimization, we re-evaluate the program on the test set to measure improvements. Comparing the optimized and initial results allows us to:\ \ * Quantify the benefits of optimization.\ * Validate that the program generalizes well to unseen data.\ \ In this case, we see that accuracy of the program on the test dataset has improved significantly.\ \ In \[8\]:\ \ Copied!\ \ evaluate\_correctness(optimized\_people\_extractor, devset\=test\_set)\ \ evaluate\_correctness(optimized\_people\_extractor, devset=test\_set)\ \ Average Metric: 186.00 / 200 (93.0%): 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████| 200/200 \[00:23<00:00, 8.58it/s\]\ \ 2024/11/18 21:15:00 INFO dspy.evaluate.evaluate: Average Metric: 186 / 200 (93.0%)\ \ | | tokens | expected\_extracted\_people | rationale | extracted\_people | extraction\_correctness\_metric |\ | --- | --- | --- | --- | --- | --- |\ | 0 | \[SOCCER, -, JAPAN, GET, LUCKY, WIN, ,, CHINA, IN, SURPRISE, DEFEAT... | \[CHINA\] | There are no specific people mentioned in the provided tokens. The... | \[\] | |\ | 1 | \[Nadim, Ladki\] | \[Nadim, Ladki\] | The tokens "Nadim Ladki" refer to a specific individual. Both toke... | \[Nadim, Ladki\] | ✔️ \[True\] |\ | 2 | \[AL-AIN, ,, United, Arab, Emirates, 1996-12-06\] | \[\] | There are no tokens referring to specific people in the provided l... | \[\] | ✔️ \[True\] |\ | 3 | \[Japan, began, the, defence, of, their, Asian, Cup, title, with, a... | \[\] | There are no specific people mentioned in the provided tokens. The... | \[\] | ✔️ \[True\] |\ | 4 | \[But, China, saw, their, luck, desert, them, in, the, second, matc... | \[\] | There are no tokens referring to specific people in the provided l... | \[\] | ✔️ \[True\] |\ | ... | ... | ... | ... | ... | ... |\ | 195 | \['The', 'Wallabies', 'have', 'their', 'sights', 'set', 'on', 'a', ... | \[David, Campese\] | The extracted tokens refer to a specific person mentioned in the t... | \[David, Campese\] | ✔️ \[True\] |\ | 196 | \['The', 'Wallabies', 'currently', 'have', 'no', 'plans', 'to', 'ma... | \[\] | There are no specific individuals mentioned in the provided tokens... | \[\] | ✔️ \[True\] |\ | 197 | \['Campese', 'will', 'be', 'up', 'against', 'a', 'familiar', 'foe',... | \[Campese, Rob, Andrew\] | The tokens include the names "Campese" and "Rob Andrew," both of w... | \[Campese, Rob, Andrew\] | ✔️ \[True\] |\ | 198 | \['"', 'Campo', 'has', 'a', 'massive', 'following', 'in', 'this', '... | \[Campo, Andrew\] | The extracted tokens refer to specific people mentioned in the tex... | \[Campo, Andrew\] | ✔️ \[True\] |\ | 199 | \['On', 'tour', ',', 'Australia', 'have', 'won', 'all', 'four', 'te... | \[\] | There are no specific people mentioned in the provided tokens. The... | \[\] | ✔️ \[True\] |\ \ 200 rows × 5 columns\ \ Out\[8\]:\ \ 93.0\ \ Inspect Optimized Program's Prompt[¶](#inspect-optimized-programs-prompt)\ \ --------------------------------------------------------------------------\ \ After optimizing the program, we can inspect the history of interactions to see how DSPy has augmented the program's prompt with few-shot examples. This step demonstrates:\ \ * The structure of the prompt used by the program.\ * How few-shot examples are added to guide the model's behavior.\ \ Use `inspect_history(n=1)` to view the last interaction and analyze the generated prompt.\ \ In \[9\]:\ \ Copied!\ \ dspy.inspect\_history(n\=1)\ \ dspy.inspect\_history(n=1)\ \ \[2024-11-18T21:15:00.584497\]\ \ System message:\ \ Your input fields are:\ 1. \`tokens\` (list\[str\]): tokenized text\ \ Your output fields are:\ 1. \`rationale\` (str): ${produce the extracted\_people}. We ...\ 2. \`extracted\_people\` (list\[str\]): all tokens referring to specific people extracted from the tokenized text\ \ All interactions will be structured in the following way, with the appropriate values filled in.\ \ \[\[ ## tokens ## \]\]\ {tokens}\ \ \[\[ ## rationale ## \]\]\ {rationale}\ \ \[\[ ## extracted\_people ## \]\]\ {extracted\_people} # note: the value you produce must be pareseable according to the following JSON schema: {"type": "array", "items": {"type": "string"}}\ \ \[\[ ## completed ## \]\]\ \ In adhering to this structure, your objective is: \ In a high-stakes situation where accurate identification of individuals is critical for regulatory compliance and public health communication, extract contiguous tokens referring to specific people from the provided list of string tokens. Ensure that you output each identified individual as separate tokens without combining multiple tokens into a single value. This task is essential for ensuring clarity and accountability in communications pertaining to EU regulations and health matters.\ \ User message:\ \ \[\[ ## tokens ## \]\]\ \["He", "said", "further", "scientific", "study", "was", "required", "and", "if", "it", "was", "found", "that", "action", "was", "needed", "it", "should", "be", "taken", "by", "the", "European", "Union", "."\]\ \ Respond with the corresponding output fields, starting with the field \`\[\[ ## rationale ## \]\]\`, then \`\[\[ ## extracted\_people ## \]\]\` (must be formatted as a valid Python list\[str\]), and then ending with the marker for \`\[\[ ## completed ## \]\]\`.\ \ Assistant message:\ \ \[\[ ## rationale ## \]\]\ There are no specific people mentioned in the provided tokens. The only reference is to the "European Union," which is an organization rather than a specific person. We therefore conclude that there are no extracted people.\ \ \[\[ ## extracted\_people ## \]\]\ \[\]\ \ \[\[ ## completed ## \]\]\ \ User message:\ \ \[\[ ## tokens ## \]\]\ \["BRUSSELS", "1996-08-22"\]\ \ Respond with the corresponding output fields, starting with the field \`\[\[ ## rationale ## \]\]\`, then \`\[\[ ## extracted\_people ## \]\]\` (must be formatted as a valid Python list\[str\]), and then ending with the marker for \`\[\[ ## completed ## \]\]\`.\ \ Assistant message:\ \ \[\[ ## rationale ## \]\]\ There are no tokens referring to specific people in the provided list. We only have a location and a date.\ \ \[\[ ## extracted\_people ## \]\]\ \[\]\ \ \[\[ ## completed ## \]\]\ \ User message:\ \ \[\[ ## tokens ## \]\]\ \["Fischler", "proposed", "EU-wide", "measures", "after", "reports", "from", "Britain", "and", "France", "that", "under", "laboratory", "conditions", "sheep", "could", "contract", "Bovine", "Spongiform", "Encephalopathy", "(", "BSE", ")", "--", "mad", "cow", "disease", "."\]\ \ Respond with the corresponding output fields, starting with the field \`\[\[ ## rationale ## \]\]\`, then \`\[\[ ## extracted\_people ## \]\]\` (must be formatted as a valid Python list\[str\]), and then ending with the marker for \`\[\[ ## completed ## \]\]\`.\ \ Assistant message:\ \ \[\[ ## rationale ## \]\]\ The extracted tokens refer to specific people mentioned in the text. In this case, "Fischler" is the only token that refers to a specific person. We extract this token as it is contiguous and represents an individual involved in the proposed measures.\ \ \[\[ ## extracted\_people ## \]\]\ \["Fischler"\]\ \ \[\[ ## completed ## \]\]\ \ User message:\ \ \[\[ ## tokens ## \]\]\ \["Germany", "'s", "representative", "to", "the", "European", "Union", "'s", "veterinary", "committee", "Werner", "Zwingmann", "said", "on", "Wednesday", "consumers", "should", "buy", "sheepmeat", "from", "countries", "other", "than", "Britain", "until", "the", "scientific", "advice", "was", "clearer", "."\]\ \ Respond with the corresponding output fields, starting with the field \`\[\[ ## rationale ## \]\]\`, then \`\[\[ ## extracted\_people ## \]\]\` (must be formatted as a valid Python list\[str\]), and then ending with the marker for \`\[\[ ## completed ## \]\]\`.\ \ Assistant message:\ \ \[\[ ## rationale ## \]\]\ We extracted the name "Werner Zwingmann" from the tokenized text as it refers to a specific person mentioned in the context of a statement regarding sheepmeat consumption.\ \ \[\[ ## extracted\_people ## \]\]\ \["Werner", "Zwingmann"\]\ \ \[\[ ## completed ## \]\]\ \ User message:\ \ \[\[ ## tokens ## \]\]\ \["LONDON", "1996-12-06"\]\ \ Respond with the corresponding output fields, starting with the field \`\[\[ ## rationale ## \]\]\`, then \`\[\[ ## extracted\_people ## \]\]\` (must be formatted as a valid Python list\[str\]), and then ending with the marker for \`\[\[ ## completed ## \]\]\`.\ \ Response:\ \ \[\[ ## rationale ## \]\]\ There are no tokens referring to specific people in the provided list. The tokens only include a location and a date.\ \ \[\[ ## extracted\_people ## \]\]\ \[\]\ \ \[\[ ## completed ## \]\]\ \ Keeping an eye on cost[¶](#keeping-an-eye-on-cost)\ \ ---------------------------------------------------\ \ DSPy allows you to track the cost of your programs. The following code demonstrates how to obtain the cost of all LM calls made by the DSPy extractor program so far.\ \ In \[10\]:\ \ Copied!\ \ cost \= sum(\[x\['cost'\] for x in lm.history if x\['cost'\] is not None\]) \# cost in USD, as calculated by LiteLLM for certain providers\ cost\ \ cost = sum(\[x\['cost'\] for x in lm.history if x\['cost'\] is not None\]) # cost in USD, as calculated by LiteLLM for certain providers cost\ \ Out\[10\]:\ \ 0.26362742999999983\ \ Saving and Loading Optimized Programs[¶](#saving-and-loading-optimized-programs)\ \ ---------------------------------------------------------------------------------\ \ DSPy supports saving and loading programs, enabling you to reuse optimized systems without the need to re-optimize from scratch. This feature is especially useful for deploying your programs in production environments or sharing them with collaborators.\ \ In this step, we'll save the optimized program to a file and demonstrate how to load it back for future use.\ \ In \[11\]:\ \ Copied!\ \ optimized\_people\_extractor.save("optimized\_extractor.json")\ \ loaded\_people\_extractor \= dspy.ChainOfThought(PeopleExtraction)\ loaded\_people\_extractor.load("optimized\_extractor.json")\ \ loaded\_people\_extractor(tokens\=\["Italy", "recalled", "Marcello", "Cuttitta"\]).extracted\_people\ \ optimized\_people\_extractor.save("optimized\_extractor.json") loaded\_people\_extractor = dspy.ChainOfThought(PeopleExtraction) loaded\_people\_extractor.load("optimized\_extractor.json") loaded\_people\_extractor(tokens=\["Italy", "recalled", "Marcello", "Cuttitta"\]).extracted\_people\ \ Out\[11\]:\ \ \['Marcello', 'Cuttitta'\]\ \ Conclusion[¶](#conclusion)\ \ ---------------------------\ \ In this tutorial, we demonstrated how to:\ \ * Use DSPy to build a modular, interpretable system for entity extraction.\ * Evaluate and optimize the system using DSPy's built-in tools.\ \ By leveraging structured inputs and outputs, we ensured that the system was easy to understand and improve. The optimization process allowed us to quickly improve performance without manually crafting prompts or tweaking parameters.\ \ **Next Steps:**\ \ * Experiment with extraction of other entity types (e.g., locations or organizations).\ * Explore DSPy's other builtin modules like `ReAct` for more complex reasoning tasks.\ * Use the system in larger workflows, such as large scale document processing or summarization.\ \ Back to top --- # FAQ - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/faqs.md "Edit this page") FAQs ==== Is DSPy right for me? DSPy vs. other frameworks ----------------------------------------------- The **DSPy** philosophy and abstraction differ significantly from other libraries and frameworks, so it's usually straightforward to decide when **DSPy** is (or isn't) the right framework for your usecase. If you're a NLP/AI researcher (or a practitioner exploring new pipelines or new tasks), the answer is generally an invariable **yes**. If you're a practitioner doing other things, please read on. **DSPy vs. thin wrappers for prompts (OpenAI API, MiniChain, basic templating)** In other words: _Why can't I just write my prompts directly as string templates?_ Well, for extremely simple settings, this _might_ work just fine. (If you're familiar with neural networks, this is like expressing a tiny two-layer NN as a Python for-loop. It kinda works.) However, when you need higher quality (or manageable cost), then you need to iteratively explore multi-stage decomposition, improved prompting, data bootstrapping, careful finetuning, retrieval augmentation, and/or using smaller (or cheaper, or local) models. The true expressive power of building with foundation models lies in the interactions between these pieces. But every time you change one piece, you likely break (or weaken) multiple other components. **DSPy** cleanly abstracts away (_and_ powerfully optimizes) the parts of these interactions that are external to your actual system design. It lets you focus on designing the module-level interactions: the _same program_ expressed in 10 or 20 lines of **DSPy** can easily be compiled into multi-stage instructions for `GPT-4`, detailed prompts for `Llama2-13b`, or finetunes for `T5-base`. Oh, and you wouldn't need to maintain long, brittle, model-specific strings at the core of your project anymore. **DSPy vs. application development libraries like LangChain, LlamaIndex** LangChain and LlamaIndex target high-level application development; they offer _batteries-included_, pre-built application modules that plug in with your data or configuration. If you'd be happy to use a generic, off-the-shelf prompt for question answering over PDFs or standard text-to-SQL, you will find a rich ecosystem in these libraries. **DSPy** doesn't internally contain hand-crafted prompts that target specific applications. Instead, **DSPy** introduces a small set of much more powerful and general-purpose modules _that can learn to prompt (or finetune) your LM within your pipeline on your data_. when you change your data, make tweaks to your program's control flow, or change your target LM, the **DSPy compiler** can map your program into a new set of prompts (or finetunes) that are optimized specifically for this pipeline. Because of this, you may find that **DSPy** obtains the highest quality for your task, with the least effort, provided you're willing to implement (or extend) your own short program. In short, **DSPy** is for when you need a lightweight but automatically-optimizing programming model — not a library of predefined prompts and integrations. If you're familiar with neural networks: This is like the difference between PyTorch (i.e., representing **DSPy**) and HuggingFace Transformers (i.e., representing the higher-level libraries). **DSPy vs. generation control libraries like Guidance, LMQL, RELM, Outlines** These are all exciting new libraries for controlling the individual completions of LMs, e.g., if you want to enforce JSON output schema or constrain sampling to a particular regular expression. This is very useful in many settings, but it's generally focused on low-level, structured control of a single LM call. It doesn't help ensure the JSON (or structured output) you get is going to be correct or useful for your task. In contrast, **DSPy** automatically optimizes the prompts in your programs to align them with various task needs, which may also include producing valid structured outputs. That said, we are considering allowing **Signatures** in **DSPy** to express regex-like constraints that are implemented by these libraries. Basic Usage ----------- **How should I use DSPy for my task?** We wrote a [eight-step guide](/building-blocks/solving_your_task) on this. In short, using DSPy is an iterative process. You first define your task and the metrics you want to maximize, and prepare a few example inputs — typically without labels (or only with labels for the final outputs, if your metric requires them). Then, you build your pipeline by selecting built-in layers (`modules`) to use, giving each layer a `signature` (input/output spec), and then calling your modules freely in your Python code. Lastly, you use a DSPy `optimizer` to compile your code into high-quality instructions, automatic few-shot examples, or updated LM weights for your LM. **How do I convert my complex prompt into a DSPy pipeline?** See the same answer above. **What do DSPy optimizers tune?** Or, _what does compiling actually do?_ Each optimizer is different, but they all seek to maximize a metric on your program by updating prompts or LM weights. Current DSPy `optimizers` can inspect your data, simulate traces through your program to generate good/bad examples of each step, propose or refine instructions for each step based on past results, finetune the weights of your LM on self-generated examples, or combine several of these to improve quality or cut cost. We'd love to merge new optimizers that explore a richer space: most manual steps you currently go through for prompt engineering, "synthetic data" generation, or self-improvement can probably generalized into a DSPy optimizer that acts on arbitrary LM programs. Other FAQs. We welcome PRs to add formal answers to each of these here. You will find the answer in existing issues, tutorials, or the papers for all or most of these. * **How do I get multiple outputs?** You can specify multiple output fields. For the short-form signature, you can list multiple outputs as comma separated values, following the "->" indicator (e.g. "inputs -> output1, output2"). For the long-form signature, you can include multiple `dspy.OutputField`s. * **How do I define my own metrics? Can metrics return a float?** You can define metrics as simply Python functions that process model generations and evaluate them based on user-defined requirements. Metrics can compare existent data (e.g. gold labels) to model predictions or they can be used to assess various components of an output using validation feedback from LMs (e.g. LLMs-as-Judges). Metrics can return `bool`, `int`, and `float` types scores. Check out the official [Metrics docs](/building-blocks/5-metrics) to learn more about defining custom metrics and advanced evaluations using AI feedback and/or DSPy programs. * **How expensive or slow is compiling??** To reflect compiling metrics, we highlight an experiment for reference, compiling the [`SimplifiedBaleen`](/tutorials/simplified-baleen) using the [`dspy.BootstrapFewShotWithRandomSearch`](/deep-dive/teleprompter/bootstrap-fewshot) optimizer on the `gpt-3.5-turbo-1106` model over 7 candidate programs and 10 threads. We report that compiling this program takes around 6 minutes with 3200 API calls, 2.7 million input tokens and 156,000 output tokens, reporting a total cost of $3 USD (at the current pricing of the OpenAI model). Compiling DSPy `optimizers` naturally will incur additional LM calls, but we substantiate this overhead with minimalistic executions with the goal of maximizing performance. This invites avenues to enhance performance of smaller models by compiling DSPy programs with larger models to learn enhanced behavior during compile-time and propagate such behavior to the tested smaller model during inference-time. Deployment or Reproducibility Concerns -------------------------------------- * **How do I save a checkpoint of my compiled program?** Here is an example of saving/loading a compiled module: `[](#__codelineno-0-1) cot_compiled = teleprompter.compile(CoT(), trainset=trainset, valset=devset) [](#__codelineno-0-2) [](#__codelineno-0-3) #Saving [](#__codelineno-0-4) cot_compiled.save('compiled_cot_gsm8k.json') [](#__codelineno-0-5) [](#__codelineno-0-6) #Loading: [](#__codelineno-0-7) cot = CoT() [](#__codelineno-0-8) cot.load('compiled_cot_gsm8k.json')` * **How do I export for deployment?** Exporting DSPy programs is simply saving them as highlighted above! * **How do I search my own data?** Open source libraries such as [RAGautouille](https://github.com/bclavie/ragatouille) enable you to search for your own data through advanced retrieval models like ColBERT with tools to embed and index documents. Feel free to integrate such libraries to create searchable datasets while developing your DSPy programs! * **How do I turn off the cache? How do I export the cache?** From v2.5, you can turn off the cache by setting `cache` parameter in `dspy.LM` to `False`: `[](#__codelineno-1-1) dspy.LM('openai/gpt-4o-mini', cache=False)` Your local cache will be saved to the global env directory `os.environ["DSP_CACHEDIR"]` or for notebooks `os.environ["DSP_NOTEBOOK_CACHEDIR"]`. You can usually set the cachedir to `os.path.join(repo_path, 'cache')` and export this cache from here: `[](#__codelineno-2-1) os.environ["DSP_NOTEBOOK_CACHEDIR"] = os.path.join(os.getcwd(), 'cache')` Important `DSP_CACHEDIR` is responsible for old clients (including dspy.OpenAI, dspy.ColBERTv2, etc.) and `DSPY_CACHEDIR` is responsible for the new dspy.LM client. In the AWS lambda deployment, you should disable both DSP\_\* and DSPY\_\*. Advanced Usage -------------- * **How do I parallelize?** You can parallelize DSPy programs during both compilation and evaluation by specifying multiple thread settings in the respective DSPy `optimizers` or within the `dspy.Evaluate` utility function. * **How do freeze a module?** Modules can be frozen by setting their `._compiled` attribute to be True, indicating the module has gone through optimizer compilation and should not have its parameters adjusted. This is handled internally in optimizers such as `dspy.BootstrapFewShot` where the student program is ensured to be frozen before the teacher propagates the gathered few-shot demonstrations in the bootstrapping process. * **How do I use DSPy assertions?** a) **How to Add Assertions to Your Program**: - **Define Constraints**: Use `dspy.Assert` and/or `dspy.Suggest` to define constraints within your DSPy program. These are based on boolean validation checks for the outcomes you want to enforce, which can simply be Python functions to validate the model outputs. - **Integrating Assertions**: Keep your Assertion statements following a model generations (hint: following a module layer) b) **How to Activate the Assertions**: 1. **Using `assert_transform_module`**: - Wrap your DSPy module with assertions using the `assert_transform_module` function, along with a `backtrack_handler`. This function transforms your program to include internal assertions backtracking and retry logic, which can be customized as well: `program_with_assertions = assert_transform_module(ProgramWithAssertions(), backtrack_handler)` 2. **Activate Assertions**: - Directly call `activate_assertions` on your DSPy program with assertions: `program_with_assertions = ProgramWithAssertions().activate_assertions()` **Note**: To use Assertions properly, you must **activate** a DSPy program that includes `dspy.Assert` or `dspy.Suggest` statements from either of the methods above. Errors ------ * **How do I deal with "context too long" errors?** If you're dealing with "context too long" errors in DSPy, you're likely using DSPy optimizers to include demonstrations within your prompt, and this is exceeding your current context window. Try reducing these parameters (e.g. `max_bootstrapped_demos` and `max_labeled_demos`). Additionally, you can also reduce the number of retrieved passages/docs/embeddings to ensure your prompt is fitting within your model context length. A more general fix is simply increasing the number of `max_tokens` specified to the LM request (e.g. `lm = dspy.OpenAI(model = ..., max_tokens = ...`). Back to top --- # Community Resources - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/community/community-resources.md "Edit this page") Resources ========= This is the list of tutorials and blog posts on DSPy. If you would like to add your own tutorial, please make a PR. A Few Blogs & Videos on using DSPy ---------------------------------- ### Blogs | **Name** | **Link** | | --- | --- | | **Why I bet on DSPy** | [Blog](https://blog.isaacbmiller.com/posts/dspy) | | **Not Your Average Prompt Engineering** | [Blog](https://jina.ai/news/dspy-not-your-average-prompt-engineering/) | | **Why I'm excited about DSPy** | [Blog](https://substack.stephen.so/p/why-im-excited-about-dspy) | | **Achieving GPT-4 Performance at Lower Cost** | [Link](https://gradient.ai/blog/achieving-gpt-4-level-performance-at-lower-cost-using-dspy) | | **Prompt engineering is a task best left to AI models** | [Link](https://www.theregister.com/2024/02/22/prompt_engineering_ai_models/) | | **What makes DSPy a valuable framework for developing complex language model pipelines?** | [Link](https://medium.com/@sujathamudadla1213/what-makes-dspy-a-valuable-framework-for-developing-complex-language-model-pipelines-edfa5b4bcf9b) | | **DSPy: A new framework to program your foundation models just by prompting** | [Link](https://www.linkedin.com/pulse/dspy-new-framework-program-your-foundation-models-just-prompting-lli4c/) | | **Intro to DSPy: Goodbye Prompting, Hello Programming** | [Link](https://medium.com/towards-data-science/intro-to-dspy-goodbye-prompting-hello-programming-4ca1c6ce3eb9) | | **DSPyGen: Revolutionizing AI** | [Link](https://www.linkedin.com/pulse/launch-alert-dspygen-20242252-revolutionizing-ai-sean-chatman--g9f1c/) | | **Building an AI Assistant with DSPy** | [Link](https://www.linkedin.com/pulse/building-ai-assistant-dspy-valliappa-lakshmanan-vgnsc/) | ### Videos | **Name** | **Link** | | --- | --- | | **DSPy Explained! (60K views)** | [Link](https://www.youtube.com/watch?v=41EfOY0Ldkc) | | **DSPy Intro from Sephora (25K views)** | [Link](https://www.youtube.com/watch?v=D2HurSldDkE) | | **Structured Outputs with DSPy** | [Link](https://www.youtube.com/watch?v=tVw3CwrN5-8) | | **DSPy and ColBERT - Weaviate Podcast** | [Link](https://www.youtube.com/watch?v=CDung1LnLbY) | | **SBTB23 DSPy** | [Link](https://www.youtube.com/watch?v=Dt3H2ninoeY) | | **Optimization with DSPy and LangChain** | [Link](https://www.youtube.com/watch?v=4EXOmWeqXRc) | | **Automated Prompt Engineering + Visualization** | [Link](https://www.youtube.com/watch?v=eAZ2LtJ6D5k) | | **Transforming LM Calls into Pipelines** | [Link](https://www.youtube.com/watch?v=NoaDWKHdkHg) | | **NeurIPS Hacker Cup: DSPy for Code Gen** | [Link](https://www.youtube.com/watch?v=gpe-rtJN8z8) | | **MIPRO and DSPy - Weaviate Podcast** | [Link](https://www.youtube.com/watch?v=skMH3DOV_UQ) | | **Getting Started with RAG in DSPy** | [Link](https://www.youtube.com/watch?v=CEuUG4Umfxs) | | **Adding Depth to DSPy Programs** | [Link](https://www.youtube.com/watch?v=0c7Ksd6BG88) | | **Programming Foundation Models with DSPy** | [Link](https://www.youtube.com/watch?v=Y94tw4eDHW0) | | **DSPy End-to-End: SF Meetup** | [Link](https://www.youtube.com/watch?v=Y81DoFmt-2U) | | **Monitoring & Tracing DSPy with Langtrace** | [Link](https://langtrace.ai/blog/announcing-dspy-support-in-langtrace) | | **Teaching chat models to solve chess puzzles using DSPy + Finetuning** | [Link](https://raw.sh/posts/chess_puzzles) | ### Podcasts Weaviate has a directory of 10 amazing notebooks and 6 podcasts! Huge shoutout to them for the massive support ❤️. See the [Weaviate DSPy directory](https://weaviate.io/developers/weaviate/more-resources/dspy) . TODO: This list in particular is highly incomplete. There are dozens of other good ones. To allow space, divide into opintionated blogs / podcasts / interviews vs. tutorials & talks. Credit: Some of these resources were originally compiled in the [Awesome DSPy](https://github.com/ganarajpr/awesome-dspy/tree/master) repo. Back to top --- # Learning DSPy - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/learn/index.md "Edit this page") Learning DSPy: An Overview ========================== DSPy exposes a very small API that you can learn quickly. However, building a new AI system is a more open-ended journey of iterative development, in which you compose the tools and design patterns of DSPy to optimize for _your_ objectives. The three stages of building AI systems in DSPy are: 1) **DSPy Programming.** This is about defining your task, its constraints, exploring a few examples, and using that to inform your initial pipeline design. 2) **DSPy Evaluation.** Once your system starts working, this is the stage where you collect an initial development set, define your DSPy metric, and use these to iterate on your system more systematically. 3) **DSPy Optimization.** Once you have a way to evaluate your system, you use DSPy optimizers to tune the prompts or weights in your program. We suggest learning and applying DSPy in this order. For example, it's unproductive to launch optimization runs using a poorly-design program or a bad metric. Back to top --- # DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/index.md "Edit this page") ![DSPy](static/img/dspy_logo.png) _Programming_—not prompting—_LMs_ ================================= DSPy is the framework for _programming—rather than prompting—language models_. It allows you to iterate fast on **building modular AI systems** and offers algorithms for **optimizing their prompts and weights**, whether you're building simple classifiers, sophisticated RAG pipelines, or Agent loops. DSPy stands for Declarative Self-improving Python. Instead of brittle prompts, you write compositional _Python code_ and use DSPy to **teach your LM to deliver high-quality outputs**. This [lecture](https://www.youtube.com/watch?v=JEMYuzrKLUw) is a good conceptual introduction. Meet the community, seek help, or start contributing via our [GitHub repo](https://github.com/stanfordnlp/dspy) and [Discord server](https://discord.gg/XCGy2WDCQB) . Getting Started I: Install DSPy and set up your LM `[](#__codelineno-0-1) > pip install -U dspy` OpenAIAnthropicDatabricksLocal LMs on your laptopLocal LMs on a GPU serverOther providers You can authenticate by setting the `OPENAI_API_KEY` env variable or passing `api_key` below. | | | | --- | --- | | [1](#__codelineno-1-1)
[2](#__codelineno-1-2)
[3](#__codelineno-1-3) | `import dspy lm = dspy.LM('openai/gpt-4o-mini', api_key='YOUR_OPENAI_API_KEY') dspy.configure(lm=lm)` | You can authenticate by setting the ANTHROPIC\_API\_KEY env variable or passing `api_key` below. | | | | --- | --- | | [1](#__codelineno-2-1)
[2](#__codelineno-2-2)
[3](#__codelineno-2-3) | `import dspy lm = dspy.LM('anthropic/claude-3-opus-20240229', api_key='YOUR_ANTHROPIC_API_KEY') dspy.configure(lm=lm)` | If you're on the Databricks platform, authentication is automatic via their SDK. If not, you can set the env variables `DATABRICKS_API_KEY` and `DATABRICKS_API_BASE`, or pass `api_key` and `api_base` below. | | | | --- | --- | | [1](#__codelineno-3-1)
[2](#__codelineno-3-2)
[3](#__codelineno-3-3) | `import dspy lm = dspy.LM('databricks/databricks-meta-llama-3-1-70b-instruct') dspy.configure(lm=lm)` | First, install [Ollama](https://github.com/ollama/ollama) and launch its server with your LM. `[](#__codelineno-4-1) > curl -fsSL https://ollama.ai/install.sh | sh [](#__codelineno-4-2) > ollama run llama3.2:1b` Then, connect to it from your DSPy code. | | | | --- | --- | | [1](#__codelineno-5-1)
[2](#__codelineno-5-2)
[3](#__codelineno-5-3) | `import dspy lm = dspy.LM('ollama_chat/llama3.2', api_base='http://localhost:11434', api_key='') dspy.configure(lm=lm)` | First, install [SGLang](https://sgl-project.github.io/start/install.html) and launch its server with your LM. `[](#__codelineno-6-1) > pip install "sglang[all]" [](#__codelineno-6-2) > pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/ [](#__codelineno-6-3) [](#__codelineno-6-4) > CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server --port 7501 --model-path meta-llama/Llama-3.1-8B-Instruct` If you don't have access from Meta to download `meta-llama/Llama-3.1-8B-Instruct`, use `Qwen/Qwen2.5-7B-Instruct` for example. Next, connect to your local LM from your DSPy code as an `OpenAI`\-compatible endpoint. | | | | --- | --- | | [1](#__codelineno-7-1)
[2](#__codelineno-7-2)
[3](#__codelineno-7-3)
[4](#__codelineno-7-4) | `lm = dspy.LM("openai/meta-llama/Llama-3.1-8B-Instruct", api_base="http://localhost:7501/v1", # ensure this points to your port api_key="local", model_type='chat') dspy.configure(lm=lm)` | In DSPy, you can use any of the dozens of [LLM providers supported by LiteLLM](https://docs.litellm.ai/docs/providers) . Simply follow their instructions for which `{PROVIDER}_API_KEY` to set and how to write pass the `{provider_name}/{model_name}` to the constructor. Some examples: * `anyscale/mistralai/Mistral-7B-Instruct-v0.1`, with `ANYSCALE_API_KEY` * `together_ai/togethercomputer/llama-2-70b-chat`, with `TOGETHERAI_API_KEY` * `sagemaker/`, with `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_REGION_NAME` * `azure/`, with `AZURE_API_KEY`, `AZURE_API_BASE`, `AZURE_API_VERSION`, and the optional `AZURE_AD_TOKEN` and `AZURE_API_TYPE` If your provider offers an OpenAI-compatible endpoint, just add an `openai/` prefix to your full model name. | | | | --- | --- | | [1](#__codelineno-8-1)
[2](#__codelineno-8-2)
[3](#__codelineno-8-3) | `import dspy lm = dspy.LM('openai/your-model-name', api_key='PROVIDER_API_KEY', api_base='YOUR_PROVIDER_URL') dspy.configure(lm=lm)` | Calling the LM directly. Idiomatic DSPy involves using _modules_, which we define in the rest of this page. However, it's still easy to call the `lm` you configured above directly. This gives you a unified API and lets you benefit from utilities like automatic caching. | | | | --- | --- | | [1](#__codelineno-9-1)
[2](#__codelineno-9-2) | `lm("Say this is a test!", temperature=0.7) # => ['This is a test!'] lm(messages=[{"role": "user", "content": "Say this is a test!"}]) # => ['This is a test!']` | 1) **Modules** help you describe AI behavior as _code_, not strings. -------------------------------------------------------------------- To build reliable AI systems, you must iterate fast. But maintaining prompts makes that hard: it forces you to tinker with strings or data _every time you change your LM, metrics, or pipeline_. Having built over a dozen best-in-class compound LM systems since 2020, we learned this the hard way—and so built DSPy to decouple AI system design from messy incidental choices about specific LMs or prompting strategies. DSPy shifts your focus from tinkering with prompt strings to **programming with structured and declarative natural-language modules**. For every AI component in your system, you specify input/output behavior as a _signature_ and select a _module_ to assign a strategy for invoking your LM. DSPy expands your signatures into prompts and parses your typed outputs, so you can compose different modules together into ergonomic, portable, and optimizable AI systems. Getting Started II: Build DSPy modules for various tasks Try the examples below after configuring your `lm` above. Adjust the fields to explore what tasks your LM can do well out of the box. Each tab below sets up a DSPy module, like `dspy.Predict`, `dspy.ChainOfThought`, or `dspy.ReAct`, with a task-specific _signature_. For example, `question -> answer: float` tells the module to take a question and to produce a `float` answer. MathRAGClassificationInformation ExtractionAgentsMulti-Stage Pipelines | | | | --- | --- | | [1](#__codelineno-10-1)
[2](#__codelineno-10-2) | `math = dspy.ChainOfThought("question -> answer: float") math(question="Two dice are tossed. What is the probability that the sum equals two?")` | **Possible Output:** `[](#__codelineno-11-1) Prediction( [](#__codelineno-11-2) reasoning='When two dice are tossed, each die has 6 faces, resulting in a total of 6 x 6 = 36 possible outcomes. The sum of the numbers on the two dice equals two only when both dice show a 1. This is just one specific outcome: (1, 1). Therefore, there is only 1 favorable outcome. The probability of the sum being two is the number of favorable outcomes divided by the total number of possible outcomes, which is 1/36.', [](#__codelineno-11-3) answer=0.0277776 [](#__codelineno-11-4) )` | | | | --- | --- | | [1](#__codelineno-12-1)
[2](#__codelineno-12-2)
[3](#__codelineno-12-3)
[4](#__codelineno-12-4)
[5](#__codelineno-12-5)
[6](#__codelineno-12-6)
[7](#__codelineno-12-7)
[8](#__codelineno-12-8) | `def search_wikipedia(query: str) -> list[str]: results = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')(query, k=3) return [x['text'] for x in results] rag = dspy.ChainOfThought('context, question -> response') question = "What's the name of the castle that David Gregory inherited?" rag(context=search_wikipedia(question), question=question)` | **Possible Output:** `[](#__codelineno-13-1) Prediction( [](#__codelineno-13-2) reasoning='The context provides information about David Gregory, a Scottish physician and inventor. It specifically mentions that he inherited Kinnairdy Castle in 1664. This detail directly answers the question about the name of the castle that David Gregory inherited.', [](#__codelineno-13-3) response='Kinnairdy Castle' [](#__codelineno-13-4) )` | | | | --- | --- | | [1](#__codelineno-14-1)
[2](#__codelineno-14-2)
[3](#__codelineno-14-3)
[4](#__codelineno-14-4)
[5](#__codelineno-14-5)
[6](#__codelineno-14-6)
[7](#__codelineno-14-7)
[8](#__codelineno-14-8)
[9](#__codelineno-14-9)
[10](#__codelineno-14-10)
[11](#__codelineno-14-11) | `from typing import Literal class Classify(dspy.Signature): """Classify sentiment of a given sentence.""" sentence: str = dspy.InputField() sentiment: Literal['positive', 'negative', 'neutral'] = dspy.OutputField() confidence: float = dspy.OutputField() classify = dspy.Predict(Classify) classify(sentence="This book was super fun to read, though not the last chapter.")` | **Possible Output:** `[](#__codelineno-15-1) Prediction( [](#__codelineno-15-2) sentiment='positive', [](#__codelineno-15-3) confidence=0.75 [](#__codelineno-15-4) )` | | | | --- | --- | | [1](#__codelineno-16-1)
[2](#__codelineno-16-2)
[3](#__codelineno-16-3)
[4](#__codelineno-16-4)
[5](#__codelineno-16-5)
[6](#__codelineno-16-6)
[7](#__codelineno-16-7)
[8](#__codelineno-16-8)
[9](#__codelineno-16-9)
[10](#__codelineno-16-10)
[11](#__codelineno-16-11)
[12](#__codelineno-16-12)
[13](#__codelineno-16-13)
[14](#__codelineno-16-14)
[15](#__codelineno-16-15)
[16](#__codelineno-16-16)
[17](#__codelineno-16-17) | `class ExtractInfo(dspy.Signature): """Extract structured information from text.""" text: str = dspy.InputField() title: str = dspy.OutputField() headings: list[str] = dspy.OutputField() entities: list[dict[str, str]] = dspy.OutputField(desc="a list of entities and their metadata") module = dspy.Predict(ExtractInfo) text = "Apple Inc. announced its latest iPhone 14 today." \ "The CEO, Tim Cook, highlighted its new features in a press release." response = module(text=text) print(response.title) print(response.headings) print(response.entities)` | **Possible Output:** `[](#__codelineno-17-1) Apple Inc. Announces iPhone 14 [](#__codelineno-17-2) ['Introduction', "CEO's Statement", 'New Features'] [](#__codelineno-17-3) [{'name': 'Apple Inc.', 'type': 'Organization'}, {'name': 'iPhone 14', 'type': 'Product'}, {'name': 'Tim Cook', 'type': 'Person'}]` | | | | --- | --- | | [1](#__codelineno-18-1)
[2](#__codelineno-18-2)
[3](#__codelineno-18-3)
[4](#__codelineno-18-4)
[5](#__codelineno-18-5)
[6](#__codelineno-18-6)
[7](#__codelineno-18-7)
[8](#__codelineno-18-8)
[9](#__codelineno-18-9)
[10](#__codelineno-18-10)
[11](#__codelineno-18-11) | `def evaluate_math(expression: str): return dspy.PythonInterpreter({}).execute(expression) def search_wikipedia(query: str): results = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')(query, k=3) return [x['text'] for x in results] react = dspy.ReAct("question -> answer: float", tools=[evaluate_math, search_wikipedia]) pred = react(question="What is 9362158 divided by the year of birth of David Gregory of Kinnairdy castle?") print(pred.answer)` | **Possible Output:** `[](#__codelineno-19-1) 5761.328` | | | | --- | --- | | [1](#__codelineno-20-1)
[2](#__codelineno-20-2)
[3](#__codelineno-20-3)
[4](#__codelineno-20-4)
[5](#__codelineno-20-5)
[6](#__codelineno-20-6)
[7](#__codelineno-20-7)
[8](#__codelineno-20-8)
[9](#__codelineno-20-9)
[10](#__codelineno-20-10)
[11](#__codelineno-20-11)
[12](#__codelineno-20-12)
[13](#__codelineno-20-13)
[14](#__codelineno-20-14)
[15](#__codelineno-20-15)
[16](#__codelineno-20-16)
[17](#__codelineno-20-17)
[18](#__codelineno-20-18)
[19](#__codelineno-20-19)
[20](#__codelineno-20-20)
[21](#__codelineno-20-21)
[22](#__codelineno-20-22)
[23](#__codelineno-20-23)
[24](#__codelineno-20-24)
[25](#__codelineno-20-25)
[26](#__codelineno-20-26)
[27](#__codelineno-20-27)
[28](#__codelineno-20-28)
[29](#__codelineno-20-29)
[30](#__codelineno-20-30)
[31](#__codelineno-20-31)
[32](#__codelineno-20-32) | `class Outline(dspy.Signature): """Outline a thorough overview of a topic.""" topic: str = dspy.InputField() title: str = dspy.OutputField() sections: list[str] = dspy.OutputField() section_subheadings: dict[str, list[str]] = dspy.OutputField(desc="mapping from section headings to subheadings") class DraftSection(dspy.Signature): """Draft a top-level section of an article.""" topic: str = dspy.InputField() section_heading: str = dspy.InputField() section_subheadings: list[str] = dspy.InputField() content: str = dspy.OutputField(desc="markdown-formatted section") class DraftArticle(dspy.Module): def __init__(self): self.build_outline = dspy.ChainOfThought(Outline) self.draft_section = dspy.ChainOfThought(DraftSection) def forward(self, topic): outline = self.build_outline(topic=topic) sections = [] for heading, subheadings in outline.section_subheadings.items(): section, subheadings = f"## {heading}", [f"### {subheading}" for subheading in subheadings] section = self.draft_section(topic=outline.title, section_heading=section, section_subheadings=subheadings) sections.append(section.content) return dspy.Prediction(title=outline.title, sections=sections) draft_article = DraftArticle() article = draft_article(topic="World Cup 2002")` | **Possible Output:** A 1500-word article on the topic, e.g. `[](#__codelineno-21-1) ## Qualification Process [](#__codelineno-21-2) [](#__codelineno-21-3) The qualification process for the 2002 FIFA World Cup involved a series of..... [shortened here for presentation]. [](#__codelineno-21-4) [](#__codelineno-21-5) ### UEFA Qualifiers [](#__codelineno-21-6) [](#__codelineno-21-7) The UEFA qualifiers involved 50 teams competing for 13..... [shortened here for presentation]. [](#__codelineno-21-8) [](#__codelineno-21-9) .... [rest of the article]` Note that DSPy makes it straightforward to optimize multi-stage modules like this. As long as you can evaluate the _final_ output of the system, every DSPy optimizer can tune all of the intermediate modules. Using DSPy in practice: from quick scripting to building sophisticated systems. Standard prompts conflate interface (“what should the LM do?”) with implementation (“how do we tell it to do that?”). DSPy isolates the former as _signatures_ so we can infer the latter or learn it from data — in the context of a bigger program. Even before you start using optimizers, DSPy's modules allow you to script effective LM systems as ergonomic, portable _code_. Across many tasks and LMs, we maintain _signature test suites_ that assess the reliability of the built-in DSPy adapters. Adapters are the components that map signatures to prompts prior to optimization. If you find a task where a simple prompt consistently outperforms idiomatic DSPy for your LM, consider that a bug and [file an issue](https://github.com/stanfordnlp/dspy/issues) . We'll use this to improve the built-in adapters. 2) **Optimizers** tune the prompts and weights of your AI modules. ------------------------------------------------------------------ DSPy provides you with the tools to compile high-level code with natural language annotations into the low-level computations, prompts, or weight updates that align your LM with your program’s structure and metrics. If you change your code or your metrics, you can simply re-compile accordingly. Given a few tens or hundreds of representative _inputs_ of your task and a _metric_ that can measure the quality of your system's outputs, you can use a DSPy optimizer. Different optimizers in DSPy work by **synthesizing good few-shot examples** for every module, like `dspy.BootstrapRS`,[1](https://arxiv.org/abs/2310.03714) **proposing and intelligently exploring better natural-language instructions** for every prompt, like `dspy.MIPROv2`,[2](https://arxiv.org/abs/2406.11695) and **building datasets for your modules and using them to finetune the LM weights** in your system, like `dspy.BootstrapFinetune`.[3](https://arxiv.org/abs/2407.10930) Getting Started III: Optimizing the LM prompts or weights in DSPy programs A typical simple optimization run costs on the order of $2 USD and takes around 20 minutes, but be careful when running optimizers with very large LMs or very large datasets. Optimization can cost as little as a few cents or up to tens of dollars, depending on your LM, dataset, and configuration. Optimizing prompts for a ReAct agentOptimizing prompts for RAGOptimizing weights for Classification This is a minimal but fully runnable example of setting up a `dspy.ReAct` agent that answers questions via search from Wikipedia and then optimizing it using `dspy.MIPROv2` in the cheap `light` mode on 500 question-answer pairs sampled from the `HotPotQA` dataset. | | | | --- | --- | | [1](#__codelineno-22-1)
[2](#__codelineno-22-2)
[3](#__codelineno-22-3)
[4](#__codelineno-22-4)
[5](#__codelineno-22-5)
[6](#__codelineno-22-6)
[7](#__codelineno-22-7)
[8](#__codelineno-22-8)
[9](#__codelineno-22-9)
[10](#__codelineno-22-10)
[11](#__codelineno-22-11)
[12](#__codelineno-22-12)
[13](#__codelineno-22-13)
[14](#__codelineno-22-14) | `import dspy from dspy.datasets import HotPotQA dspy.configure(lm=dspy.LM('openai/gpt-4o-mini')) def search_wikipedia(query: str) -> list[str]: results = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')(query, k=3) return [x['text'] for x in results] trainset = [x.with_inputs('question') for x in HotPotQA(train_seed=2024, train_size=500).train] react = dspy.ReAct("question -> answer", tools=[search_wikipedia]) tp = dspy.MIPROv2(metric=dspy.evaluate.answer_exact_match, auto="light", num_threads=24) optimized_react = tp.compile(react, trainset=trainset)` | An informal run like this raises ReAct's score from 24% to 51%, by teaching `gpt-4o-mini` more about the specifics of the task. Given a retrieval index to `search`, your favorite `dspy.LM`, and a small `trainset` of questions and ground-truth responses, the following code snippet can optimize your RAG system with long outputs against the built-in `SemanticF1` metric, which is implemented as a DSPy module. | | | | --- | --- | | [1](#__codelineno-23-1)
[2](#__codelineno-23-2)
[3](#__codelineno-23-3)
[4](#__codelineno-23-4)
[5](#__codelineno-23-5)
[6](#__codelineno-23-6)
[7](#__codelineno-23-7)
[8](#__codelineno-23-8)
[9](#__codelineno-23-9)
[10](#__codelineno-23-10)
[11](#__codelineno-23-11) | `class RAG(dspy.Module): def __init__(self, num_docs=5): self.num_docs = num_docs self.respond = dspy.ChainOfThought('context, question -> response') def forward(self, question): context = search(question, k=self.num_docs) # defined in tutorial linked below return self.respond(context=context, question=question) tp = dspy.MIPROv2(metric=dspy.evaluate.SemanticF1(decompositional=True), auto="medium", num_threads=24) optimized_rag = tp.compile(RAG(), trainset=trainset, max_bootstrapped_demos=2, max_labeled_demos=2)` | For a complete RAG example that you can run, start this [tutorial](/tutorials/rag/) . It improves the quality of a RAG system over a subset of StackExchange communities by 10% relative gain. This is a minimal but fully runnable example of setting up a `dspy.ChainOfThought` module that classifies short texts into one of 77 banking labels and then using `dspy.BootstrapFinetune` with 2000 text-label pairs from the `Banking77` to finetune the weights of GPT-4o-mini for this task. We use the variant `dspy.ChainOfThoughtWithHint`, which takes an optional `hint` at bootstrapping time, to maximize the utility of the training data. Naturally, hints are not available at test time. Click to show dataset setup code. | | | | --- | --- | | [1](#__codelineno-24-1)
[2](#__codelineno-24-2)
[3](#__codelineno-24-3)
[4](#__codelineno-24-4)
[5](#__codelineno-24-5)
[6](#__codelineno-24-6)
[7](#__codelineno-24-7)
[8](#__codelineno-24-8)
[9](#__codelineno-24-9)
[10](#__codelineno-24-10)
[11](#__codelineno-24-11)
[12](#__codelineno-24-12)
[13](#__codelineno-24-13)
[14](#__codelineno-24-14)
[15](#__codelineno-24-15) | `import random from typing import Literal from dspy.datasets import DataLoader from datasets import load_dataset # Load the Banking77 dataset. CLASSES = load_dataset("PolyAI/banking77", split="train", trust_remote_code=True).features['label'].names kwargs = dict(fields=("text", "label"), input_keys=("text",), split="train", trust_remote_code=True) # Load the first 2000 examples from the dataset, and assign a hint to each *training* example. trainset = [ dspy.Example(x, hint=CLASSES[x.label], label=CLASSES[x.label]).with_inputs("text", "hint") for x in DataLoader().from_huggingface(dataset_name="PolyAI/banking77", **kwargs)[:2000] ] random.Random(0).shuffle(trainset)` | | | | | --- | --- | | [1](#__codelineno-25-1)
[2](#__codelineno-25-2)
[3](#__codelineno-25-3)
[4](#__codelineno-25-4)
[5](#__codelineno-25-5)
[6](#__codelineno-25-6)
[7](#__codelineno-25-7)
[8](#__codelineno-25-8)
[9](#__codelineno-25-9)
[10](#__codelineno-25-10)
[11](#__codelineno-25-11)
[12](#__codelineno-25-12) | `import dspy dspy.configure(lm=dspy.LM('gpt-4o-mini-2024-07-18')) # Define the DSPy module for classification. It will use the hint at training time, if available. signature = dspy.Signature("text -> label").with_updated_fields('label', type_=Literal[tuple(CLASSES)]) classify = dspy.ChainOfThoughtWithHint(signature) # Optimize via BootstrapFinetune. optimizer = dspy.BootstrapFinetune(metric=(lambda x, y, trace=None: x.label == y.label), num_threads=24) optimized = optimizer.compile(classify, trainset=trainset) optimized_classifier(text="What does a pending cash withdrawal mean?")` | **Possible Output (from the last line):** `[](#__codelineno-26-1) Prediction( [](#__codelineno-26-2) reasoning='A pending cash withdrawal indicates that a request to withdraw cash has been initiated but has not yet been completed or processed. This status means that the transaction is still in progress and the funds have not yet been deducted from the account or made available to the user.', [](#__codelineno-26-3) label='pending_cash_withdrawal' [](#__codelineno-26-4) )` An informal run similar to this on DSPy 2.5.29 raises GPT-4o-mini's score 66% to 87%. What's an example of a DSPy optimizer? How do different optimizers work? Take the `dspy.MIPROv2` optimizer as an example. First, MIPRO starts with the **bootstrapping stage**. It takes your program, which may be unoptimized at this point, and runs it many times across different inputs to collect traces of input/output behavior for each one of your modules. It filters these traces to keep only those that appear in trajectories scored highly by your metric. Second, MIPRO enters its **grounded proposal stage**. It previews your DSPy program's code, your data, and traces from running your program, and uses them to draft many potential instructions for every prompt in your program. Third, MIPRO launches the **discrete search stage**. It samples mini-batches from your training set, proposes a combination of instructions and traces to use for constructing every prompt in the pipeline, and evaluates the candidate program on the mini-batch. Using the resulting score, MIPRO updates a surrogate model that helps the proposals get better over time. One thing that makes DSPy optimizers so powerful is that they can be composed. You can run `dspy.MIPROv2` and use the produced program as an input to `dspy.MIPROv2` again or, say, to `dspy.BootstrapFinetune` to get better results. This is partly the essence of `dspy.BetterTogether`. Alternatively, you can run the optimizer and then extract the top-5 candidate programs and build a `dspy.Ensemble` of them. This allows you to scale _inference-time compute_ (e.g., ensembles) as well as DSPy's unique _pre-inference time compute_ (i.e., optimization budget) in highly systematic ways. 3) **DSPy's Ecosystem** advances open-source AI research. --------------------------------------------------------- Compared to monolithic LMs, DSPy's modular paradigm enables a large community to improve the compositional architectures, inference-time strategies, and optimizers for LM programs in an open, distributed way. This gives DSPy users more control, helps them iterate much faster, and allows their programs to get better over time by applying the latest optimizers or modules. The DSPy research effort started at Stanford NLP in Feb 2022, building on what we had learned from developing early [compound LM systems](https://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems/) like [ColBERT-QA](https://arxiv.org/abs/2007.00814) , [Baleen](https://arxiv.org/abs/2101.00436) , and [Hindsight](https://arxiv.org/abs/2110.07752) . The first version was released as [DSP](https://arxiv.org/abs/2212.14024) in Dec 2022 and evolved by Oct 2023 into [DSPy](https://arxiv.org/abs/2310.03714) . Thanks to [250 contributors](https://github.com/stanfordnlp/dspy/graphs/contributors) , DSPy has introduced tens of thousands of people to building and optimizing modular LM programs. Since then, DSPy's community has produced a large body of work on optimizers, like [MIPROv2](https://arxiv.org/abs/2406.11695) , [BetterTogether](https://arxiv.org/abs/2407.10930) , and [LeReT](https://arxiv.org/abs/2410.23214) , on program architectures, like [STORM](https://arxiv.org/abs/2402.14207) , [IReRa](https://arxiv.org/abs/2401.12178) , and [DSPy Assertions](https://arxiv.org/abs/2312.13382) , and on successful applications to new problems, like [PAPILLON](https://arxiv.org/abs/2410.17127) , [PATH](https://arxiv.org/abs/2406.11706) , [WangLab@MEDIQA](https://arxiv.org/abs/2404.14544) , [UMD's Prompting Case Study](https://arxiv.org/abs/2406.06608) , and [Haize's Red-Teaming Program](https://blog.haizelabs.com/posts/dspy/) , in addition to many open-source projects, production applications, and other [use cases](/dspy-usecases/) . Back to top --- # Tutorials Overview - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/tutorials/index.md "Edit this page") Tutorials Overview ================== * [Retrieval-Augmented Generation](/tutorials/rag/) * [Agents](/tutorials/agents/) * [Reasoning](/tutorials/math/) * [Entity Extraction](/tutorials/entity_extraction/) * [Classification](/tutorials/classification/) * [Multi-Hop Search](/tutorials/multihop_search/) * [Privacy-Conscious Delegation](/tutorials/papillon/) * [Finetuning Agents](/tutorials/games/) * [Saving and Loading](/tutorials/saving/index.md) * [Deployment](/tutorials/deployment/) * [Debugging & Observability](/tutorials/observability/) We are working on upgrading more tutorials and other examples to [DSPy 2.5](https://github.com/stanfordnlp/dspy/blob/main/examples/migration.ipynb) from earlier DSPy versions. Back to top --- # Adapter - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/api/adapters/Adapter.md "Edit this page") dspy.Adapter ============ `dspy.Adapter(callbacks=None)` ------------------------------ Bases: `ABC` Source code in `dspy/adapters/base.py` | | | | --- | --- | | [6](#__codelineno-0-6)
[7](#__codelineno-0-7) | `def __init__(self, callbacks=None): self.callbacks = callbacks or []` | ### Functions #### `__call__(lm, lm_kwargs, signature, demos, inputs)` Source code in `dspy/adapters/base.py` | | | | --- | --- | | [16](#__codelineno-0-16)
[17](#__codelineno-0-17)
[18](#__codelineno-0-18)
[19](#__codelineno-0-19)
[20](#__codelineno-0-20)
[21](#__codelineno-0-21)
[22](#__codelineno-0-22)
[23](#__codelineno-0-23)
[24](#__codelineno-0-24)
[25](#__codelineno-0-25)
[26](#__codelineno-0-26)
[27](#__codelineno-0-27)
[28](#__codelineno-0-28)
[29](#__codelineno-0-29)
[30](#__codelineno-0-30)
[31](#__codelineno-0-31)
[32](#__codelineno-0-32)
[33](#__codelineno-0-33)
[34](#__codelineno-0-34)
[35](#__codelineno-0-35)
[36](#__codelineno-0-36)
[37](#__codelineno-0-37)
[38](#__codelineno-0-38)
[39](#__codelineno-0-39)
[40](#__codelineno-0-40)
[41](#__codelineno-0-41)
[42](#__codelineno-0-42)
[43](#__codelineno-0-43)
[44](#__codelineno-0-44)
[45](#__codelineno-0-45)
[46](#__codelineno-0-46) | `def __call__(self, lm, lm_kwargs, signature, demos, inputs): inputs_ = self.format(signature, demos, inputs) inputs_ = dict(prompt=inputs_) if isinstance(inputs_, str) else dict(messages=inputs_) outputs = lm(**inputs_, **lm_kwargs) values = [] try: for output in outputs: output_logprobs = None if isinstance(output, dict): output, output_logprobs = output["text"], output["logprobs"] value = self.parse(signature, output) assert set(value.keys()) == set(signature.output_fields.keys()), \ f"Expected {signature.output_fields.keys()} but got {value.keys()}" if output_logprobs is not None: value["logprobs"] = output_logprobs values.append(value) return values except Exception as e: from .json_adapter import JSONAdapter if not isinstance(self, JSONAdapter): return JSONAdapter()(lm, lm_kwargs, signature, demos, inputs) raise e` | #### `format(signature, demos, inputs)` `abstractmethod` Source code in `dspy/adapters/base.py` | | | | --- | --- | | [48](#__codelineno-0-48)
[49](#__codelineno-0-49)
[50](#__codelineno-0-50) | `@abstractmethod def format(self, signature, demos, inputs): raise NotImplementedError` | #### `format_finetune_data(signature, demos, inputs, outputs)` Source code in `dspy/adapters/base.py` | | | | --- | --- | | [56](#__codelineno-0-56)
[57](#__codelineno-0-57) | `def format_finetune_data(self, signature, demos, inputs, outputs): raise NotImplementedError` | #### `parse(signature, completion)` `abstractmethod` Source code in `dspy/adapters/base.py` | | | | --- | --- | | [52](#__codelineno-0-52)
[53](#__codelineno-0-53)
[54](#__codelineno-0-54) | `@abstractmethod def parse(self, signature, completion): raise NotImplementedError` | Back to top --- # Programming Overview - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/learn/programming/overview.md "Edit this page") Programming in DSPy =================== DSPy is a bet on _writing code instead of strings_. In other words, building the right control flow is crucial. Start by **defining your task**. What are the inputs to your system and what should your system produce as output? Is it a chatbot over your data or perhaps a code assistant? Or maybe a system for translation, for highlighting snippets from search results, or for generating reports with citations? Next, **define your initial pipeline**. Can your DSPy program just be a single module or do you need to break it down into a few steps? Do you need retrieval or other tools, like a calculator or a calendar API? Is there a typical workflow for solving your problem in multiple well-scoped steps, or do you want more open-ended tool use with agents for your task? Think about these but start simple, perhaps with just a single `dspy.ChainOfThought` module, then add complexity incrementally based on observations. As you do this, **craft and try a handful of examples** of the inputs to your program. Consider using a powerful LM at this point, or a couple of different LMs, just to understand what's possible. Record interesting (both easy and hard) examples you try. This will be useful when you are doing evaluation and optimization later. Beyond encouraging good design patterns, how does DSPy help here? Conventional prompts couple your fundamental system architecture with incidental choices not portable to new LMs, objectives, or pipelines. A conventional prompt asks the LM to take some inputs and produce some outputs of certain types (a _signature_), formats the inputs in certain ways and requests outputs in a form it can parse accurately (an _adapter_), asks the LM to apply certain strategies like "thinking step by step" or using tools (a _module_'s logic), and relies on substantial trial-and-error to discover the right way to ask each LM to do this (a form of manual _optimization_). DSPy separates these concerns and automates the lower-level ones until you need to consider them. This allow you to write much shorter code, with much higher portability. For example, if you write a program using DSPy modules, you can swap the LM or its adapter without changing the rest of your logic. Or you can exchange one _module_, like `dspy.ChainOfThought`, with another, like `dspy.ProgramOfThought`, without modifying your signatures. When you're ready to use optimizers, the same program can have its prompts optimized or its LM weights fine-tuned. Back to top --- # Language Models - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/learn/programming/language_models.md "Edit this page") Language Models =============== The first step in any DSPy code is to set up your language model. For example, you can configure OpenAI's GPT-4o-mini as your default LM as follows. | | | | --- | --- | | [1](#__codelineno-0-1)
[2](#__codelineno-0-2)
[3](#__codelineno-0-3) | ``# Authenticate via `OPENAI_API_KEY` env: import os; os.environ['OPENAI_API_KEY'] = 'here' lm = dspy.LM('openai/gpt-4o-mini') dspy.configure(lm=lm)`` | A few different LMs OpenAIAnthropicDatabricksLocal LMs on a GPU serverLocal LMs on your laptopOther providers You can authenticate by setting the `OPENAI_API_KEY` env variable or passing `api_key` below. | | | | --- | --- | | [1](#__codelineno-1-1)
[2](#__codelineno-1-2)
[3](#__codelineno-1-3) | `import dspy lm = dspy.LM('openai/gpt-4o-mini', api_key='YOUR_OPENAI_API_KEY') dspy.configure(lm=lm)` | You can authenticate by setting the ANTHROPIC\_API\_KEY env variable or passing `api_key` below. | | | | --- | --- | | [1](#__codelineno-2-1)
[2](#__codelineno-2-2)
[3](#__codelineno-2-3) | `import dspy lm = dspy.LM('anthropic/claude-3-opus-20240229', api_key='YOUR_ANTHROPIC_API_KEY') dspy.configure(lm=lm)` | If you're on the Databricks platform, authentication is automatic via their SDK. If not, you can set the env variables `DATABRICKS_API_KEY` and `DATABRICKS_API_BASE`, or pass `api_key` and `api_base` below. | | | | --- | --- | | [1](#__codelineno-3-1)
[2](#__codelineno-3-2)
[3](#__codelineno-3-3) | `import dspy lm = dspy.LM('databricks/databricks-meta-llama-3-1-70b-instruct') dspy.configure(lm=lm)` | First, install [SGLang](https://sgl-project.github.io/start/install.html) and launch its server with your LM. `[](#__codelineno-4-1) > pip install "sglang[all]" [](#__codelineno-4-2) > pip install flashinfer -i https://flashinfer.ai/whl/cu121/torch2.4/ [](#__codelineno-4-3) [](#__codelineno-4-4) > CUDA_VISIBLE_DEVICES=0 python -m sglang.launch_server --port 7501 --model-path meta-llama/Meta-Llama-3-8B-Instruct` Then, connect to it from your DSPy code as an OpenAI-compatible endpoint. | | | | --- | --- | | [1](#__codelineno-5-1)
[2](#__codelineno-5-2)
[3](#__codelineno-5-3)
[4](#__codelineno-5-4) | `lm = dspy.LM("openai/meta-llama/Meta-Llama-3-8B-Instruct", api_base="http://localhost:7501/v1", # ensure this points to your port api_key="", model_type='chat') dspy.configure(lm=lm)` | First, install [Ollama](https://github.com/ollama/ollama) and launch its server with your LM. `[](#__codelineno-6-1) > curl -fsSL https://ollama.ai/install.sh | sh [](#__codelineno-6-2) > ollama run llama3.2:1b` Then, connect to it from your DSPy code. | | | | --- | --- | | [1](#__codelineno-7-1)
[2](#__codelineno-7-2)
[3](#__codelineno-7-3) | `import dspy lm = dspy.LM('ollama_chat/llama3.2', api_base='http://localhost:11434', api_key='') dspy.configure(lm=lm)` | In DSPy, you can use any of the dozens of [LLM providers supported by LiteLLM](https://docs.litellm.ai/docs/providers) . Simply follow their instructions for which `{PROVIDER}_API_KEY` to set and how to write pass the `{provider_name}/{model_name}` to the constructor. Some examples: * `anyscale/mistralai/Mistral-7B-Instruct-v0.1`, with `ANYSCALE_API_KEY` * `together_ai/togethercomputer/llama-2-70b-chat`, with `TOGETHERAI_API_KEY` * `sagemaker/`, with `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, and `AWS_REGION_NAME` * `azure/`, with `AZURE_API_KEY`, `AZURE_API_BASE`, `AZURE_API_VERSION`, and the optional `AZURE_AD_TOKEN` and `AZURE_API_TYPE` If your provider offers an OpenAI-compatible endpoint, just add an `openai/` prefix to your full model name. | | | | --- | --- | | [1](#__codelineno-8-1)
[2](#__codelineno-8-2)
[3](#__codelineno-8-3) | `import dspy lm = dspy.LM('openai/your-model-name', api_key='PROVIDER_API_KEY', api_base='YOUR_PROVIDER_URL') dspy.configure(lm=lm)` | Calling the LM directly. ------------------------ It's easy to call the `lm` you configured above directly. This gives you a unified API and lets you benefit from utilities like automatic caching. | | | | --- | --- | | [1](#__codelineno-9-1)
[2](#__codelineno-9-2) | `lm("Say this is a test!", temperature=0.7) # => ['This is a test!'] lm(messages=[{"role": "user", "content": "Say this is a test!"}]) # => ['This is a test!']` | Using the LM with DSPy modules. ------------------------------- Idiomatic DSPy involves using _modules_, which we discuss in the next guide. | | | | --- | --- | | [1](#__codelineno-10-1)
[2](#__codelineno-10-2)
[3](#__codelineno-10-3)
[4](#__codelineno-10-4)
[5](#__codelineno-10-5)
[6](#__codelineno-10-6) | ``# Define a module (ChainOfThought) and assign it a signature (return an answer, given a question). qa = dspy.ChainOfThought('question -> answer') # Run with the default LM configured with `dspy.configure` above. response = qa(question="How many floors are in the castle David Gregory inherited?") print(response.answer)`` | **Possible Output:** `[](#__codelineno-11-1) The castle David Gregory inherited has 7 floors.` Using multiple LMs. ------------------- You can change the default LM globally with `dspy.configure` or change it inside a block of code with `dspy.context`. Tip Using `dspy.configure` and `dspy.context` is thread-safe! | | | | --- | --- | | [1](#__codelineno-12-1)
[2](#__codelineno-12-2)
[3](#__codelineno-12-3)
[4](#__codelineno-12-4)
[5](#__codelineno-12-5)
[6](#__codelineno-12-6)
[7](#__codelineno-12-7) | `dspy.configure(lm=dspy.LM('openai/gpt-4o-mini')) response = qa(question="How many floors are in the castle David Gregory inherited?") print('GPT-4o-mini:', response.answer) with dspy.context(lm=dspy.LM('openai/gpt-3.5-turbo')): response = qa(question="How many floors are in the castle David Gregory inherited?") print('GPT-3.5-turbo:', response.answer)` | **Possible Output:** `[](#__codelineno-13-1) GPT-4o: The number of floors in the castle David Gregory inherited cannot be determined with the information provided. [](#__codelineno-13-2) GPT-3.5-turbo: The castle David Gregory inherited has 7 floors.` Configuring LM generation. -------------------------- For any LM, you can configure any of the following attributes at initialization or in each subsequent call. | | | | --- | --- | | [1](#__codelineno-14-1) | `gpt_4o_mini = dspy.LM('openai/gpt-4o-mini', temperature=0.9, max_tokens=3000, stop=None, cache=False)` | By default LMs in DSPy are cached. If you repeat the same call, you will get the same outputs. But you can turn off caching by setting `cache=False`. Inspecting output and usage metadata. ------------------------------------- Every LM object maintains the history of its interactions, including inputs, outputs, token usage (and $$$ cost), and metadata. | | | | --- | --- | | [1](#__codelineno-15-1)
[2](#__codelineno-15-2)
[3](#__codelineno-15-3) | `len(lm.history) # e.g., 3 calls to the LM lm.history[-1].keys() # access the last call to the LM, with all metadata` | **Output:** `[](#__codelineno-16-1) dict_keys(['prompt', 'messages', 'kwargs', 'response', 'outputs', 'usage', 'cost'])` ### Advanced: Building customer LMs and writing your own Adapters. Though rarely needed, you can write custom LMs by inheriting from `dspy.BaseLM`. Another advanced layer in the DSPy ecosystem is that of _adapters_, which sit between DSPy signatures and LMs. A future version of this guide will discuss these advanced features, though you likely don't need them. Back to top --- # Modules - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/learn/programming/modules.md "Edit this page") Modules ======= A **DSPy module** is a building block for programs that use LMs. * Each built-in module abstracts a **prompting technique** (like chain of thought or ReAct). Crucially, they are generalized to handle any signature. * A DSPy module has **learnable parameters** (i.e., the little pieces comprising the prompt and the LM weights) and can be invoked (called) to process inputs and return outputs. * Multiple modules can be composed into bigger modules (programs). DSPy modules are inspired directly by NN modules in PyTorch, but applied to LM programs. How do I use a built-in module, like `dspy.Predict` or `dspy.ChainOfThought`? ----------------------------------------------------------------------------- Let's start with the most fundamental module, `dspy.Predict`. Internally, all other DSPy modules are built using `dspy.Predict`. We'll assume you are already at least a little familiar with [DSPy signatures](/building-blocks/2-signatures) , which are declarative specs for defining the behavior of any module we use in DSPy. To use a module, we first **declare** it by giving it a signature. Then we **call** the module with the input arguments, and extract the output fields! `[](#__codelineno-0-1) sentence = "it's a charming and often affecting journey." # example from the SST-2 dataset. [](#__codelineno-0-2) [](#__codelineno-0-3) # 1) Declare with a signature. [](#__codelineno-0-4) classify = dspy.Predict('sentence -> sentiment: bool') [](#__codelineno-0-5) [](#__codelineno-0-6) # 2) Call with input argument(s). [](#__codelineno-0-7) response = classify(sentence=sentence) [](#__codelineno-0-8) [](#__codelineno-0-9) # 3) Access the output. [](#__codelineno-0-10) print(response.sentiment)` **Output:** `[](#__codelineno-1-1) True` When we declare a module, we can pass configuration keys to it. Below, we'll pass `n=5` to request five completions. We can also pass `temperature` or `max_len`, etc. Let's use `dspy.ChainOfThought`. In many cases, simply swapping `dspy.ChainOfThought` in place of `dspy.Predict` improves quality. `[](#__codelineno-2-1) question = "What's something great about the ColBERT retrieval model?" [](#__codelineno-2-2) [](#__codelineno-2-3) # 1) Declare with a signature, and pass some config. [](#__codelineno-2-4) classify = dspy.ChainOfThought('question -> answer', n=5) [](#__codelineno-2-5) [](#__codelineno-2-6) # 2) Call with input argument. [](#__codelineno-2-7) response = classify(question=question) [](#__codelineno-2-8) [](#__codelineno-2-9) # 3) Access the outputs. [](#__codelineno-2-10) response.completions.answer` **Possible Output:** `[](#__codelineno-3-1) ['One great thing about the ColBERT retrieval model is its superior efficiency and effectiveness compared to other models.', [](#__codelineno-3-2) 'Its ability to efficiently retrieve relevant information from large document collections.', [](#__codelineno-3-3) 'One great thing about the ColBERT retrieval model is its superior performance compared to other models and its efficient use of pre-trained language models.', [](#__codelineno-3-4) 'One great thing about the ColBERT retrieval model is its superior efficiency and accuracy compared to other models.', [](#__codelineno-3-5) 'One great thing about the ColBERT retrieval model is its ability to incorporate user feedback and support complex queries.']` Let's discuss the output object here. The `dspy.ChainOfThought` module will generally inject a `reasoning` before the output field(s) of your signature. Let's inspect the (first) reasoning and answer! `[](#__codelineno-4-1) print(f"Reasoning: {response.reasoning}") [](#__codelineno-4-2) print(f"Answer: {response.answer}")` **Possible Output:** `[](#__codelineno-5-1) Reasoning: We can consider the fact that ColBERT has shown to outperform other state-of-the-art retrieval models in terms of efficiency and effectiveness. It uses contextualized embeddings and performs document retrieval in a way that is both accurate and scalable. [](#__codelineno-5-2) Answer: One great thing about the ColBERT retrieval model is its superior efficiency and effectiveness compared to other models.` This is accessible whether we request one or many completions. We can also access the different completions as a list of `Prediction`s or as several lists, one for each field. `[](#__codelineno-6-1) response.completions[3].reasoning == response.completions.reasoning[3]` **Output:** `[](#__codelineno-7-1) True` What other DSPy modules are there? How can I use them? ------------------------------------------------------ The others are very similar. They mainly change the internal behavior with which your signature is implemented! 1. **`dspy.Predict`**: Basic predictor. Does not modify the signature. Handles the key forms of learning (i.e., storing the instructions and demonstrations and updates to the LM). 2. **`dspy.ChainOfThought`**: Teaches the LM to think step-by-step before committing to the signature's response. 3. **`dspy.ProgramOfThought`**: Teaches the LM to output code, whose execution results will dictate the response. 4. **`dspy.ReAct`**: An agent that can use tools to implement the given signature. 5. **`dspy.MultiChainComparison`**: Can compare multiple outputs from `ChainOfThought` to produce a final prediction. We also have some function-style modules: 1. **`dspy.majority`**: Can do basic voting to return the most popular response from a set of predictions. A few examples of DSPy modules on simple tasks. Try the examples below after configuring your `lm`. Adjust the fields to explore what tasks your LM can do well out of the box. MathRetrieval-Augmented GenerationClassificationInformation ExtractionAgents | | | | --- | --- | | [1](#__codelineno-8-1)
[2](#__codelineno-8-2) | `math = dspy.ChainOfThought("question -> answer: float") math(question="Two dice are tossed. What is the probability that the sum equals two?")` | **Possible Output:** `[](#__codelineno-9-1) Prediction( [](#__codelineno-9-2) reasoning='When two dice are tossed, each die has 6 faces, resulting in a total of 6 x 6 = 36 possible outcomes. The sum of the numbers on the two dice equals two only when both dice show a 1. This is just one specific outcome: (1, 1). Therefore, there is only 1 favorable outcome. The probability of the sum being two is the number of favorable outcomes divided by the total number of possible outcomes, which is 1/36.', [](#__codelineno-9-3) answer=0.0277776 [](#__codelineno-9-4) )` | | | | --- | --- | | [1](#__codelineno-10-1)
[2](#__codelineno-10-2)
[3](#__codelineno-10-3)
[4](#__codelineno-10-4)
[5](#__codelineno-10-5)
[6](#__codelineno-10-6)
[7](#__codelineno-10-7)
[8](#__codelineno-10-8)
[9](#__codelineno-10-9) | `def search(query: str) -> list[str]: """Retrieves abstracts from Wikipedia.""" results = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')(query, k=3) return [x['text'] for x in results] rag = dspy.ChainOfThought('context, question -> response') question = "What's the name of the castle that David Gregory inherited?" rag(context=search(question), question=question)` | **Possible Output:** `[](#__codelineno-11-1) Prediction( [](#__codelineno-11-2) reasoning='The context provides information about David Gregory, a Scottish physician and inventor. It specifically mentions that he inherited Kinnairdy Castle in 1664. This detail directly answers the question about the name of the castle that David Gregory inherited.', [](#__codelineno-11-3) response='Kinnairdy Castle' [](#__codelineno-11-4) )` | | | | --- | --- | | [1](#__codelineno-12-1)
[2](#__codelineno-12-2)
[3](#__codelineno-12-3)
[4](#__codelineno-12-4)
[5](#__codelineno-12-5)
[6](#__codelineno-12-6)
[7](#__codelineno-12-7)
[8](#__codelineno-12-8)
[9](#__codelineno-12-9)
[10](#__codelineno-12-10)
[11](#__codelineno-12-11) | `from typing import Literal class Classify(dspy.Signature): """Classify sentiment of a given sentence.""" sentence: str = dspy.InputField() sentiment: Literal['positive', 'negative', 'neutral'] = dspy.OutputField() confidence: float = dspy.OutputField() classify = dspy.Predict(Classify) classify(sentence="This book was super fun to read, though not the last chapter.")` | **Possible Output:** `[](#__codelineno-13-1) Prediction( [](#__codelineno-13-2) sentiment='positive', [](#__codelineno-13-3) confidence=0.75 [](#__codelineno-13-4) )` | | | | --- | --- | | [1](#__codelineno-14-1)
[2](#__codelineno-14-2)
[3](#__codelineno-14-3)
[4](#__codelineno-14-4)
[5](#__codelineno-14-5)
[6](#__codelineno-14-6)
[7](#__codelineno-14-7)
[8](#__codelineno-14-8) | `text = "Apple Inc. announced its latest iPhone 14 today. The CEO, Tim Cook, highlighted its new features in a press release." module = dspy.Predict("text -> title, headings: list[str], entities_and_metadata: list[dict[str, str]]") response = module(text=text) print(response.title) print(response.headings) print(response.entities_and_metadata)` | **Possible Output:** `[](#__codelineno-15-1) Apple Unveils iPhone 14 [](#__codelineno-15-2) ['Introduction', 'Key Features', "CEO's Statement"] [](#__codelineno-15-3) [{'entity': 'Apple Inc.', 'type': 'Organization'}, {'entity': 'iPhone 14', 'type': 'Product'}, {'entity': 'Tim Cook', 'type': 'Person'}]` | | | | --- | --- | | [1](#__codelineno-16-1)
[2](#__codelineno-16-2)
[3](#__codelineno-16-3)
[4](#__codelineno-16-4)
[5](#__codelineno-16-5)
[6](#__codelineno-16-6)
[7](#__codelineno-16-7)
[8](#__codelineno-16-8)
[9](#__codelineno-16-9)
[10](#__codelineno-16-10)
[11](#__codelineno-16-11) | `def evaluate_math(expression: str) -> float: return dspy.PythonInterpreter({}).execute(expression) def search_wikipedia(query: str) -> str: results = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')(query, k=3) return [x['text'] for x in results] react = dspy.ReAct("question -> answer: float", tools=[evaluate_math, search_wikipedia]) pred = react(question="What is 9362158 divided by the year of birth of David Gregory of Kinnairdy castle?") print(pred.answer)` | **Possible Output:** `[](#__codelineno-17-1) 5761.328` How do I compose multiple modules into a bigger program? -------------------------------------------------------- DSPy is just Python code that uses modules in any control flow you like, with a little magic internally at `compile` time to trace your LM calls. What this means is that, you can just call the modules freely. See tutorials like [multi-hop search](https://dspy.ai/tutorials/multihop_search/) , whose module is reproduced below as an example. | | | | --- | --- | | [1](#__codelineno-18-1)
[2](#__codelineno-18-2)
[3](#__codelineno-18-3)
[4](#__codelineno-18-4)
[5](#__codelineno-18-5)
[6](#__codelineno-18-6)
[7](#__codelineno-18-7)
[8](#__codelineno-18-8)
[9](#__codelineno-18-9)
[10](#__codelineno-18-10)
[11](#__codelineno-18-11)
[12](#__codelineno-18-12)
[13](#__codelineno-18-13)
[14](#__codelineno-18-14)
[15](#__codelineno-18-15)
[16](#__codelineno-18-16)
[17](#__codelineno-18-17)
[18](#__codelineno-18-18) | `class Hop(dspy.Module): def __init__(self, num_docs=10, num_hops=4): self.num_docs, self.num_hops = num_docs, num_hops self.generate_query = dspy.ChainOfThought('claim, notes -> query') self.append_notes = dspy.ChainOfThought('claim, notes, context -> new_notes: list[str], titles: list[str]') def forward(self, claim: str) -> list[str]: notes = [] titles = [] for _ in range(self.num_hops): query = self.generate_query(claim=claim, notes=notes).query context = search(query, k=self.num_docs) prediction = self.append_notes(claim=claim, notes=notes, context=context) notes.extend(prediction.new_notes) titles.extend(prediction.titles) return dspy.Prediction(notes=notes, titles=list(set(titles)))` | Back to top --- # Signatures - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/learn/programming/signatures.md "Edit this page") Signatures ========== When we assign tasks to LMs in DSPy, we specify the behavior we need as a Signature. **A signature is a declarative specification of input/output behavior of a DSPy module.** Signatures allow you to tell the LM _what_ it needs to do, rather than specify _how_ we should ask the LM to do it. You're probably familiar with function signatures, which specify the input and output arguments and their types. DSPy signatures are similar, but with a couple of differences. While typical function signatures just _describe_ things, DSPy Signatures _declare and initialize the behavior_ of modules. Moreover, the field names matter in DSPy Signatures. You express semantic roles in plain English: a `question` is different from an `answer`, a `sql_query` is different from `python_code`. Why should I use a DSPy Signature? ---------------------------------- For modular and clean code, in which LM calls can be optimized into high-quality prompts (or automatic finetunes). Most people coerce LMs to do tasks by hacking long, brittle prompts. Or by collecting/generating data for fine-tuning. Writing signatures is far more modular, adaptive, and reproducible than hacking at prompts or finetunes. The DSPy compiler will figure out how to build a highly-optimized prompt for your LM (or finetune your small LM) for your signature, on your data, and within your pipeline. In many cases, we found that compiling leads to better prompts than humans write. Not because DSPy optimizers are more creative than humans, but simply because they can try more things and tune the metrics directly. **Inline** DSPy Signatures -------------------------- Signatures can be defined as a short string, with argument names and optional types that define semantic roles for inputs/outputs. 1. Question Answering: `"question -> answer"`, which is equivalent to `"question: str -> answer: str"` as the default type is always `str` 2. Sentiment Classification: `"sentence -> sentiment: bool"`, e.g. `True` if positive 3. Summarization: `"document -> summary"` Your signatures can also have multiple input/output fields with types: 1. Retrieval-Augmented Question Answering: `"context: list[str], question: str -> answer: str"` 2. Multiple-Choice Question Answering with Reasoning: `"question, choices: list[str] -> reasoning: str, selection: int"` **Tip:** For fields, any valid variable names work! Field names should be semantically meaningful, but start simple and don't prematurely optimize keywords! Leave that kind of hacking to the DSPy compiler. For example, for summarization, it's probably fine to say `"document -> summary"`, `"text -> gist"`, or `"long_context -> tldr"`. ### Example A: Sentiment Classification `[](#__codelineno-0-1) sentence = "it's a charming and often affecting journey." # example from the SST-2 dataset. [](#__codelineno-0-2) [](#__codelineno-0-3) classify = dspy.Predict('sentence -> sentiment: bool') # we'll see an example with Literal[] later [](#__codelineno-0-4) classify(sentence=sentence).sentiment` **Output:** `[](#__codelineno-1-1) True` ### Example B: Summarization `[](#__codelineno-2-1) # Example from the XSum dataset. [](#__codelineno-2-2) document = """The 21-year-old made seven appearances for the Hammers and netted his only goal for them in a Europa League qualification round match against Andorran side FC Lustrains last season. Lee had two loan spells in League One last term, with Blackpool and then Colchester United. He scored twice for the U's but was unable to save them from relegation. The length of Lee's contract with the promoted Tykes has not been revealed. Find all the latest football transfers on our dedicated page.""" [](#__codelineno-2-3) [](#__codelineno-2-4) summarize = dspy.ChainOfThought('document -> summary') [](#__codelineno-2-5) response = summarize(document=document) [](#__codelineno-2-6) [](#__codelineno-2-7) print(response.summary)` **Possible Output:** `[](#__codelineno-3-1) The 21-year-old Lee made seven appearances and scored one goal for West Ham last season. He had loan spells in League One with Blackpool and Colchester United, scoring twice for the latter. He has now signed a contract with Barnsley, but the length of the contract has not been revealed.` Many DSPy modules (except `dspy.Predict`) return auxiliary information by expanding your signature under the hood. For example, `dspy.ChainOfThought` also adds a `reasoning` field that includes the LM's reasoning before it generates the output `summary`. `[](#__codelineno-4-1) print("Reasoning:", response.reasoning)` **Possible Output:** `[](#__codelineno-5-1) Reasoning: We need to highlight Lee's performance for West Ham, his loan spells in League One, and his new contract with Barnsley. We also need to mention that his contract length has not been disclosed.` **Class-based** DSPy Signatures ------------------------------- For some advanced tasks, you need more verbose signatures. This is typically to: 1. Clarify something about the nature of the task (expressed below as a `docstring`). 2. Supply hints on the nature of an input field, expressed as a `desc` keyword argument for `dspy.InputField`. 3. Supply constraints on an output field, expressed as a `desc` keyword argument for `dspy.OutputField`. ### Example C: Classification `[](#__codelineno-6-1) from typing import Literal [](#__codelineno-6-2) [](#__codelineno-6-3) class Emotion(dspy.Signature): [](#__codelineno-6-4) """Classify emotion.""" [](#__codelineno-6-5) [](#__codelineno-6-6) sentence: str = dspy.InputField() [](#__codelineno-6-7) sentiment: Literal['sadness', 'joy', 'love', 'anger', 'fear', 'surprise'] = dspy.OutputField() [](#__codelineno-6-8) [](#__codelineno-6-9) sentence = "i started feeling a little vulnerable when the giant spotlight started blinding me" # from dair-ai/emotion [](#__codelineno-6-10) [](#__codelineno-6-11) classify = dspy.Predict(Emotion) [](#__codelineno-6-12) classify(sentence=sentence)` **Possible Output:** `[](#__codelineno-7-1) Prediction( [](#__codelineno-7-2) sentiment='fear' [](#__codelineno-7-3) )` **Tip:** There's nothing wrong with specifying your requests to the LM more clearly. Class-based Signatures help you with that. However, don't prematurely tune the keywords of your signature by hand. The DSPy optimizers will likely do a better job (and will transfer better across LMs). ### Example D: A metric that evaluates faithfulness to citations `[](#__codelineno-8-1) class CheckCitationFaithfulness(dspy.Signature): [](#__codelineno-8-2) """Verify that the text is based on the provided context.""" [](#__codelineno-8-3) [](#__codelineno-8-4) context: str = dspy.InputField(desc="facts here are assumed to be true") [](#__codelineno-8-5) text: str = dspy.InputField() [](#__codelineno-8-6) faithfulness: bool = dspy.OutputField() [](#__codelineno-8-7) evidence: dict[str, list[str]] = dspy.OutputField(desc="Supporting evidence for claims") [](#__codelineno-8-8) [](#__codelineno-8-9) context = "The 21-year-old made seven appearances for the Hammers and netted his only goal for them in a Europa League qualification round match against Andorran side FC Lustrains last season. Lee had two loan spells in League One last term, with Blackpool and then Colchester United. He scored twice for the U's but was unable to save them from relegation. The length of Lee's contract with the promoted Tykes has not been revealed. Find all the latest football transfers on our dedicated page." [](#__codelineno-8-10) [](#__codelineno-8-11) text = "Lee scored 3 goals for Colchester United." [](#__codelineno-8-12) [](#__codelineno-8-13) faithfulness = dspy.ChainOfThought(CheckCitationFaithfulness) [](#__codelineno-8-14) faithfulness(context=context, text=text)` **Possible Output:** `[](#__codelineno-9-1) Prediction( [](#__codelineno-9-2) reasoning="Let's check the claims against the context. The text states Lee scored 3 goals for Colchester United, but the context clearly states 'He scored twice for the U's'. This is a direct contradiction.", [](#__codelineno-9-3) faithfulness=False, [](#__codelineno-9-4) evidence={'goal_count': ["scored twice for the U's"]} [](#__codelineno-9-5) )` Using signatures to build modules & compiling them -------------------------------------------------- While signatures are convenient for prototyping with structured inputs/outputs, that's not the only reason to use them! You should compose multiple signatures into bigger [DSPy modules](../modules/) and [compile these modules into optimized prompts](../../optimization/optimizers/) and finetunes. Back to top --- # Evaluation Overview - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/learn/evaluation/overview.md "Edit this page") Evaluation in DSPy ================== Once you have an initial system, it's time to **collect an initial development set** so you can refine it more systematically. Even 20 input examples of your task can be useful, though 200 goes a long way. Depending on your _metric_, you either just need inputs and no labels at all, or you need inputs and the _final_ outputs of your system. (You almost never need labels for the intermediate steps in your program in DSPy.) You can probably find datasets that are adjacent to your task on, say, HuggingFace datasets or in a naturally occuring source like StackExchange. If there's data whose licenses are permissive enough, we suggest you use them. Otherwise, you can label a few examples by hand or start deploying a demo of your system and collect initial data that way. Next, you should **define your DSPy metric**. What makes outputs from your system good or bad? Invest in defining metrics and improving them incrementally over time; it's hard to consistently improve what you aren't able to define. A metric is a function that takes examples from your data and takes the output of your system, and returns a score. For simple tasks, this could be just "accuracy", e.g. for simple classification or short-form QA tasks. For most applications, your system will produce long-form outputs, so your metric will be a smaller DSPy program that checks multiple properties of the output. Getting this right on the first try is unlikely: start with something simple and iterate. Now that you have some data and a metric, run development evaluations on your pipeline designs to understand their tradeoffs. Look at the outputs and the metric scores. This will probably allow you to spot any major issues, and it will define a baseline for your next steps. If your metric is itself a DSPy program... If your metric is itself a DSPy program, a powerful way to iterate is to optimize your metric itself. That's usually easy because the output of the metric is usually a simple value (e.g., a score out of 5), so the metric's metric is easy to define and optimize by collecting a few examples. Back to top --- # Data Handling - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/learn/evaluation/data.md "Edit this page") Data ==== DSPy is a machine learning framework, so working in it involves training sets, development sets, and test sets. For each example in your data, we distinguish typically between three types of values: the inputs, the intermediate labels, and the final label. You can use DSPy effectively without any intermediate or final labels, but you will need at least a few example inputs. DSPy `Example` objects ---------------------- The core data type for data in DSPy is `Example`. You will use **Examples** to represent items in your training set and test set. DSPy **Examples** are similar to Python `dict`s but have a few useful utilities. Your DSPy modules will return values of the type `Prediction`, which is a special sub-class of `Example`. When you use DSPy, you will do a lot of evaluation and optimization runs. Your individual datapoints will be of type `Example`: `[](#__codelineno-0-1) qa_pair = dspy.Example(question="This is a question?", answer="This is an answer.") [](#__codelineno-0-2) [](#__codelineno-0-3) print(qa_pair) [](#__codelineno-0-4) print(qa_pair.question) [](#__codelineno-0-5) print(qa_pair.answer)` **Output:** `[](#__codelineno-1-1) Example({'question': 'This is a question?', 'answer': 'This is an answer.'}) (input_keys=None) [](#__codelineno-1-2) This is a question? [](#__codelineno-1-3) This is an answer.` Examples can have any field keys and any value types, though usually values are strings. `[](#__codelineno-2-1) object = Example(field1=value1, field2=value2, field3=value3, ...)` You can now express your training set for example as: `[](#__codelineno-3-1) trainset = [dspy.Example(report="LONG REPORT 1", summary="short summary 1"), ...]` ### Specifying Input Keys In traditional ML, there are separated "inputs" and "labels". In DSPy, the `Example` objects have a `with_inputs()` method, which can mark specific fields as inputs. (The rest are just metadata or labels.) `[](#__codelineno-4-1) # Single Input. [](#__codelineno-4-2) print(qa_pair.with_inputs("question")) [](#__codelineno-4-3) [](#__codelineno-4-4) # Multiple Inputs; be careful about marking your labels as inputs unless you mean it. [](#__codelineno-4-5) print(qa_pair.with_inputs("question", "answer"))` Values can be accessed using the `.`(dot) operator. You can access the value of key `name` in defined object `Example(name="John Doe", job="sleep")` through `object.name`. To access or exclude certain keys, use `inputs()` and `labels()` methods to return new Example objects containing only input or non-input keys, respectively. `[](#__codelineno-5-1) article_summary = dspy.Example(article= "This is an article.", summary= "This is a summary.").with_inputs("article") [](#__codelineno-5-2) [](#__codelineno-5-3) input_key_only = article_summary.inputs() [](#__codelineno-5-4) non_input_key_only = article_summary.labels() [](#__codelineno-5-5) [](#__codelineno-5-6) print("Example object with Input fields only:", input_key_only) [](#__codelineno-5-7) print("Example object with Non-Input fields only:", non_input_key_only)` **Output** `[](#__codelineno-6-1) Example object with Input fields only: Example({'article': 'This is an article.'}) (input_keys=None) [](#__codelineno-6-2) Example object with Non-Input fields only: Example({'summary': 'This is a summary.'}) (input_keys=None)` Back to top --- # Metrics - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/learn/evaluation/metrics.md "Edit this page") Metrics ======= DSPy is a machine learning framework, so you must think about your **automatic metrics** for evaluation (to track your progress) and optimization (so DSPy can make your programs more effective). What is a metric and how do I define a metric for my task? ---------------------------------------------------------- A metric is just a function that will take examples from your data and the output of your system and return a score that quantifies how good the output is. What makes outputs from your system good or bad? For simple tasks, this could be just "accuracy" or "exact match" or "F1 score". This may be the case for simple classification or short-form QA tasks. However, for most applications, your system will output long-form outputs. There, your metric should probably be a smaller DSPy program that checks multiple properties of the output (quite possibly using AI feedback from LMs). Getting this right on the first try is unlikely, but you should start with something simple and iterate. Simple metrics -------------- A DSPy metric is just a function in Python that takes `example` (e.g., from your training or dev set) and the output `pred` from your DSPy program, and outputs a `float` (or `int` or `bool`) score. Your metric should also accept an optional third argument called `trace`. You can ignore this for a moment, but it will enable some powerful tricks if you want to use your metric for optimization. Here's a simple example of a metric that's comparing `example.answer` and `pred.answer`. This particular metric will return a `bool`. `[](#__codelineno-0-1) def validate_answer(example, pred, trace=None): [](#__codelineno-0-2) return example.answer.lower() == pred.answer.lower()` Some people find these utilities (built-in) convenient: * `dspy.evaluate.metrics.answer_exact_match` * `dspy.evaluate.metrics.answer_passage_match` Your metrics could be more complex, e.g. check for multiple properties. The metric below will return a `float` if `trace is None` (i.e., if it's used for evaluation or optimization), and will return a `bool` otherwise (i.e., if it's used to bootstrap demonstrations). `[](#__codelineno-1-1) def validate_context_and_answer(example, pred, trace=None): [](#__codelineno-1-2) # check the gold label and the predicted answer are the same [](#__codelineno-1-3) answer_match = example.answer.lower() == pred.answer.lower() [](#__codelineno-1-4) [](#__codelineno-1-5) # check the predicted answer comes from one of the retrieved contexts [](#__codelineno-1-6) context_match = any((pred.answer.lower() in c) for c in pred.context) [](#__codelineno-1-7) [](#__codelineno-1-8) if trace is None: # if we're doing evaluation or optimization [](#__codelineno-1-9) return (answer_match + context_match) / 2.0 [](#__codelineno-1-10) else: # if we're doing bootstrapping, i.e. self-generating good demonstrations of each step [](#__codelineno-1-11) return answer_match and context_match` Defining a good metric is an iterative process, so doing some initial evaluations and looking at your data and outputs is key. Evaluation ---------- Once you have a metric, you can run evaluations in a simple Python loop. `[](#__codelineno-2-1) scores = [] [](#__codelineno-2-2) for x in devset: [](#__codelineno-2-3) pred = program(**x.inputs()) [](#__codelineno-2-4) score = metric(x, pred) [](#__codelineno-2-5) scores.append(score)` If you need some utilities, you can also use the built-in `Evaluate` utility. It can help with things like parallel evaluation (multiple threads) or showing you a sample of inputs/outputs and the metric scores. `[](#__codelineno-3-1) from dspy.evaluate import Evaluate [](#__codelineno-3-2) [](#__codelineno-3-3) # Set up the evaluator, which can be re-used in your code. [](#__codelineno-3-4) evaluator = Evaluate(devset=YOUR_DEVSET, num_threads=1, display_progress=True, display_table=5) [](#__codelineno-3-5) [](#__codelineno-3-6) # Launch evaluation. [](#__codelineno-3-7) evaluator(YOUR_PROGRAM, metric=YOUR_METRIC)` Intermediate: Using AI feedback for your metric ----------------------------------------------- For most applications, your system will output long-form outputs, so your metric should check multiple dimensions of the output using AI feedback from LMs. This simple signature could come in handy. `[](#__codelineno-4-1) # Define the signature for automatic assessments. [](#__codelineno-4-2) class Assess(dspy.Signature): [](#__codelineno-4-3) """Assess the quality of a tweet along the specified dimension.""" [](#__codelineno-4-4) [](#__codelineno-4-5) assessed_text = dspy.InputField() [](#__codelineno-4-6) assessment_question = dspy.InputField() [](#__codelineno-4-7) assessment_answer: bool = dspy.OutputField()` For example, below is a simple metric that checks a generated tweet (1) answers a given question correctly and (2) whether it's also engaging. We also check that (3) `len(tweet) <= 280` characters. ``[](#__codelineno-5-1) def metric(gold, pred, trace=None): [](#__codelineno-5-2) question, answer, tweet = gold.question, gold.answer, pred.output [](#__codelineno-5-3) [](#__codelineno-5-4) engaging = "Does the assessed text make for a self-contained, engaging tweet?" [](#__codelineno-5-5) correct = f"The text should answer `{question}` with `{answer}`. Does the assessed text contain this answer?" [](#__codelineno-5-6) [](#__codelineno-5-7) correct = dspy.Predict(Assess)(assessed_text=tweet, assessment_question=correct) [](#__codelineno-5-8) engaging = dspy.Predict(Assess)(assessed_text=tweet, assessment_question=engaging) [](#__codelineno-5-9) [](#__codelineno-5-10) correct, engaging = [m.assessment_answer for m in [correct, engaging]] [](#__codelineno-5-11) score = (correct + engaging) if correct and (len(tweet) <= 280) else 0 [](#__codelineno-5-12) [](#__codelineno-5-13) if trace is not None: return score >= 2 [](#__codelineno-5-14) return score / 2.0`` When compiling, `trace is not None`, and we want to be strict about judging things, so we will only return `True` if `score >= 2`. Otherwise, we return a score out of 1.0 (i.e., `score / 2.0`). Advanced: Using a DSPy program as your metric --------------------------------------------- If your metric is itself a DSPy program, one of the most powerful ways to iterate is to compile (optimize) your metric itself. That's usually easy because the output of the metric is usually a simple value (e.g., a score out of 5) so the metric's metric is easy to define and optimize by collecting a few examples. ### Advanced: Accessing the `trace` When your metric is used during evaluation runs, DSPy will not try to track the steps of your program. But during compiling (optimization), DSPy will trace your LM calls. The trace will contain inputs/outputs to each DSPy predictor and you can leverage that to validate intermediate steps for optimization. `[](#__codelineno-6-1) def validate_hops(example, pred, trace=None): [](#__codelineno-6-2) hops = [example.question] + [outputs.query for *_, outputs in trace if 'query' in outputs] [](#__codelineno-6-3) [](#__codelineno-6-4) if max([len(h) for h in hops]) > 100: return False [](#__codelineno-6-5) if any(dspy.evaluate.answer_exact_match_str(hops[idx], hops[:idx], frac=0.8) for idx in range(2, len(hops))): return False [](#__codelineno-6-6) [](#__codelineno-6-7) return True` Back to top --- # Optimization Overview - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/learn/optimization/overview.md "Edit this page") Optimization in DSPy ==================== Once you have a system and a way to evaluate it, you can use DSPy optimizers to tune the prompts or weights in your program. Now it's useful to expand your data collection effort into building a training set and a held-out test set, in addition to the development set you've been using for exploration. For the training set (and its subset, validation set), you can often get substantial value out of 30 examples, but aim for at least 300 examples. Some optimizers accept a `trainset` only. Others ask for a `trainset` and a `valset`. For prompt optimizers, we suggest starting with a 20% split for training and 80% for validation, which is often the _opposite_ of what one does for DNNs. After your first few optimization runs, you are either very happy with everything or you've made a lot of progress but you don't like something about the final program or the metric. At this point, go back to step 1 (Programming in DSPy) and revisit the major questions. Did you define your task well? Do you need to collect (or find online) more data for your problem? Do you want to update your metric? And do you want to use a more sophisticated optimizer? Do you need to consider advanced features like DSPy Assertions? Or, perhaps most importantly, do you want to add some more complexity or steps in your DSPy program itself? Do you want to use multiple optimizers in a sequence? Iterative development is key. DSPy gives you the pieces to do that incrementally: iterating on your data, your program structure, your assertions, your metric, and your optimization steps. Optimizing complex LM programs is an entirely new paradigm that only exists in DSPy at the time of writing (update: there are now numerous DSPy extension frameworks, so this part is no longer true :-), so naturally the norms around what to do are still emerging. If you need help, we recently created a [Discord server](https://discord.gg/XCGy2WDCQB) for the community. Back to top --- # Optimizers - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/learn/optimization/optimizers.md "Edit this page") DSPy Optimizers (formerly Teleprompters) ======================================== A **DSPy optimizer** is an algorithm that can tune the parameters of a DSPy program (i.e., the prompts and/or the LM weights) to maximize the metrics you specify, like accuracy. A typical DSPy optimizer takes three things: * Your **DSPy program**. This may be a single module (e.g., `dspy.Predict`) or a complex multi-module program. * Your **metric**. This is a function that evaluates the output of your program, and assigns it a score (higher is better). * A few **training inputs**. This may be very small (i.e., only 5 or 10 examples) and incomplete (only inputs to your program, without any labels). If you happen to have a lot of data, DSPy can leverage that. But you can start small and get strong results. **Note:** Formerly called teleprompters. We are making an official name update, which will be reflected throughout the library and documentation. What does a DSPy Optimizer tune? How does it tune them? ------------------------------------------------------- Different optimizers in DSPy will tune your program's quality by **synthesizing good few-shot examples** for every module, like `dspy.BootstrapRS`,[1](https://arxiv.org/abs/2310.03714) **proposing and intelligently exploring better natural-language instructions** for every prompt, like `dspy.MIPROv2`,[2](https://arxiv.org/abs/2406.11695) and **building datasets for your modules and using them to finetune the LM weights** in your system, like `dspy.BootstrapFinetune`.[3](https://arxiv.org/abs/2407.10930) What's an example of a DSPy optimizer? How do different optimizers work? Take the `dspy.MIPROv2` optimizer as an example. First, MIPRO starts with the **bootstrapping stage**. It takes your program, which may be unoptimized at this point, and runs it many times across different inputs to collect traces of input/output behavior for each one of your modules. It filters these traces to keep only those that appear in trajectories scored highly by your metric. Second, MIPRO enters its **grounded proposal stage**. It previews your DSPy program's code, your data, and traces from running your program, and uses them to draft many potential instructions for every prompt in your program. Third, MIPRO launches the **discrete search stage**. It samples mini-batches from your training set, proposes a combination of instructions and traces to use for constructing every prompt in the pipeline, and evaluates the candidate program on the mini-batch. Using the resulting score, MIPRO updates a surrogate model that helps the proposals get better over time. One thing that makes DSPy optimizers so powerful is that they can be composed. You can run `dspy.MIPROv2` and use the produced program as an input to `dspy.MIPROv2` again or, say, to `dspy.BootstrapFinetune` to get better results. This is partly the essence of `dspy.BetterTogether`. Alternatively, you can run the optimizer and then extract the top-5 candidate programs and build a `dspy.Ensemble` of them. This allows you to scale _inference-time compute_ (e.g., ensembles) as well as DSPy's unique _pre-inference time compute_ (i.e., optimization budget) in highly systematic ways. What DSPy Optimizers are currently available? --------------------------------------------- Optimizers can be accessed via `from dspy.teleprompt import *`. ### Automatic Few-Shot Learning These optimizers extend the signature by automatically generating and including **optimized** examples within the prompt sent to the model, implementing few-shot learning. 1. **`LabeledFewShot`**: Simply constructs few-shot examples (demos) from provided labeled input and output data points. Requires `k` (number of examples for the prompt) and `trainset` to randomly select `k` examples from. 2. **`BootstrapFewShot`**: Uses a `teacher` module (which defaults to your program) to generate complete demonstrations for every stage of your program, along with labeled examples in `trainset`. Parameters include `max_labeled_demos` (the number of demonstrations randomly selected from the `trainset`) and `max_bootstrapped_demos` (the number of additional examples generated by the `teacher`). The bootstrapping process employs the metric to validate demonstrations, including only those that pass the metric in the "compiled" prompt. Advanced: Supports using a `teacher` program that is a _different_ DSPy program that has compatible structure, for harder tasks. 3. [**`BootstrapFewShotWithRandomSearch`**](/deep-dive/optimizers/bootstrap-fewshot) : Applies `BootstrapFewShot` several times with random search over generated demonstrations, and selects the best program over the optimization. Parameters mirror those of `BootstrapFewShot`, with the addition of `num_candidate_programs`, which specifies the number of random programs evaluated over the optimization, including candidates of the uncompiled program, `LabeledFewShot` optimized program, `BootstrapFewShot` compiled program with unshuffled examples and `num_candidate_programs` of `BootstrapFewShot` compiled programs with randomized example sets. 4. **`KNNFewShot`**. Uses k-Nearest Neighbors algorithm to find the nearest training example demonstrations for a given input example. These nearest neighbor demonstrations are then used as the trainset for the BootstrapFewShot optimization process. See [this notebook](https://github.com/stanfordnlp/dspy/blob/main/examples/knn.ipynb) for an example. ### Automatic Instruction Optimization These optimizers produce optimal instructions for the prompt and, in the case of MIPROv2 can also optimize the set of few-shot demonstrations. 1. [**`COPRO`**](/deep-dive/optimizers/copro) : Generates and refines new instructions for each step, and optimizes them with coordinate ascent (hill-climbing using the metric function and the `trainset`). Parameters include `depth` which is the number of iterations of prompt improvement the optimizer runs over. 2. [**`MIPROv2`**](/deep-dive/optimizers/miprov2) : Generates instructions _and_ few-shot examples in each step. The instruction generation is data-aware and demonstration-aware. Uses Bayesian Optimization to effectively search over the space of generation instructions/demonstrations across your modules. ### Automatic Finetuning This optimizer is used to fine-tune the underlying LLM(s). 1. **`BootstrapFinetune`**: Distills a prompt-based DSPy program into weight updates. The output is a DSPy program that has the same steps, but where each step is conducted by a finetuned model instead of a prompted LM. ### Program Transformations 1. **`Ensemble`**: Ensembles a set of DSPy programs and either uses the full set or randomly samples a subset into a single program. Which optimizer should I use? ----------------------------- Ultimately, finding the ‘right’ optimizer to use & the best configuration for your task will require experimentation. Success in DSPy is still an iterative process - getting the best performance on your task will require you to explore and iterate. That being said, here's the general guidance on getting started: * If you have **very few examples** (around 10), start with `BootstrapFewShot`. * If you have **more data** (50 examples or more), try `BootstrapFewShotWithRandomSearch`. * If you prefer to do **instruction optimization only** (i.e. you want to keep your prompt 0-shot), use `MIPROv2` [configured for 0-shot optimization to optimize](/deep-dive/optimizers/miprov2#optimizing-instructions-only-with-miprov2-0-shot) . * If you’re willing to use more inference calls to perform **longer optimization runs** (e.g. 40 trials or more), and have enough data (e.g. 200 examples or more to prevent overfitting) then try `MIPROv2`. * If you have been able to use one of these with a large LM (e.g., 7B parameters or above) and need a very **efficient program**, finetune a small LM for your task with `BootstrapFinetune`. How do I use an optimizer? -------------------------- They all share this general interface, with some differences in the keyword arguments (hyperparameters). Detailed documentation for key optimizers can be found [here](/deep-dive/optimizers/vfrs) , and a full list can be found [here](/cheatsheet) . Let's see this with the most common one, `BootstrapFewShotWithRandomSearch`. `[](#__codelineno-0-1) from dspy.teleprompt import BootstrapFewShotWithRandomSearch [](#__codelineno-0-2) [](#__codelineno-0-3) # Set up the optimizer: we want to "bootstrap" (i.e., self-generate) 8-shot examples of your program's steps. [](#__codelineno-0-4) # The optimizer will repeat this 10 times (plus some initial attempts) before selecting its best attempt on the devset. [](#__codelineno-0-5) config = dict(max_bootstrapped_demos=4, max_labeled_demos=4, num_candidate_programs=10, num_threads=4) [](#__codelineno-0-6) [](#__codelineno-0-7) teleprompter = BootstrapFewShotWithRandomSearch(metric=YOUR_METRIC_HERE, **config) [](#__codelineno-0-8) optimized_program = teleprompter.compile(YOUR_PROGRAM_HERE, trainset=YOUR_TRAINSET_HERE)` Getting Started III: Optimizing the LM prompts or weights in DSPy programs A typical simple optimization run costs on the order of $2 USD and takes around ten minutes, but be careful when running optimizers with very large LMs or very large datasets. Optimizer runs can cost as little as a few cents or up to tens of dollars, depending on your LM, dataset, and configuration. Optimizing prompts for a ReAct agentOptimizing prompts for RAGOptimizing weights for Classification This is a minimal but fully runnable example of setting up a `dspy.ReAct` agent that answers questions via search from Wikipedia and then optimizing it using `dspy.MIPROv2` in the cheap `light` mode on 500 question-answer pairs sampled from the `HotPotQA` dataset. | | | | --- | --- | | [1](#__codelineno-1-1)
[2](#__codelineno-1-2)
[3](#__codelineno-1-3)
[4](#__codelineno-1-4)
[5](#__codelineno-1-5)
[6](#__codelineno-1-6)
[7](#__codelineno-1-7)
[8](#__codelineno-1-8)
[9](#__codelineno-1-9)
[10](#__codelineno-1-10)
[11](#__codelineno-1-11)
[12](#__codelineno-1-12)
[13](#__codelineno-1-13)
[14](#__codelineno-1-14)
[15](#__codelineno-1-15) | `import dspy from dspy.datasets import HotPotQA dspy.configure(lm=dspy.LM('openai/gpt-4o-mini')) def search(query: str) -> list[str]: """Retrieves abstracts from Wikipedia.""" results = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts')(query, k=3) return [x['text'] for x in results] trainset = [x.with_inputs('question') for x in HotPotQA(train_seed=2024, train_size=500).train] react = dspy.ReAct("question -> answer", tools=[search]) tp = dspy.MIPROv2(metric=dspy.evaluate.answer_exact_match, auto="light", num_threads=24) optimized_react = tp.compile(react, trainset=trainset)` | An informal run similar to this on DSPy 2.5.29 raises ReAct's score from 24% to 51%. Given a retrieval index to `search`, your favorite `dspy.LM`, and a small `trainset` of questions and ground-truth responses, the following code snippet can optimize your RAG system with long outputs against the built-in `dspy.SemanticF1` metric, which is implemented as a DSPy module. | | | | --- | --- | | [1](#__codelineno-2-1)
[2](#__codelineno-2-2)
[3](#__codelineno-2-3)
[4](#__codelineno-2-4)
[5](#__codelineno-2-5)
[6](#__codelineno-2-6)
[7](#__codelineno-2-7)
[8](#__codelineno-2-8)
[9](#__codelineno-2-9)
[10](#__codelineno-2-10)
[11](#__codelineno-2-11) | `class RAG(dspy.Module): def __init__(self, num_docs=5): self.num_docs = num_docs self.respond = dspy.ChainOfThought('context, question -> response') def forward(self, question): context = search(question, k=self.num_docs) # not defined in this snippet, see link above return self.respond(context=context, question=question) tp = dspy.MIPROv2(metric=dspy.SemanticF1(), auto="medium", num_threads=24) optimized_rag = tp.compile(RAG(), trainset=trainset, max_bootstrapped_demos=2, max_labeled_demos=2)` | For a complete RAG example that you can run, start this [tutorial](http://127.0.0.1:8000/quick-start/getting-started-01/) . It improves the quality of a RAG system over a subset of StackExchange communities from 53% to 61%. This is a minimal but fully runnable example of setting up a `dspy.ChainOfThought` module that classifies short texts into one of 77 banking labels and then using `dspy.BootstrapFinetune` with 2000 text-label pairs from the `Banking77` to finetune the weights of GPT-4o-mini for this task. We use the variant `dspy.ChainOfThoughtWithHint`, which takes an optional `hint` at bootstrapping time, to maximize the utility of the training data. Naturally, hints are not available at test time. Click to show dataset setup code. | | | | --- | --- | | [1](#__codelineno-3-1)
[2](#__codelineno-3-2)
[3](#__codelineno-3-3)
[4](#__codelineno-3-4)
[5](#__codelineno-3-5)
[6](#__codelineno-3-6)
[7](#__codelineno-3-7)
[8](#__codelineno-3-8)
[9](#__codelineno-3-9)
[10](#__codelineno-3-10)
[11](#__codelineno-3-11)
[12](#__codelineno-3-12)
[13](#__codelineno-3-13)
[14](#__codelineno-3-14)
[15](#__codelineno-3-15) | `import random from typing import Literal from dspy.datasets import DataLoader from datasets import load_dataset # Load the Banking77 dataset. CLASSES = load_dataset("PolyAI/banking77", split="train", trust_remote_code=True).features['label'].names kwargs = dict(fields=("text", "label"), input_keys=("text",), split="train", trust_remote_code=True) # Load the first 2000 examples from the dataset, and assign a hint to each *training* example. trainset = [ dspy.Example(x, hint=CLASSES[x.label], label=CLASSES[x.label]).with_inputs("text", "hint") for x in DataLoader().from_huggingface(dataset_name="PolyAI/banking77", **kwargs)[:2000] ] random.Random(0).shuffle(trainset)` | | | | | --- | --- | | [1](#__codelineno-4-1)
[2](#__codelineno-4-2)
[3](#__codelineno-4-3)
[4](#__codelineno-4-4)
[5](#__codelineno-4-5)
[6](#__codelineno-4-6)
[7](#__codelineno-4-7)
[8](#__codelineno-4-8)
[9](#__codelineno-4-9)
[10](#__codelineno-4-10)
[11](#__codelineno-4-11)
[12](#__codelineno-4-12) | `import dspy dspy.configure(lm=dspy.LM('gpt-4o-mini-2024-07-18')) # Define the DSPy module for classification. It will use the hint at training time, if available. signature = dspy.Signature("text -> label").with_updated_fields('label', type_=Literal[tuple(CLASSES)]) classify = dspy.ChainOfThoughtWithHint(signature) # Optimize via BootstrapFinetune. optimizer = dspy.BootstrapFinetune(metric=(lambda x, y, trace=None: x.label == y.label), num_threads=24) optimized = optimizer.compile(classify, trainset=trainset) optimized(text="What does a pending cash withdrawal mean?")` | **Possible Output (from the last line):** `[](#__codelineno-5-1) Prediction( [](#__codelineno-5-2) reasoning='A pending cash withdrawal indicates that a request to withdraw cash has been initiated but has not yet been completed or processed. This status means that the transaction is still in progress and the funds have not yet been deducted from the account or made available to the user.', [](#__codelineno-5-3) label='pending_cash_withdrawal' [](#__codelineno-5-4) )` An informal run similar to this on DSPy 2.5.29 raises GPT-4o-mini's score 66% to 87%. Saving and loading optimizer output ----------------------------------- After running a program through an optimizer, it's useful to also save it. At a later point, a program can be loaded from a file and used for inference. For this, the `load` and `save` methods can be used. `[](#__codelineno-6-1) optimized_program.save(YOUR_SAVE_PATH)` The resulting file is in plain-text JSON format. It contains all the parameters and steps in the source program. You can always read it and see what the optimizer generated. You can add `save_field_meta` to additionally save the list of fields with the keys, `name`, `field_type`, `description`, and `prefix` with: \`optimized\_program.save(YOUR\_SAVE\_PATH, save\_field\_meta=True). To load a program from a file, you can instantiate an object from that class and then call the load method on it. `[](#__codelineno-7-1) loaded_program = YOUR_PROGRAM_CLASS() [](#__codelineno-7-2) loaded_program.load(path=YOUR_SAVE_PATH)` Back to top --- # ChromadbRM - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/deep-dive/retrieval_models_clients/ChromadbRM.md "Edit this page") ChromadbRM ========== Note Adapted from documentation provided by https://github.com/animtel ChromadbRM have the flexibility from a variety of embedding functions as outlined in the [chromadb embeddings documentation](https://docs.trychroma.com/embeddings) . While different options are available, this example demonstrates how to utilize OpenAI embeddings specifically. Setting up the ChromadbRM Client -------------------------------- The constructor initializes an instance of the `ChromadbRM` class, with the option to use OpenAI's embeddings or any alternative supported by chromadb, as detailed in the official [chromadb embeddings documentation](https://docs.trychroma.com/guides/embeddings) . * `collection_name` (_str_): The name of the chromadb collection. * `persist_directory` (_str_): Path to the directory where chromadb data is persisted. * `embedding_function` (_Optional\[EmbeddingFunction\[Embeddable\]\]_, _optional_): The function used for embedding documents and queries. Defaults to `DefaultEmbeddingFunction()` if not specified. * `k` (_int_, _optional_): The number of top passages to retrieve. Defaults to 7. Example of the ChromadbRM constructor: `[](#__codelineno-0-1) ChromadbRM( [](#__codelineno-0-2) collection_name: str, [](#__codelineno-0-3) persist_directory: str, [](#__codelineno-0-4) embedding_function: Optional[EmbeddingFunction[Embeddable]] = OpenAIEmbeddingFunction(), [](#__codelineno-0-5) k: int = 7, [](#__codelineno-0-6) )` Under the Hood -------------- ### `forward(self, query_or_queries: Union[str, List[str]], k: Optional[int] = None) -> dspy.Prediction` **Parameters:** - `query_or_queries` (_Union\[str, List\[str\]\]_): The query or list of queries to search for. - `k` (_Optional\[int\]_, _optional_): The number of results to retrieve. If not specified, defaults to the value set during initialization. **Returns:** - `dspy.Prediction`: Contains the retrieved passages, each represented as a `dotdict` with a `long_text` attribute. Search the chromadb collection for the top `k` passages matching the given query or queries, using embeddings generated via the specified `embedding_function`. Sending Retrieval Requests via ChromadbRM Client ------------------------------------------------ `[](#__codelineno-1-1) from dspy.retrieve.chromadb_rm import ChromadbRM [](#__codelineno-1-2) import os [](#__codelineno-1-3) import openai [](#__codelineno-1-4) from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction [](#__codelineno-1-5) [](#__codelineno-1-6) embedding_function = OpenAIEmbeddingFunction( [](#__codelineno-1-7) api_key=os.environ.get('OPENAI_API_KEY'), [](#__codelineno-1-8) model_name="text-embedding-ada-002" [](#__codelineno-1-9) ) [](#__codelineno-1-10) [](#__codelineno-1-11) retriever_model = ChromadbRM( [](#__codelineno-1-12) 'your_collection_name', [](#__codelineno-1-13) '/path/to/your/db', [](#__codelineno-1-14) embedding_function=embedding_function, [](#__codelineno-1-15) k=5 [](#__codelineno-1-16) ) [](#__codelineno-1-17) [](#__codelineno-1-18) results = retriever_model("Explore the significance of quantum computing", k=5) [](#__codelineno-1-19) [](#__codelineno-1-20) for result in results: [](#__codelineno-1-21) print("Document:", result.long_text, "\n")` Back to top --- # DatabricksRM - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/deep-dive/retrieval_models_clients/DatabricksRM.md "Edit this page") DatabricksRM ============ ### Constructor Initialize an instance of the `DatabricksRM` retriever class, which enables DSPy programs to query [Databricks Mosaic AI Vector Search](https://docs.databricks.com/en/generative-ai/vector-search.html#mosaic-ai-vector-search) indexes for document retrieval. `[](#__codelineno-0-1) DatabricksRM( [](#__codelineno-0-2) databricks_index_name: str, [](#__codelineno-0-3) databricks_endpoint: Optional[str] = None, [](#__codelineno-0-4) databricks_token: Optional[str] = None, [](#__codelineno-0-5) columns: Optional[List[str]] = None, [](#__codelineno-0-6) filters_json: Optional[str] = None, [](#__codelineno-0-7) k: int = 3, [](#__codelineno-0-8) docs_id_column_name: str = "id", [](#__codelineno-0-9) text_column_name: str = "text", [](#__codelineno-0-10) )` **Parameters:** * `databricks_index_name (str)`: The name of the Databricks Vector Search Index to query. * `databricks_endpoint (Optional[str])`: The URL of the Databricks Workspace containing the Vector Search Index. Defaults to the value of the `DATABRICKS_HOST` environment variable. If unspecified, the Databricks SDK is used to identify the endpoint based on the current environment. * `databricks_token (Optional[str])`: The Databricks Workspace authentication token to use when querying the Vector Search Index. Defaults to the value of the `DATABRICKS_TOKEN` environment variable. If unspecified, the Databricks SDK is used to identify the token based on the current environment. * `columns (Optional[List[str]])`: Extra column names to include in response, in addition to the document id and text columns specified by `docs_id_column_name` and `text_column_name`. * `filters_json (Optional[str])`: A JSON string specifying additional query filters. Example filters: `{"id <": 5}` selects records that have an `id` column value less than 5, and `{"id >=": 5, "id <": 10}` selects records that have an `id` column value greater than or equal to 5 and less than 10. * `k (int)`: The number of documents to retrieve. * `docs_id_column_name (str)`: The name of the column in the Databricks Vector Search Index containing document IDs. * `text_column_name (str)`: The name of the column in the Databricks Vector Search Index containing document text to retrieve. ### Methods #### `def forward(self, query: Union[str, List[float]], query_type: str = "ANN", filters_json: Optional[str] = None) -> dspy.Prediction:` Retrieve documents from a Databricks Mosaic AI Vector Search Index that are relevant to the specified query. **Parameters:** * `query (Union[str, List[float]])`: The query text or numeric query vector for which to retrieve relevant documents. * `query_type (str)`: The type of search query to perform against the Databricks Vector Search Index. Must be either 'ANN' (approximate nearest neighbor) or 'HYBRID' (hybrid search). * `filters_json (Optional[str])`: A JSON string specifying additional query filters. Example filters: `{"id <": 5}` selects records that have an `id` column value less than 5, and `{"id >=": 5, "id <": 10}` selects records that have an `id` column value greater than or equal to 5 and less than 10. If specified, this parameter overrides the `filters_json` parameter passed to the constructor. **Returns:** * `dspy.Prediction`: A `dotdict` containing retrieved documents. The schema is `{'docs': List[str], 'doc_ids': List[Any], extra_columns: List[Dict[str, Any]]}`. The `docs` entry contains the retrieved document content. ### Quickstart To retrieve documents using Databricks Mosaic AI Vector Search, you must [create a Databricks Mosaic AI Vector Search Index](https://docs.databricks.com/en/generative-ai/create-query-vector-search.html) first. The following example code demonstrates how to set up a Databricks Mosaic AI [Direct Access Vector Search Index](https://docs.databricks.com/en/generative-ai/create-query-vector-search.html#create-a-vector-search-index) and use the `DatabricksRM` DSPy retriever module to query the index. The example requires the `databricks-vectorsearch` Python library to be installed. `[](#__codelineno-1-1) from databricks.vector_search.client import VectorSearchClient [](#__codelineno-1-2) [](#__codelineno-1-3) # Create a Databricks Vector Search Endpoint [](#__codelineno-1-4) client = VectorSearchClient() [](#__codelineno-1-5) client.create_endpoint( [](#__codelineno-1-6) name="your_vector_search_endpoint_name", [](#__codelineno-1-7) endpoint_type="STANDARD" [](#__codelineno-1-8) ) [](#__codelineno-1-9) [](#__codelineno-1-10) # Create a Databricks Direct Access Vector Search Index [](#__codelineno-1-11) index = client.create_direct_access_index( [](#__codelineno-1-12) endpoint_name="your_vector_search_endpoint_name", [](#__codelineno-1-13) index_name="your_index_name", [](#__codelineno-1-14) primary_key="id", [](#__codelineno-1-15) embedding_dimension=1024, [](#__codelineno-1-16) embedding_vector_column="text_vector", [](#__codelineno-1-17) schema={ [](#__codelineno-1-18) "id": "int", [](#__codelineno-1-19) "field2": "str", [](#__codelineno-1-20) "field3": "float", [](#__codelineno-1-21) "text_vector": "array" [](#__codelineno-1-22) } [](#__codelineno-1-23) ) [](#__codelineno-1-24) [](#__codelineno-1-25) # Create a DatabricksRM retriever and retrieve the top-3 most relevant documents from the [](#__codelineno-1-26) # Databricks Direct Access Vector Search Index corresponding to an example query [](#__codelineno-1-27) retriever = DatabricksRM( [](#__codelineno-1-28) databricks_index_name = "your_index_name", [](#__codelineno-1-29) docs_id_column_name="id", [](#__codelineno-1-30) text_column_name="field2", [](#__codelineno-1-31) k=3 [](#__codelineno-1-32) ) [](#__codelineno-1-33) retrieved_results = DatabricksRM(query="Example query text", query_type="hybrid"))` Back to top --- # Azure - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/deep-dive/retrieval_models_clients/Azure.md "Edit this page") import AuthorDetails from '@site/src/components/AuthorDetails'; AzureAISearch ============= A retrieval module that utilizes Azure AI Search to retrieve top passages for a given query. Prerequisites ------------- `[](#__codelineno-0-1) pip install azure-search-documents` Setting up the AzureAISearchRM Client ------------------------------------- The constructor initializes an instance of the `AzureAISearchRM` class and sets up parameters for sending queries and retrieving results with the Azure AI Search server. * `search_service_name` (str): The name of the Azure AI Search service. * `search_api_key` (str): The API key for accessing the Azure AI Search service. * `search_index_name` (str): The name of the search index in the Azure AI Search service. * `field_text` (str): The name of the field containing text content in the search index. This field will be mapped to the "content" field in the dsp framework. * `field_vector` (Optional\[str\]): The name of the field containing vector content in the search index. * `k` (int, optional): The default number of top passages to retrieve. Defaults to 3. * `azure_openai_client` (Optional\[openai.AzureOpenAI\]): An instance of the AzureOpenAI client. Either openai\_client or embedding\_func must be provided. Defaults to None. * `openai_embed_model` (Optional\[str\]): The name of the OpenAI embedding model. Defaults to "text-embedding-ada-002". * `embedding_func` (Optional\[Callable\]): A function for generating embeddings. Either openai\_client or embedding\_func must be provided. Defaults to None. * `semantic_ranker` (bool, optional): Whether to use semantic ranking. Defaults to False. * `filter` (str, optional): Additional filter query. Defaults to None. * `query_language` (str, optional): The language of the query. Defaults to "en-Us". * `query_speller` (str, optional): The speller mode. Defaults to "lexicon". * `use_semantic_captions` (bool, optional): Whether to use semantic captions. Defaults to False. * `query_type` (Optional\[QueryType\], optional): The type of query. Defaults to QueryType.FULL. * `semantic_configuration_name` (str, optional): The name of the semantic configuration. Defaults to None. * `is_vector_search` (Optional\[bool\]): Whether to enable vector search. Defaults to False. * `is_hybrid_search` (Optional\[bool\]): Whether to enable hybrid search. Defaults to False. * `is_fulltext_search` (Optional\[bool\]): Whether to enable fulltext search. Defaults to True. * `vector_filter_mode` (Optional\[VectorFilterMode\]): The vector filter mode. Defaults to None. **Available Query Types:** * SIMPLE """Uses the simple query syntax for searches. Search text is interpreted using a simple query #: language that allows for symbols such as +, \* and "". Queries are evaluated across all #: searchable fields by default, unless the searchFields parameter is specified.""" * FULL """Uses the full Lucene query syntax for searches. Search text is interpreted using the Lucene #: query language which allows field-specific and weighted searches, as well as other advanced #: features.""" * SEMANTIC """Best suited for queries expressed in natural language as opposed to keywords. Improves #: precision of search results by re-ranking the top search results using a ranking model trained #: on the Web corpus."" More Details: https://learn.microsoft.com/en-us/azure/search/search-query-overview **Available Vector Filter Mode:** * POST\_FILTER = "postFilter" """The filter will be applied after the candidate set of vector results is returned. Depending on #: the filter selectivity, this can result in fewer results than requested by the parameter 'k'.""" * PRE\_FILTER = "preFilter" """The filter will be applied before the search query.""" More Details: https://learn.microsoft.com/en-us/azure/search/vector-search-filters **Note** * The `AzureAISearchRM` client allows you to perform Vector search, Hybrid search, or Full text search. * By default, the `AzureAISearchRM` client uses the Azure OpenAI Client for generating embeddings. If you want to use something else, you can provide your custom embedding\_func, but either the openai\_client or embedding\_func must be provided. * If you need to enable semantic search, either with vector, hybrid, or full text search, then set the `semantic_ranker` flag to True. * If `semantic_ranker` is True, always set the `query_type` to QueryType.SEMANTIC and always provide the `semantic_configuration_name`. Example of the AzureAISearchRM constructor: `[](#__codelineno-1-1) AzureAISearchRM( [](#__codelineno-1-2) search_service_name: str, [](#__codelineno-1-3) search_api_key: str, [](#__codelineno-1-4) search_index_name: str, [](#__codelineno-1-5) field_text: str, [](#__codelineno-1-6) field_vector: Optional[str] = None, [](#__codelineno-1-7) k: int = 3, [](#__codelineno-1-8) azure_openai_client: Optional[openai.AzureOpenAI] = None, [](#__codelineno-1-9) openai_embed_model: Optional[str] = "text-embedding-ada-002", [](#__codelineno-1-10) embedding_func: Optional[Callable] = None, [](#__codelineno-1-11) semantic_ranker: bool = False, [](#__codelineno-1-12) filter: str = None, [](#__codelineno-1-13) query_language: str = "en-Us", [](#__codelineno-1-14) query_speller: str = "lexicon", [](#__codelineno-1-15) use_semantic_captions: bool = False, [](#__codelineno-1-16) query_type: Optional[QueryType] = QueryType.FULL, [](#__codelineno-1-17) semantic_configuration_name: str = None, [](#__codelineno-1-18) is_vector_search: Optional[bool] = False, [](#__codelineno-1-19) is_hybrid_search: Optional[bool] = False, [](#__codelineno-1-20) is_fulltext_search: Optional[bool] = True, [](#__codelineno-1-21) vector_filter_mode: Optional[VectorFilterMode.PRE_FILTER] = None [](#__codelineno-1-22) )` Under the Hood -------------- ### `forward(self, query_or_queries: Union[str, List[str]], k: Optional[int] = None) -> dspy.Prediction` **Parameters:** * `query_or_queries` (Union\[str, List\[str\]\]): The query or queries to search for. * `k` (_Optional\[int\]_, _optional_): The number of results to retrieve. If not specified, defaults to the value set during initialization. **Returns:** * `dspy.Prediction`: Contains the retrieved passages, each represented as a `dotdict` with a `long_text` attribute. Internally, the method handles the specifics of preparing the request query to the Azure AI Search service and corresponding payload to obtain the response. The function handles the retrieval of the top-k passages based on the provided query. Sending Retrieval Requests via AzureAISearchRM Client ----------------------------------------------------- 1. _**Recommended**_ Configure default RM using `dspy.configure`. This allows you to define programs in DSPy and have DSPy internally conduct retrieval using `dsp.retrieve` on the query on the configured RM. `[](#__codelineno-2-1) import dspy [](#__codelineno-2-2) from dspy.retrieve.azureaisearch_rm import AzureAISearchRM [](#__codelineno-2-3) [](#__codelineno-2-4) azure_search = AzureAISearchRM( [](#__codelineno-2-5) "search_service_name", [](#__codelineno-2-6) "search_api_key", [](#__codelineno-2-7) "search_index_name", [](#__codelineno-2-8) "field_text", [](#__codelineno-2-9) "k"=3 [](#__codelineno-2-10) ) [](#__codelineno-2-11) [](#__codelineno-2-12) dspy.settings.configure(rm=azure_search) [](#__codelineno-2-13) retrieve = dspy.Retrieve(k=3) [](#__codelineno-2-14) retrieval_response = retrieve("What is Thermodynamics").passages [](#__codelineno-2-15) [](#__codelineno-2-16) for result in retrieval_response: [](#__codelineno-2-17) print("Text:", result, "\n")` 1. Generate responses using the client directly. `[](#__codelineno-3-1) from dspy.retrieve.azureaisearch_rm import AzureAISearchRM [](#__codelineno-3-2) [](#__codelineno-3-3) azure_search = AzureAISearchRM( [](#__codelineno-3-4) "search_service_name", [](#__codelineno-3-5) "search_api_key", [](#__codelineno-3-6) "search_index_name", [](#__codelineno-3-7) "field_text", [](#__codelineno-3-8) "k"=3 [](#__codelineno-3-9) ) [](#__codelineno-3-10) [](#__codelineno-3-11) retrieval_response = azure_search("What is Thermodynamics", k=3) [](#__codelineno-3-12) for result in retrieval_response: [](#__codelineno-3-13) print("Text:", result.long_text, "\n")` 1. Example of Semantic Hybrid Search. `[](#__codelineno-4-1) from dspy.retrieve.azureaisearch_rm import AzureAISearchRM [](#__codelineno-4-2) [](#__codelineno-4-3) azure_search = AzureAISearchRM( [](#__codelineno-4-4) search_service_name="search_service_name", [](#__codelineno-4-5) search_api_key="search_api_key", [](#__codelineno-4-6) search_index_name="search_index_name", [](#__codelineno-4-7) field_text="field_text", [](#__codelineno-4-8) field_vector="field_vector", [](#__codelineno-4-9) k=3, [](#__codelineno-4-10) azure_openai_client="azure_openai_client", [](#__codelineno-4-11) openai_embed_model="text-embedding-ada-002" [](#__codelineno-4-12) semantic_ranker=True, [](#__codelineno-4-13) query_type=QueryType.SEMANTIC, [](#__codelineno-4-14) semantic_configuration_name="semantic_configuration_name", [](#__codelineno-4-15) is_hybrid_search=True, [](#__codelineno-4-16) ) [](#__codelineno-4-17) [](#__codelineno-4-18) retrieval_response = azure_search("What is Thermodynamics", k=3) [](#__codelineno-4-19) for result in retrieval_response: [](#__codelineno-4-20) print("Text:", result.long_text, "\n")` * * * Back to top --- # FalkordbRM - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/deep-dive/retrieval_models_clients/FalkordbRM.md "Edit this page") FalkordbRM ========== ### Constructor Initialize an instance of the `FalkordbRM` class. `[](#__codelineno-0-1) FalkordbRM( [](#__codelineno-0-2) node_label: str, [](#__codelineno-0-3) text_node_property: str, [](#__codelineno-0-4) embedding_node_property: str, [](#__codelineno-0-5) k: int = 5, [](#__codelineno-0-6) retrieval_query: str, [](#__codelineno-0-7) embedding_provider: str = "openai", [](#__codelineno-0-8) embedding_model: str = "text-embedding-ada-002", [](#__codelineno-0-9) )` **Environment Variables:** You need to define the credentials as environment variables * `FALKORDB_HOST` (_str_): Specifies the host required for connecting with the Falkordb database. If not provided, the system will default to `localhost` * `FALKORDB_PORT` (_int_): Specifies the port required for connecting with the Falkordb database. If not provided, the system will default to `6379` * `FALKORDB_USERNAME` (_str_, optional): Specifies the username required for authenticating with a [Falkordb Cloud](https://app.falkordb.cloud/signin) database. * `FALKORDB_PASSWORD` (_str_, optional): Specifies the password required for authenticating with a [Falkordb Cloud](https://app.falkordb.cloud/signin) database. * `FALKORDB_DATABASE` (_str_, optional): Specifies the name of the database to connect to within the Falkordb instance. If not provided, the systems defaults to using a randomly generated four ascii\_letters character string e.g "tari". * `OPENAI_API_KEY` (_str_): Specifies the API key required for authenticating with OpenAI's services. **Parameters:** * `node_label` (_str_): Specifies the label of the node to be used within Falkordb for organizing and querying data. * `text_node_property` (_str_, _optional_): Defines the specific text property of the node that will be returned. * `embedding_node_property` (_str_): Defines the specific embedding property of the node that will be used within Falkordb for querying data. * `k` (_int_, _optional_): The number of top results to return from the retrieval operation. It defaults to 5 if not explicitly specified. * `retrieval_query` (_str_, _optional_): A custom query string provided for retrieving data. If not provided, a default query tailored to the `text_node_property` will be used. * `embedding_provider` (_str_, _optional_): The name of the service provider for generating embeddings. Only "openai" is supported. * `embedding_model` (_str_, _optional_): The specific embedding model to use from the provider. By default, it uses the "text-embedding-ada-002" model from OpenAI. ### Methods #### `forward(self, query: [str], k: Optional[int] = None) -> dspy.Prediction` Search the Falkordb vector index for the top `k` passages matching the given query or queries, using embeddings generated via the specified `embedding_model`. **Parameters:** * `query` (str\_): The query. * `k` (_Optional\[int\]_, _optional_): The number of results to retrieve. If not specified, defaults to the value set during initialization. **Returns:** * `dspy.Prediction`: Contains the retrieved passages as a list of string with the prediction signature. ex: `[](#__codelineno-1-1) Prediction( [](#__codelineno-1-2) passages=['Passage 1 Lorem Ipsum awesom', 'Passage 2 Lorem Ipsum Youppidoo', 'Passage 3 Lorem Ipsum Yassssss'] [](#__codelineno-1-3) )` ### Quick Example how to use Falkordb in a local environment. `[](#__codelineno-2-1) from dspy.retrieve.falkordb_rm import FalkordbRM [](#__codelineno-2-2) import os [](#__codelineno-2-3) [](#__codelineno-2-4) [](#__codelineno-2-5) os.environ["FALKORDB_HOST"] = 'localhost' [](#__codelineno-2-6) os.environ["FALKORDB_PORT"] = 6379 [](#__codelineno-2-7) os.environ["OPENAI_API_KEY"] = 'sk-' [](#__codelineno-2-8) [](#__codelineno-2-9) retriever_model = FalkordbRM( [](#__codelineno-2-10) node_label="myIndex", [](#__codelineno-2-11) text_node_property="text", [](#__codelineno-2-12) embedding_node_property="embedding" [](#__codelineno-2-13) ) [](#__codelineno-2-14) [](#__codelineno-2-15) results = retriever_model("Explore the significance of quantum computing", k=3) [](#__codelineno-2-16) [](#__codelineno-2-17) for passage in results: [](#__codelineno-2-18) print("Document:", passage, "\n")` Back to top --- # ClarifaiRM - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/deep-dive/retrieval_models_clients/ClarifaiRM.md "Edit this page") ClarifaiRM ========== [Clarifai](https://clarifai.com/) is a powerful AI platform that provides vector search capabilities through its search API. DSPy has integrated ClarifaiRM to support efficient text search and retrieval through its specialized indexing and ability to handle large-scale document collections. To support passage retrieval, ClarifaiRM assumes that documents have been properly ingested into a Clarifai application with the following: - Text data properly indexed and stored - Appropriate search configurations set up in the Clarifai platform - Valid authentication credentials (PAT key) with appropriate permissions The ClarifaiRM module requires the `clarifai` Python package. If not already installed, you can install it using: `[](#__codelineno-0-1) pip install clarifai` **Note:** Before using ClarifaiRM, ensure you have: 1. Created a Clarifai account and application 2. Ingested your documents into the application 3. Obtained your User ID, App ID, and Personal Access Token (PAT) Setting up the ClarifaiRM Client -------------------------------- The constructor initializes an instance of the `ClarifaiRM` class, which requires authentication credentials and configuration to connect to your Clarifai application. * `clarifai_user_id` (_str_): Your unique Clarifai user identifier. * `clarifai_app_id` (_str_): The ID of your Clarifai application where documents are stored. * `clarifai_pat` (_Optional\[str\]_): Your Clarifai Personal Access Token (PAT). It will look for `CLARIFAI_PAT` in environment variables if not provided. * `k` (_int_, _optional_): The number of top passages to retrieve. Defaults to 3. Example of the ClarifaiRM constructor: `[](#__codelineno-1-1) ClarifaiRM( [](#__codelineno-1-2) clarifai_user_id: str, [](#__codelineno-1-3) clarifai_app_id: str, [](#__codelineno-1-4) clarifai_pat: Optional[str] = None, [](#__codelineno-1-5) k: int = 3, [](#__codelineno-1-6) )` **Note:** The PAT can be provided either directly to the constructor or through the `CLARIFAI_PAT` environment variable. For security best practices, using environment variables is recommended. Under the Hood -------------- ### `retrieve_hits(self, hits)` **Parameters:** - `hits` (_ClarifaiHit_): A hit object from Clarifai's search response. **Returns:** - `str`: The retrieved text content. Internal method that retrieves text content from the hit's URL using authenticated requests. ### `forward(self, query_or_queries: Union[str, List[str]], k: Optional[int] = None, **kwargs) -> dspy.Prediction` **Parameters:** - `query_or_queries` (_Union\[str, List\[str\]\]_): The query or list of queries to search for. - `k` (_Optional\[int\]_, _optional_): The number of results to retrieve. If not specified, defaults to the value set during initialization. - `**kwargs`: Additional keyword arguments passed to Clarifai's search function. **Returns:** - `dspy.Prediction`: Contains the retrieved passages, each represented as a `dotdict` with a `long_text` attribute. Search the Clarifai application for the top `k` passages matching the given query or queries. Uses parallel processing with ThreadPoolExecutor to efficiently retrieve multiple results. Examples -------- ### Basic Usage `[](#__codelineno-2-1) import os [](#__codelineno-2-2) from dspy.retrieve.clarifai_rm import ClarifaiRM [](#__codelineno-2-3) import dspy [](#__codelineno-2-4) [](#__codelineno-2-5) os.environ["CLARIFAI_PAT"] = "your_pat_key" [](#__codelineno-2-6) [](#__codelineno-2-7) retriever_model = ClarifaiRM( [](#__codelineno-2-8) clarifai_user_id="your_user_id", [](#__codelineno-2-9) clarifai_app_id="your_app_id", [](#__codelineno-2-10) k=5 [](#__codelineno-2-11) ) [](#__codelineno-2-12) [](#__codelineno-2-13) turbo = dspy.OpenAI(model="gpt-3.5-turbo") [](#__codelineno-2-14) dspy.settings.configure(lm=turbo, rm=retriever_model) [](#__codelineno-2-15) [](#__codelineno-2-16) results = retriever_model("Explore the significance of quantum computing")` ### Multiple Queries `[](#__codelineno-3-1) queries = [ [](#__codelineno-3-2) "What is machine learning?", [](#__codelineno-3-3) "How does deep learning work?", [](#__codelineno-3-4) "Explain neural networks" [](#__codelineno-3-5) ] [](#__codelineno-3-6) [](#__codelineno-3-7) results = retriever_model(queries, k=3)` ### Using with DSPy Retrieve Module `[](#__codelineno-4-1) from dspy import Retrieve [](#__codelineno-4-2) [](#__codelineno-4-3) retrieve = Retrieve(k=5) [](#__codelineno-4-4) [](#__codelineno-4-5) class RAG(dspy.Module): [](#__codelineno-4-6) def __init__(self): [](#__codelineno-4-7) super().__init__() [](#__codelineno-4-8) self.retrieve = Retrieve(k=3) [](#__codelineno-4-9) [](#__codelineno-4-10) def forward(self, query): [](#__codelineno-4-11) passages = self.retrieve(query) [](#__codelineno-4-12) return passages [](#__codelineno-4-13) [](#__codelineno-4-14) rag = RAG() [](#__codelineno-4-15) result = rag("What are the latest developments in AI?")` ### Handling Results `[](#__codelineno-5-1) results = retriever_model("quantum computing advances", k=5) [](#__codelineno-5-2) [](#__codelineno-5-3) for i, result in enumerate(results, 1): [](#__codelineno-5-4) print(f"Result {i}:") [](#__codelineno-5-5) print(result.long_text) [](#__codelineno-5-6) print("-" * 50) [](#__codelineno-5-7) [](#__codelineno-5-8) first_passage = results[0].long_text [](#__codelineno-5-9) [](#__codelineno-5-10) num_results = len(results)` ### Integration with Other DSPy Components `[](#__codelineno-6-1) from dspy import ChainOfThought, Predict, Retrieve [](#__codelineno-6-2) [](#__codelineno-6-3) # Create a simple QA chain [](#__codelineno-6-4) class QAChain(dspy.Module): [](#__codelineno-6-5) def __init__(self): [](#__codelineno-6-6) super().__init__() [](#__codelineno-6-7) self.retrieve = Retrieve(k=3) [](#__codelineno-6-8) self.generate_answer = ChainOfThought("question, context -> answer") [](#__codelineno-6-9) [](#__codelineno-6-10) def forward(self, question): [](#__codelineno-6-11) context = self.retrieve(question) [](#__codelineno-6-12) answer = self.generate_answer(question=question, context=context) [](#__codelineno-6-13) return answer [](#__codelineno-6-14) [](#__codelineno-6-15) qa = QAChain() [](#__codelineno-6-16) answer = qa("What are the main applications of quantum computing?")` ### Error Handling Example `[](#__codelineno-7-1) try: [](#__codelineno-7-2) retriever_model = ClarifaiRM( [](#__codelineno-7-3) clarifai_user_id="your_user_id", [](#__codelineno-7-4) clarifai_app_id="your_app_id", [](#__codelineno-7-5) clarifai_pat="invalid_pat" [](#__codelineno-7-6) ) [](#__codelineno-7-7) results = retriever_model("test query") [](#__codelineno-7-8) except Exception as e: [](#__codelineno-7-9) print(f"Error occurred: {e}")` **Note:** These examples assume you have: * A properly configured Clarifai application * Valid authentication credentials * Documents already ingested into your Clarifai app * The necessary environment variables set up Back to top --- # ColBERTv2 - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/deep-dive/retrieval_models_clients/ColBERTv2.md "Edit this page") import AuthorDetails from '@site/src/components/AuthorDetails'; ColBERTv2 ========= Setting up the ColBERTv2 Client ------------------------------- The constructor initializes the `ColBERTv2` class instance and sets up the request parameters for interacting with the ColBERTv2 retrieval server. This server is hosted remotely at `'http://20.102.90.50:2017/wiki17_abstracts`. * `url` (_str_): URL for ColBERTv2 server. Defaults to `"http://0.0.0.0"` * `port` (_Union\[str, int\]_, _Optional_): Port endpoint for ColBERTv2 server. Defaults to `None`. * `post_requests` (_bool_, _Optional_): Flag for using HTTP POST requests. Defaults to `False`. Example of the ColBERTv2 constructor: `[](#__codelineno-0-1) class ColBERTv2: [](#__codelineno-0-2) def __init__( [](#__codelineno-0-3) self, [](#__codelineno-0-4) url: str = "http://0.0.0.0", [](#__codelineno-0-5) port: Optional[Union[str, int]] = None, [](#__codelineno-0-6) post_requests: bool = False, [](#__codelineno-0-7) ):` Under the Hood -------------- ### `__call__(self, query: str, k: int = 10, simplify: bool = False) -> Union[list[str], list[dotdict]]` **Parameters:** - `query` (_str_): Search query string used for retrieval sent to ColBERTv2 server. - `k` (_int_, _optional_): Number of passages to retrieve. Defaults to 10. - `simplify` (_bool_, _optional_): Flag for simplifying output to a list of strings. Defaults to False. **Returns:** - `Union[list[str], list[dotdict]]`: Depending on `simplify` flag, either a list of strings representing the passage content (`True`) or a list of `dotdict` instances containing passage details (`False`). Internally, the method handles the specifics of preparing the request query to the ColBERTv2 server and corresponding payload to obtain the response. The function handles the retrieval of the top-k passages based on the provided query. If `post_requests` is set, the method sends a query to the server via a POST request else via a GET request. It then processes and returns the top-k passages from the response with the list of retrieved passages dependent on the `simplify` flag return condition above. Sending Retrieval Requests via ColBERTv2 Client ----------------------------------------------- 1) _**Recommended**_ Configure default RM using `dspy.configure`. This allows you to define programs in DSPy and have DSPy internally conduct retrieval using `dsp.retrieve` on the query on the configured RM. `[](#__codelineno-1-1) import dspy [](#__codelineno-1-2) import dsp [](#__codelineno-1-3) [](#__codelineno-1-4) dspy.settings.configure(rm=colbertv2_wiki17_abstracts) [](#__codelineno-1-5) retrieval_response = dsp.retrieve("When was the first FIFA World Cup held?", k=5) [](#__codelineno-1-6) [](#__codelineno-1-7) for result in retrieval_response: [](#__codelineno-1-8) print("Text:", result, "\n")` 2) Generate responses using the client directly. `[](#__codelineno-2-1) import dspy [](#__codelineno-2-2) [](#__codelineno-2-3) retrieval_response = colbertv2_wiki17_abstracts('When was the first FIFA World Cup held?', k=5) [](#__codelineno-2-4) [](#__codelineno-2-5) for result in retrieval_response: [](#__codelineno-2-6) print("Text:", result['text'], "\n")` * * * Back to top --- # FaissRM - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/deep-dive/retrieval_models_clients/FaissRM.md "Edit this page") FaissRM ======= ### Constructor Initialize an instance of FaissRM by providing it with a vectorizer and a list of strings `[](#__codelineno-0-1) FaissRM( [](#__codelineno-0-2) document_chunks: List[str], [](#__codelineno-0-3) vectorizer: dsp.modules.sentence_vectorizer.BaseSentenceVectorizer, [](#__codelineno-0-4) k: int = 3 [](#__codelineno-0-5) )` **Parameters:** * `document_chunks` (_List\[str\]_): a list of strings that comprises the corpus to search. You cannot add/insert/upsert to this list after creating this FaissRM object. * `vectorizer` (_dsp.modules.sentence\_vectorizer.BaseSentenceVectorizer_, _optional_): If not provided, a dsp.modules.sentence\_vectorizer.SentenceTransformersVectorizer object is created and used. * `k` (_int_, _optional_): The number of top passages to retrieve. Defaults to 3. ### Methods #### `forward(self, query_or_queries: Union[str, List[str]]) -> dspy.Prediction` Search the FaissRM vector database for the top `k` passages matching the given query or queries, using embeddings generated via the vectorizer specified at FaissRM construction time **Parameters:** * `query_or_queries` (_Union\[str, List\[str\]\]_): The query or list of queries to search for. **Returns:** * `dspy.Prediction`: Contains the retrieved passages, each represented as a `dotdict` with a `long_text` attribute and an `index` attribute. The `index` attribute is the index in the document\_chunks array provided to this FaissRM object at construction time. ### Quickstart with the default vectorizer The **FaissRM** module provides a retriever that uses an in-memory Faiss vector database. This module does not include a vectorizer; instead it supports any subclass of **dsp.modules.sentence\_vectorizer.BaseSentenceVectorizer**. If a vectorizer is not provided, an instance of **dsp.modules.sentence\_vectorizer.SentenceTransformersVectorizer** is created and used by **FaissRM**. Note that the default embedding model for **SentenceTransformersVectorizer** is **all-MiniLM-L6-v2** `[](#__codelineno-1-1) import dspy [](#__codelineno-1-2) from dspy.retrieve.faiss_rm import FaissRM [](#__codelineno-1-3) [](#__codelineno-1-4) document_chunks = [ [](#__codelineno-1-5) "The superbowl this year was played between the San Francisco 49ers and the Kanasas City Chiefs", [](#__codelineno-1-6) "Pop corn is often served in a bowl", [](#__codelineno-1-7) "The Rice Bowl is a Chinese Restaurant located in the city of Tucson, Arizona", [](#__codelineno-1-8) "Mars is the fourth planet in the Solar System", [](#__codelineno-1-9) "An aquarium is a place where children can learn about marine life", [](#__codelineno-1-10) "The capital of the United States is Washington, D.C", [](#__codelineno-1-11) "Rock and Roll musicians are honored by being inducted in the Rock and Roll Hall of Fame", [](#__codelineno-1-12) "Music albums were published on Long Play Records in the 70s and 80s", [](#__codelineno-1-13) "Sichuan cuisine is a spicy cuisine from central China", [](#__codelineno-1-14) "The interest rates for mortgages is considered to be very high in 2024", [](#__codelineno-1-15) ] [](#__codelineno-1-16) [](#__codelineno-1-17) frm = FaissRM(document_chunks) [](#__codelineno-1-18) turbo = dspy.OpenAI(model="gpt-3.5-turbo") [](#__codelineno-1-19) dspy.settings.configure(lm=turbo, rm=frm) [](#__codelineno-1-20) print(frm(["I am in the mood for Chinese food"]))` Back to top --- # Custom RM Client - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/deep-dive/retrieval_models_clients/custom-rm-client.md "Edit this page") Creating Custom RM Client ========================= DSPy provides support for various retrieval modules out of the box like `ColBERTv2`, `AzureCognitiveSearch`, `Lancedb`, `Pinecone`, `Weaviate`, etc. Unlike Language Model (LM) modules, creating a custom RM module is much more simple and flexible. As of now, DSPy offers 2 ways to create a custom RM: the Pythonic way and the DSPythonic way. We'll take a look at both, understand why both are performing the same behavior, and how you can implement each! I/O of RM Client ---------------- Before understanding the implementation, let's understand the idea and I/O within RM modules. The **input** to an RM module is either 1) a single query or 2) a list of queries. The **output** is the `top-k` passages per query retrieved from a retrieval model, vector store, or search client. ![I/O in RM Module](../img/io_rm_module.png) Conventionally, we simply call the RM module object through the `__call__` method, inputting the query/queries as argument(s) of the call with the corresponding output returned as a list of strings. We'll see how this I/O is essentially same in both methods of implementation but differs in their delivery. The Pythonic Way ---------------- To account for our RM I/O, we create a class that conducts the retrieval logic, which we implement in the `__init__` and `__call__` methods: `[](#__codelineno-0-1) from typing import List, Union [](#__codelineno-0-2) [](#__codelineno-0-3) class PythonicRMClient: [](#__codelineno-0-4) def __init__(self): [](#__codelineno-0-5) pass [](#__codelineno-0-6) [](#__codelineno-0-7) def __call__(self, query: Union[str, List[str]], k:int) -> Union[List[str], List[List[str]]]: [](#__codelineno-0-8) pass` Info Don't worry about the extensive type-hinting above. `typing` is a package that provides type-definitions for function inputs and outputs. `Union` covers all possible types of the argument/output. So: \* `Union[str, List[str]]`: Assigned to `query` to work with a single query string or a list of queries strings. \* `Union[List[str], List[List[str]]]`: Assigned to the output of `__call__` to work with a single query string as a list or a list of multiple query string lists. So let's start by implementing `PythonicRMClient` for a local retrieval model hosted on a API with endpoint being `/`. We'll start by implementing the `__init__` method, which simply initializes the class attributes, `url` and `port`, and attaches the port to the url if present. ``[](#__codelineno-1-1) def __init__(self, url: str, port:int = None): [](#__codelineno-1-2) self.url = f`{url}:{port}` if port else url`` Now it's time to write the retrieval logic in `__call__` method: `[](#__codelineno-2-1) def __call__(self, query:str, k:int) -> List[str]: [](#__codelineno-2-2) params = {"query": query, "k": k} [](#__codelineno-2-3) response = requests.get(self.url, params=params) [](#__codelineno-2-4) [](#__codelineno-2-5) response = response.json()["retrieved_passages"] # List of top-k passages [](#__codelineno-2-6) return response` This serves to represent our API request call to retrieve our list of **top-k passages** which we return as the response. Let's bring it all together and see how our RM class looks like: ``[](#__codelineno-3-1) from typing import List [](#__codelineno-3-2) [](#__codelineno-3-3) class PythonicRMClient: [](#__codelineno-3-4) def __init__(self, url: str, port:int = None): [](#__codelineno-3-5) self.url = f`{url}:{port}` if port else url [](#__codelineno-3-6) [](#__codelineno-3-7) def __call__(self, query:str, k:int) -> List[str]: [](#__codelineno-3-8) # Only accept single query input, feel free to modify it to support [](#__codelineno-3-9) [](#__codelineno-3-10) params = {"query": query, "k": k} [](#__codelineno-3-11) response = requests.get(self.url, params=params) [](#__codelineno-3-12) [](#__codelineno-3-13) response = response.json()["retrieved_passages"] # List of top k passages [](#__codelineno-3-14) return response`` That's all!! This is the most basic way to implement a RM model and mirrors DSP-v1-hosted RM models like `ColBERTv2` and `AzureCognitiveSearch`. Now let's take a look at how we streamline this process in DSPy! The DSPythonic Way ------------------ The DSPythonic way mirrors the Pythonic way in maintaining the same input but now returning an object of `dspy.Prediction` class, the standard output format for any DSPy module as we've seen in previous docs. Additionally, this class would now inherit the `dspy.Retrieve` class to maintain state management within the DSPy library. So let's implement `__init__` and `forward` method where our class's `__call__` is calling the `forward` method as is=: `[](#__codelineno-4-1) import dspy [](#__codelineno-4-2) from typing import List, Union, Optional [](#__codelineno-4-3) [](#__codelineno-4-4) class DSPythonicRMClient(dspy.Retrieve): [](#__codelineno-4-5) def __init__(self, k:int): [](#__codelineno-4-6) pass [](#__codelineno-4-7) [](#__codelineno-4-8) def forward(self, query: Union[str, List[str]], k:Optional[str]) -> dspy.Prediction: [](#__codelineno-4-9) pass` Unlike `PythonicRMClient`, we initialize `k` as part of the initialization call and the `forward` method will take query/queries as arguments and the `k` number of retrieved passages as an optional argument. `k` is used within the inherited `dspy.Retrieve` initialization when we call `super().__init__()`. We'll be implementing `DSPythonicRMClient` for the same local retrieval model API we used above. We'll start by implementing the `__init__` method, which mirrors the `PythonicRMClient`. ``[](#__codelineno-5-1) def __init__(self, url: str, port:int = None, k:int = 3): [](#__codelineno-5-2) super().__init__(k=k) [](#__codelineno-5-3) [](#__codelineno-5-4) self.url = f`{url}:{port}` if port else url`` We'll now implement the `forward` method, returning the output as a `dspy.Prediction` object under the `passage` attribute which is standard among all the RM modules. The call will default to the defined `self.k` argument unless overridden in this call. `[](#__codelineno-6-1) def forward(self, query:str, k:Optional[int]) -> dspy.Prediction: [](#__codelineno-6-2) params = {"query": query, "k": k if k else self.k} [](#__codelineno-6-3) response = requests.get(self.url, params=params) [](#__codelineno-6-4) [](#__codelineno-6-5) response = response.json()["retrieved_passages"] # List of top k passages [](#__codelineno-6-6) return dspy.Prediction( [](#__codelineno-6-7) passages=response [](#__codelineno-6-8) )` Let's bring it all together and see how our RM class looks like: ``[](#__codelineno-7-1) import dspy [](#__codelineno-7-2) from typing import List, Union, Optional [](#__codelineno-7-3) [](#__codelineno-7-4) class DSPythonicRMClient(dspy.Retrieve): [](#__codelineno-7-5) def __init__(self, url: str, port:int = None, k:int = 3): [](#__codelineno-7-6) super().__init__(k=k) [](#__codelineno-7-7) [](#__codelineno-7-8) self.url = f`{url}:{port}` if port else url [](#__codelineno-7-9) [](#__codelineno-7-10) def forward(self, query_or_queries:str, k:Optional[int]) -> dspy.Prediction: [](#__codelineno-7-11) params = {"query": query_or_queries, "k": k if k else self.k} [](#__codelineno-7-12) response = requests.get(self.url, params=params) [](#__codelineno-7-13) [](#__codelineno-7-14) response = response.json()["retrieved_passages"] # List of top k passages [](#__codelineno-7-15) return dspy.Prediction( [](#__codelineno-7-16) passages=response [](#__codelineno-7-17) )`` That's all!! This is the way to implement a custom RM model client within DSPy and how more recently-supported RM models like `QdrantRM`, `WeaviateRM`, `LancedbRM`, etc. are implemented in DSPy. Let's take a look at how we use these retrievers. Using Custom RM Models ---------------------- DSPy offers two ways of using custom RM clients: Direct Method and using `dspy.Retrieve`. ### Direct Method The most straightforward way to use your custom RM is by directly using its object within the DSPy Pipeline. Let's take a look at the following pseudocode of a DSPy Pipeline as an example: `[](#__codelineno-8-1) class DSPyPipeline(dspy.Module): [](#__codelineno-8-2) def __init__(self): [](#__codelineno-8-3) super().__init__() [](#__codelineno-8-4) [](#__codelineno-8-5) url = "http://0.0.0.0" [](#__codelineno-8-6) port = 3000 [](#__codelineno-8-7) [](#__codelineno-8-8) self.pythonic_rm = PythonicRMClient(url=url, port=port) [](#__codelineno-8-9) self.dspythonic_rm = DSPythonicRMClient(url=url, port=port, k=3) [](#__codelineno-8-10) [](#__codelineno-8-11) ... [](#__codelineno-8-12) [](#__codelineno-8-13) def forward(self, *args): [](#__codelineno-8-14) ... [](#__codelineno-8-15) [](#__codelineno-8-16) passages_from_pythonic_rm = self.pythonic_rm(query) [](#__codelineno-8-17) passages_from_dspythonic_rm = self.dspythonic_rm(query).passages [](#__codelineno-8-18) [](#__codelineno-8-19) ...` This ensures you retrieve a list of passages from your RM client and can interact with the results within your forward pass in whichever way needed for your pipeline's purpose! ### Using `dspy.Retrieve` This way is more experimental in essence, allowing you to maintain the same pipeline and experiment with different RMs. How? By configuring it! `[](#__codelineno-9-1) import dspy [](#__codelineno-9-2) [](#__codelineno-9-3) lm = ... [](#__codelineno-9-4) url = "http://0.0.0.0" [](#__codelineno-9-5) port = 3000 [](#__codelineno-9-6) [](#__codelineno-9-7) # pythonic_rm = PythonicRMClient(url=url, port=port) [](#__codelineno-9-8) dspythonic_rm = DSPythonicRMClient(url=url, port=port, k=3) [](#__codelineno-9-9) [](#__codelineno-9-10) dspy.settings.configure(lm=lm, rm=dspythonic_rm)` Now, in the pipeline, you just need to use `dspy.Retrieve` which will use this `rm` client to get the top-k passage for a given query! `[](#__codelineno-10-1) class DSPyPipeline(dspy.Module): [](#__codelineno-10-2) def __init__(self): [](#__codelineno-10-3) super().__init__() [](#__codelineno-10-4) [](#__codelineno-10-5) url = "http://0.0.0.0" [](#__codelineno-10-6) port = 3000 [](#__codelineno-10-7) [](#__codelineno-10-8) self.rm = dspy.Retrieve(k=3) [](#__codelineno-10-9) ... [](#__codelineno-10-10) [](#__codelineno-10-11) def forward(self, *args): [](#__codelineno-10-12) ... [](#__codelineno-10-13) [](#__codelineno-10-14) passages = self.rm(query) [](#__codelineno-10-15) [](#__codelineno-10-16) ...` Now if you'd like to use a different RM, you can just update the `rm` parameter via `dspy.settings.configure`. How `dspy.Retrieve` uses `rm` When we call `dspy.Retrieve` the `__call__` method will execute the `forward` method as is. In `forward`, the top-k passages are received by the `dsp.retrieveEnsemble` method in [search.py](https://github.com/stanfordnlp/dspy/blob/main/dsp/primitives/search.py) . If an `rm` is not initialized in `dsp.settings`, this would raise an error. Back to top --- # LancedbRM - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/deep-dive/retrieval_models_clients/LancedbRM.md "Edit this page") LancedbRM ========= [LanceDB](http://lancedb.com/) is a developer-friendly, open source database for AI. From hyper scalable vector search and advanced retrieval for RAG, to streaming training data and interactive exploration of large scale AI datasets, LanceDB is the best foundation for your AI application. Setting Up LancedbRM -------------------- `LancedbRM` can be instantiated with any custom vectorizer and configured to return any payload field. * `table_name (str)`: The name of the table to query against. * `persist_directory (str)`: directory where database is stored. * `k (int, optional)`: The number of top passages to retrieve. Defaults to 3. Under the Hood -------------- ### `forward(self, query_or_queries: Union[str, list[str]], k: Optional[int] = None) -> dspy.Prediction` **Parameters:** * `query_or_queries (Union[str, List[str]])`: The query or queries to search for. * `k (Optional[int])`: The number of top passages to retrieve. Defaults to `self.k`. **Returns:** * `dspy.Prediction`: Contains the retrieved passages, each represented as a `dotdict` with a `long_text` attribute. Example Usage ------------- `[](#__codelineno-0-1) import os [](#__codelineno-0-2) import pandas as pd [](#__codelineno-0-3) [](#__codelineno-0-4) import dspy [](#__codelineno-0-5) from dspy.retrieve.lancedb import LancedbRM [](#__codelineno-0-6) [](#__codelineno-0-7) from lancedb.pydantic import LanceModel, Vector [](#__codelineno-0-8) from lancedb.embeddings import get_registry [](#__codelineno-0-9) [](#__codelineno-0-10) from datasets import load_dataset [](#__codelineno-0-11) [](#__codelineno-0-12) # load_dataset [](#__codelineno-0-13) ds = load_dataset("fancyzhx/dbpedia_14") [](#__codelineno-0-14) df = pd.DataFrame(ds['train']) [](#__codelineno-0-15) [](#__codelineno-0-16) os.environ["OPENAI_API_KEY"] = "" [](#__codelineno-0-17) [](#__codelineno-0-18) uri = 'tmp/db' [](#__codelineno-0-19) table_name = 'passages' [](#__codelineno-0-20) db = lancedb.connect(uri) [](#__codelineno-0-21) model = get_registry().get("sentence-transformers").create(name="BAAI/bge-small-en-v1.5", device="cpu") [](#__codelineno-0-22) [](#__codelineno-0-23) class Passages(LanceModel): [](#__codelineno-0-24) text: str = model.SourceField() [](#__codelineno-0-25) vector: Vector(model.ndims()) = model.VectorField() [](#__codelineno-0-26) [](#__codelineno-0-27) table = db.create_table(table_name, schema=Passages) [](#__codelineno-0-28) table.add(df['content']) [](#__codelineno-0-29) [](#__codelineno-0-30) [](#__codelineno-0-31) lancedb_retriever = LancedbRM( [](#__codelineno-0-32) table_name=table_name, [](#__codelineno-0-33) persist_directory=uri, [](#__codelineno-0-34) ) [](#__codelineno-0-35) [](#__codelineno-0-36) dspy.settings.configure(rm=lancedb_retriever) [](#__codelineno-0-37) retrieve = dspy.Retrieve() [](#__codelineno-0-38) [](#__codelineno-0-39) retrieve("Integrated Circuits")` Back to top --- # MyScaleRM - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/deep-dive/retrieval_models_clients/MyScaleRM.md "Edit this page") MyScaleRM ========= Constructor ----------- Initializes an instance of the `MyScaleRM` class, which is designed to use MyScaleDB (a ClickHouse fork optimized for vector similarity and full-text search) to retrieve documents based on query embeddings. This class supports embedding generation using either local models or OpenAI's API and manages database interactions efficiently. ### Syntax `[](#__codelineno-0-1) MyScaleRM( [](#__codelineno-0-2) client: clickhouse_connect.driver.client.Client, [](#__codelineno-0-3) table: str, [](#__codelineno-0-4) database: str = 'default', [](#__codelineno-0-5) metadata_columns: List[str] = ['text'], [](#__codelineno-0-6) vector_column: str = 'vector', [](#__codelineno-0-7) k: int = 3, [](#__codelineno-0-8) openai_api_key: Optional[str] = None, [](#__codelineno-0-9) openai_model: Optional[str] = None, [](#__codelineno-0-10) local_embed_model: Optional[str] = None [](#__codelineno-0-11) )` Parameters for `MyScaleRM` Constructor -------------------------------------- * `client` (_clickhouse\_connect.driver.client.Client_): A client connection to the MyScaleDB database, used to execute queries and manage interactions with the database. * `table` (_str_): Specifies the table within MyScaleDB from which data will be retrieved. This table should be equipped with a vector column for conducting similarity searches. * `database` (_str_, optional): The name of the database where the table is located, defaulting to `"default"`. * `metadata_columns` (_List\[str\], optional_): Columns to include as metadata in the output, defaulting to `["text"]`. * `vector_column` (_str, optional_): The column that contains vector data, used for similarity searches, defaulting to `"vector"`. * `k` (_int, optional_): The number of closest matches to return for each query, defaulting to 3. * `openai_api_key` (_str, optional_): API key for accessing OpenAI services, necessary if using OpenAI for embedding generation. * `openai_model` (_str, optional_): The specific OpenAI model to use for embeddings, required if an OpenAI API key is provided. * `local_embed_model` (_str, optional_): Specifies a local model for embedding generation, chosen if local computation is preferred. Methods ------- ### `forward` Executes a retrieval operation based on a user's query and returns the top `k` relevant results using the embeddings generated by the specified method. ### Syntax `[](#__codelineno-1-1) def forward(self, user_query: str, k: Optional[int] = None) -> dspy.Prediction` Parameters ---------- * `user_query` (_str_): The query to retrieve matching passages. * `k` (_Optional\[int\], optional_): The number of top matches to retrieve. If not provided, it defaults to the `k` value set during class initialization. Returns ------- * `dspy.Prediction`: Contains the retrieved passages, formatted as a list of `dotdict` objects. Each entry includes: * **long\_text (str)**: The text content of the retrieved passage. Description ----------- The `forward` method leverages the MyScaleDB's vector search capabilities to find the top `k` passages that best match the provided query. This method is integral for utilizing the MyScaleRM class to access and retrieve data efficiently based on semantic similarity, facilitated by the chosen embedding generation technique (either via a local model or the OpenAI API). Quickstart ---------- This section provides practical examples of how to instantiate and use the `MyScaleRM` class to retrieve data from MyScaleDB efficiently using text embeddings. `[](#__codelineno-2-1) from dspy.retrieve.myscaledb_rm import MyScaleRM [](#__codelineno-2-2) [](#__codelineno-2-3) MyScale_model = MyScaleRM(client=client, [](#__codelineno-2-4) table="table_name", [](#__codelineno-2-5) openai_api_key="sk-***", [](#__codelineno-2-6) openai_model="embeddings_model", [](#__codelineno-2-7) vector_column="vector_column_name", [](#__codelineno-2-8) metadata_columns=["add_your_columns_here"], [](#__codelineno-2-9) k=6) [](#__codelineno-2-10) [](#__codelineno-2-11) MyScale_model("Please suggest me some funny movies") [](#__codelineno-2-12) [](#__codelineno-2-13) passages = results.passages [](#__codelineno-2-14) [](#__codelineno-2-15) # Loop through each passage and print the 'long_text' [](#__codelineno-2-16) for passage in passages: [](#__codelineno-2-17) print(passage['long_text'], "\n")` Back to top --- # MilvusRM - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/deep-dive/retrieval_models_clients/MilvusRM.md "Edit this page") MilvusRM ======== MilvusRM uses OpenAI's `text-embedding-3-small` embedding by default or any customized embedding function. To support passage retrieval, it assumes that a Milvus collection has been created and populated with the following field: * `text`: The text of the passage Set up the MilvusRM Client -------------------------- The constructor initializes an instance of the `MilvusRM` class, with the option to use OpenAI's `text-embedding-3-small` embeddings or any customized embedding function . * `collection_name (str)`: The name of the Milvus collection to query against. * `uri (str, optional)`: The Milvus connection uri. Defaults to "http://localhost:19530". * `token (str, optional)`: The Milvus connection token. Defaults to None. * `db_name (str, optional)`: The Milvus database name. Defaults to "default". * `embedding_function (callable, optional)`: The function to convert a list of text to embeddings. The embedding function should take a list of text strings as input and output a list of embeddings. Defaults to None. By default, it will get OpenAI client by the environment variable OPENAI\_API\_KEY and use OpenAI's embedding model "text-embedding-3-small" with the default dimension. * `k (int, optional)`: The number of top passages to retrieve. Defaults to 3. Example of the MilvusRM constructor: `[](#__codelineno-0-1) MilvusRM( [](#__codelineno-0-2) collection_name: str, [](#__codelineno-0-3) uri: Optional[str] = "http://localhost:19530", [](#__codelineno-0-4) token: Optional[str] = None, [](#__codelineno-0-5) db_name: Optional[str] = "default", [](#__codelineno-0-6) embedding_function: Optional[Callable] = None, [](#__codelineno-0-7) k: int = 3, [](#__codelineno-0-8) )` Under the Hood -------------- ### `forward(self, query_or_queries: Union[str, List[str]], k: Optional[int] = None) -> dspy.Prediction` **Parameters:** - `query_or_queries` (_Union\[str, List\[str\]\]_): The query or list of queries to search for. - `k` (_Optional\[int\]_, _optional_): The number of results to retrieve. If not specified, defaults to the value set during initialization. **Returns:** - `dspy.Prediction`: Contains the retrieved passages, each represented as a `dotdict` with a `long_text` attribute. Search the Milvus collection for the top `k` passages matching the given query or queries, using embeddings generated via the default OpenAI embedding or the specified `embedding_function`. Sending Retrieval Requests via MilvusRM Client ---------------------------------------------- `[](#__codelineno-1-1) from dspy.retrieve.milvus_rm import MilvusRM [](#__codelineno-1-2) import os [](#__codelineno-1-3) [](#__codelineno-1-4) os.environ["OPENAI_API_KEY"] = "" [](#__codelineno-1-5) [](#__codelineno-1-6) retriever_model = MilvusRM( [](#__codelineno-1-7) collection_name="", [](#__codelineno-1-8) uri="", [](#__codelineno-1-9) token="" # ignore this if no token is required for Milvus connection [](#__codelineno-1-10) ) [](#__codelineno-1-11) [](#__codelineno-1-12) results = retriever_model("Explore the significance of quantum computing", k=5) [](#__codelineno-1-13) [](#__codelineno-1-14) for result in results: [](#__codelineno-1-15) print("Document:", result.long_text, "\n")` Back to top --- # RAGatouilleRM - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/deep-dive/retrieval_models_clients/RAGatouilleRM.md "Edit this page") RAGatouilleRM ============= ### Constructor The constructor initializes the `RAGatouille` class instance and sets up the required parameters for interacting with the index created using [RAGatouille library](https://github.com/bclavie/RAGatouille) . `[](#__codelineno-0-1) class RAGatouilleRM(dspy.Retrieve): [](#__codelineno-0-2) def __init__( [](#__codelineno-0-3) self, [](#__codelineno-0-4) index_root: str, [](#__codelineno-0-5) index_name: str, [](#__codelineno-0-6) k: int = 3, [](#__codelineno-0-7) ):` **Parameters:** * `index_root` (_str_): Folder path where your index is stored. * `index_name` (_str_): Name of the index you want to retrieve from. * `k` (_int_): The default number of passages to retrieve. Defaults to `3`. ### Methods #### `forward(self, query_or_queries: Union[str, List[str]], k:Optional[int]) -> dspy.Prediction` Enables making queries to the RAGatouille-made index for retrieval. Internally, the method handles the specifics of preparing the query to obtain the response. The function handles the retrieval of the top-k passages based on the provided query. **Parameters:** * `query_or_queries` (Union\[str, List\[str\]\]): Query string used for retrieval. * `k` (_int_, _optional_): Number of passages to retrieve. Defaults to 3. **Returns:** * `dspy.Prediction`: List of k passages Back to top --- # Neo4jRM - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/deep-dive/retrieval_models_clients/Neo4jRM.md "Edit this page") Neo4jRM ======= ### Constructor Initialize an instance of the `Neo4jRM` class. `[](#__codelineno-0-1) Neo4jRM( [](#__codelineno-0-2) index_name: str, [](#__codelineno-0-3) text_node_property: str, [](#__codelineno-0-4) k: int = 5, [](#__codelineno-0-5) retrieval_query: str = None, [](#__codelineno-0-6) embedding_provider: str = "openai", [](#__codelineno-0-7) embedding_model: str = "text-embedding-ada-002", [](#__codelineno-0-8) )` **Environment Variables:** You need to define the credentials as environment variables: * `NEO4J_USERNAME` (_str_): Specifies the username required for authenticating with the Neo4j database. This is a crucial security measure to ensure that only authorized users can access the database. * `NEO4J_PASSWORD` (_str_): Defines the password associated with the `NEO4J_USERNAME` for authentication purposes. This password should be kept secure to prevent unauthorized access to the database. * `NEO4J_URI` (_str_): Indicates the Uniform Resource Identifier (URI) used to connect to the Neo4j database. This URI typically includes the protocol, hostname, and port, providing the necessary information to establish a connection to the database. * `NEO4J_DATABASE` (_str_, optional): Specifies the name of the database to connect to within the Neo4j instance. If not set, the system defaults to using `"neo4j"` as the database name. This allows for flexibility in connecting to different databases within a single Neo4j server. * `OPENAI_API_KEY` (_str_): Specifies the API key required for authenticiating with OpenAI's services. **Parameters:** * `index_name` (_str_): Specifies the name of the vector index to be used within Neo4j for organizing and querying data. * `text_node_property` (_str_, _optional_): Defines the specific property of nodes that will be returned. * `k` (_int_, _optional_): The number of top results to return from the retrieval operation. It defaults to 5 if not explicitly specified. * `retrieval_query` (_str_, _optional_): A custom query string provided for retrieving data. If not provided, a default query tailored to the `text_node_property` will be used. * `embedding_provider` (_str_, _optional_): The name of the service provider for generating embeddings. Defaults to "openai" if not specified. * `embedding_model` (_str_, _optional_): The specific embedding model to use from the provider. By default, it uses the "text-embedding-ada-002" model from OpenAI. ### Methods #### `forward(self, query: [str], k: Optional[int] = None) -> dspy.Prediction` Search the neo4j vector index for the top `k` passages matching the given query or queries, using embeddings generated via the specified `embedding_model`. **Parameters:** * `query` (str\_): The query. * `k` (_Optional\[int\]_, _optional_): The number of results to retrieve. If not specified, defaults to the value set during initialization. **Returns:** * `dspy.Prediction`: Contains the retrieved passages as a list of string with the prediction signature. ex: `[](#__codelineno-1-1) Prediction( [](#__codelineno-1-2) passages=['Passage 1 Lorem Ipsum awesome', 'Passage 2 Lorem Ipsum Youppidoo', 'Passage 3 Lorem Ipsum Yassssss'] [](#__codelineno-1-3) )` ### Quick Example how to use Neo4j in a local environment. `[](#__codelineno-2-1) from dspy.retrieve.neo4j_rm import Neo4jRM [](#__codelineno-2-2) import os [](#__codelineno-2-3) [](#__codelineno-2-4) os.environ["NEO4J_URI"] = 'bolt://localhost:7687' [](#__codelineno-2-5) os.environ["NEO4J_USERNAME"] = 'neo4j' [](#__codelineno-2-6) os.environ["NEO4J_PASSWORD"] = 'password' [](#__codelineno-2-7) os.environ["OPENAI_API_KEY"] = 'sk-' [](#__codelineno-2-8) [](#__codelineno-2-9) retriever_model = Neo4jRM( [](#__codelineno-2-10) index_name="vector", [](#__codelineno-2-11) text_node_property="text" [](#__codelineno-2-12) ) [](#__codelineno-2-13) [](#__codelineno-2-14) results = retriever_model("Explore the significance of quantum computing", k=3) [](#__codelineno-2-15) [](#__codelineno-2-16) for passage in results: [](#__codelineno-2-17) print("Document:", passage, "\n")` Back to top --- # QdrantRM - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/deep-dive/retrieval_models_clients/QdrantRM.md "Edit this page") QdrantRM ======== [Qdrant](https://qdrant.tech/) is an open-source, high-performance vector search engine/database written in Rust. It can be used to retrieve semantically relevant passages to pass as context to your language model. You can refer to the [end-to-end example](https://github.com/stanfordnlp/dspy/blob/main/examples/integrations/qdrant/qdrant_retriever_example.ipynb) demonstrating the use of DSPy and Qdrant. Setting Up QdrantRM ------------------- `QdrantRM` can be instantiated with any custom vectorizer and configured to return any payload field. * `qdrant_collection_name (str)`: The name of the Qdrant collection. * `qdrant_client (QdrantClient)`: An instance of `qdrant_client.QdrantClient`. * `k (int, optional)`: The default number of top passages to retrieve. Default: 3. * `document_field (str, optional)`: The key in the Qdrant payload with the content. Default: `"document"`. * `vectorizer (BaseSentenceVectorizer, optional)`: An implementation `sentence_vectorizer.BaseSentenceVectorizer`. Default: `sentence_vectorizer.FastEmbedVectorizer`. * `vector_name (str, optional)`: Name of the vector in the collection. Default: The first available vector name. Under the Hood -------------- ### `forward(self, query_or_queries: Union[str, list[str]], k: Optional[int] = None, filter: Optional[models.Filter] = None) -> dspy.Prediction` **Parameters:** * `query_or_queries (Union[str, List[str]])`: The query or queries to search for. * `k (Optional[int])`: The number of top passages to retrieve. Defaults to `self.k`. * `filter (Optional[qdrant_client.models.Filter])`: "Only include points satisfying the [filter conditions](https://qdrant.tech/documentation/concepts/filtering/) ". Default: `None`. **Returns:** * `dspy.Prediction`: Contains the retrieved passages, each represented as a `dotdict` with a `long_text` attribute. Example Usage ------------- `[](#__codelineno-0-1) import os [](#__codelineno-0-2) [](#__codelineno-0-3) from qdrant_client import QdrantClient [](#__codelineno-0-4) [](#__codelineno-0-5) import dspy [](#__codelineno-0-6) from dsp.modules.sentence_vectorizer import OpenAIVectorizer [](#__codelineno-0-7) from dspy.retrieve.qdrant_rm import QdrantRM [](#__codelineno-0-8) [](#__codelineno-0-9) os.environ["OPENAI_API_KEY"] = "" [](#__codelineno-0-10) [](#__codelineno-0-11) client = QdrantClient(url="http://localhost:6333/") [](#__codelineno-0-12) vectorizer = OpenAIVectorizer(model="text-embedding-3-small") [](#__codelineno-0-13) [](#__codelineno-0-14) qdrant_retriever = QdrantRM( [](#__codelineno-0-15) qdrant_client=client, [](#__codelineno-0-16) qdrant_collection_name="{collection_name}", [](#__codelineno-0-17) vectorizer=vectorizer, [](#__codelineno-0-18) document_field="text", [](#__codelineno-0-19) ) [](#__codelineno-0-20) [](#__codelineno-0-21) dspy.settings.configure(rm=qdrant_retriever) [](#__codelineno-0-22) retrieve = dspy.Retrieve() [](#__codelineno-0-23) [](#__codelineno-0-24) retrieve("Some computer programs.")` Back to top --- # WatsonDiscovery - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/deep-dive/retrieval_models_clients/WatsonDiscovery.md "Edit this page") WatsonDiscoveryRM ================= ### Constructor The constructor initializes the `WatsonDiscoveryRM` class instance and sets up the request parameters for interacting with Watson Discovery service at IBM Cloud. `[](#__codelineno-0-1) class WatsonDiscoveryRM: [](#__codelineno-0-2) def __init__( [](#__codelineno-0-3) self, [](#__codelineno-0-4) apikey: str, [](#__codelineno-0-5) url:str, [](#__codelineno-0-6) version:str, [](#__codelineno-0-7) project_id:str, [](#__codelineno-0-8) collection_ids:list=[], [](#__codelineno-0-9) k: int = 7, [](#__codelineno-0-10) ):` **Parameters:** * `apikey` (str): apikey for authentication purposes, * `url` (str): endpoint URL that includes the service instance ID * `version` (str): Release date of the version of the API you want to use. Specify dates in YYYY-MM-DD format. * `project_id` (str): The Universally Unique Identifier (UUID) of the project. * `collection_ids` (list): An array containing the collections on which the search will be executed. * `k` (int, optional): The number of top passages to retrieve. Defaults to 7. ### Methods #### `forward(self, query_or_queries: Union[str, list[str]], k: Optional[int]= None) -> dspy.Prediction:` Search the Watson Discovery collection for the top `k` passages matching the given query or queries. **Parameters:** * `query_or_queries` (_Union\[str, list\[str\]\]_): The query or list of queries to search for. * `k` (_Optional\[int\]_, _optional_): The number of results to retrieve. If not specified, defaults to the value set during initialization. **Returns:** * `dspy.Prediction`: Contains the retrieved passages, each represented as a `dotdict` with schema `[{"title":str, "long_text": str, "passage_score": float, "document_id": str, "collection_id": str, "start_offset": int, "end_offset": int, "field": str}]` ### Quickstart `[](#__codelineno-1-1) import dspy [](#__codelineno-1-2) [](#__codelineno-1-3) retriever_model = WatsonDiscoveryRM( [](#__codelineno-1-4) apikey = "Your API Key", [](#__codelineno-1-5) url = "URL of the Watson Discovery Service", [](#__codelineno-1-6) version = "2023-03-31", [](#__codelineno-1-7) project_id = "Project Id", [](#__codelineno-1-8) collection_ids = ["Collection ID"], [](#__codelineno-1-9) k = 5 [](#__codelineno-1-10) ) [](#__codelineno-1-11) [](#__codelineno-1-12) retrieval_response = retriever_model("Explore the significance of quantum computing",k=5) [](#__codelineno-1-13) [](#__codelineno-1-14) for result in retrieval_response: [](#__codelineno-1-15) print("Document:", result.long_text, "\n")` Back to top --- # SnowflakeRM - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/deep-dive/retrieval_models_clients/SnowflakeRM.md "Edit this page") SnowflakeRM =========== ### Constructor Initialize an instance of the `SnowflakeRM` class, which enables user to leverage the Cortex Search service for hybrid retrieval. Before using this, ensure the Cortex Search service is configured as outlined in the documentation [here](https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-search/cortex-search-overview#overview) `[](#__codelineno-0-1) SnowflakeRM( [](#__codelineno-0-2) snowflake_session: object, [](#__codelineno-0-3) cortex_search_service: str, [](#__codelineno-0-4) snowflake_database: str, [](#__codelineno-0-5) snowflake_schema: dict, [](#__codelineno-0-6) auto_filter:bool, [](#__codelineno-0-7) k: int = 3, [](#__codelineno-0-8) )` **Parameters:** * `snowflake_session (str)`: Snowflake Snowpark session for connecting to Snowflake. * `cortex_search_service (str)`: The name of the Cortex Search service to be used. * `snowflake_database (str)`: The name of the Snowflake database to be used with the Cortex Search service. * `snowflake_schema (str)`: The name of the Snowflake schema to be used with the Cortex Search service. * `auto_filter (bool)`: Auto-generate metadata filter and push it down to Cortex Search service prior to retrieving results. * `k (int, optional)`: The number of top passages to retrieve. Defaults to 3. ### Methods #### `def forward(self,query_or_queries: Union[str, list[str]],response_columns:list[str],filters:dict = None, k: Optional[int] = None)-> dspy.Prediction:` Query the Cortex Search service to retrieve the top k relevant results given a query. **Parameters:** * `query_or_queries` (_Union\[str, List\[str\]\]_): The query or list of queries to search for. * `retrieval_columns` (str)\`: A list of columns to return for each relevant result in the response. * `search_filter` (_Optional\[dict\]_): Optional filter object used for filtering results based on data in the ATTRIBUTES columns. See [Filter syntax](https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-search/query-cortex-search-service#filter-syntax) * `k` (_Optional\[int\]_): The number of results to retrieve. If not specified, defaults to the value set during initialization. **Returns:** * `dspy.Prediction`: Contains the retrieved passages, each represented as a `dotdict` with schema `[{"long_text": str}]` ### Quickstart To support passage retrieval from a Snowflake table with this integration, a [Cortex Search](https://docs.snowflake.com/en/user-guide/snowflake-cortex/cortex-search/cortex-search-overview) endpoint must first be configured. `[](#__codelineno-1-1) from dspy.retrieve.snowflake_rm import SnowflakeRM [](#__codelineno-1-2) from snowflake.snowpark import Session [](#__codelineno-1-3) import os [](#__codelineno-1-4) [](#__codelineno-1-5) connection_parameters = { [](#__codelineno-1-6) [](#__codelineno-1-7) "account": os.getenv('SNOWFLAKE_ACCOUNT'), [](#__codelineno-1-8) "user": os.getenv('SNOWFLAKE_USER'), [](#__codelineno-1-9) "password": os.getenv('SNOWFLAKE_PASSWORD'), [](#__codelineno-1-10) "role": os.getenv('SNOWFLAKE_ROLE'), [](#__codelineno-1-11) "warehouse": os.getenv('SNOWFLAKE_WAREHOUSE'), [](#__codelineno-1-12) "database": os.getenv('SNOWFLAKE_DATABASE'), [](#__codelineno-1-13) "schema": os.getenv('SNOWFLAKE_SCHEMA')} [](#__codelineno-1-14) [](#__codelineno-1-15) # Establish connection to Snowflake [](#__codelineno-1-16) snowpark = Session.builder.configs(connection_parameters).create() [](#__codelineno-1-17) [](#__codelineno-1-18) snowflake_retriever = SnowflakeRM(snowflake_session=snowpark, [](#__codelineno-1-19) cortex_search_service="", [](#__codelineno-1-20) snowflake_database="", [](#__codelineno-1-21) snowflake_schema="", [](#__codelineno-1-22) auto_filter=True, [](#__codelineno-1-23) k = 5) [](#__codelineno-1-24) [](#__codelineno-1-25) results = snowflake_retriever("Explore the meaning of life", [](#__codelineno-1-26) response_columns=[""," dspy.Prediction` ### Parameters * `query_or_queries` (Union\[str, List\[str\]\]): The query or queries to search for * `k (Optional[int]): The number of top passages to retrieve. It defaults to`self.k\` * `**kwargs`: Additional keyword arguments like `rerank` for example ### Returns * `dspy.Prediction`: An object containing the retrieved passages Sending Retrieval Requests via the WeaviateRM Client ---------------------------------------------------- Here is an example of the Weaviate constructor using embedded: `[](#__codelineno-1-1) import weaviate [](#__codelineno-1-2) import dspy [](#__codelineno-1-3) from dspy.retrieve.weaviate_rm import WeaviateRM [](#__codelineno-1-4) [](#__codelineno-1-5) weaviate_client = weaviate.connect_to_embedded() # you can also use local or WCD [](#__codelineno-1-6) [](#__codelineno-1-7) retriever_model = WeaviateRM( [](#__codelineno-1-8) weaviate_collection_name="", [](#__codelineno-1-9) weaviate_client=weaviate_client [](#__codelineno-1-10) ) [](#__codelineno-1-11) [](#__codelineno-1-12) results = retriever_model("Explore the significance of quantum computing", k=5) [](#__codelineno-1-13) [](#__codelineno-1-14) for result in results: [](#__codelineno-1-15) print("Document:", result.long_text, "\n")` You can follow along with more DSPy and Weaviate examples [here](https://weaviate.io/developers/integrations/llm-frameworks/dspy) ! Back to top --- # Retrieval-Augmented Generation - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/tutorials/rag/index.ipynb "Edit this page") Tutorial: Retrieval-Augmented Generation (RAG)[¶](#tutorial-retrieval-augmented-generation-rag) ================================================================================================ Let's walk through a quick example of **basic question answering** with and without **retrieval-augmented generation** (RAG) in DSPy. Specifically, let's build **a system for answering Tech questions**, e.g. about Linux or iPhone apps. Install the latest DSPy via `pip install -U dspy` and follow along. If you're looking instead for a conceptual overview of DSPy, this [recent lecture](https://www.youtube.com/live/JEMYuzrKLUw) is a good place to start. Configuring the DSPy environment.[¶](#configuring-the-dspy-environment) ------------------------------------------------------------------------ Let's tell DSPy that we will use OpenAI's `gpt-4o-mini` in our modules. To authenticate, DSPy will look into your `OPENAI_API_KEY`. You can easily swap this out for [other providers or local models](https://github.com/stanfordnlp/dspy/blob/main/examples/migration.ipynb) . In \[1\]: Copied! import dspy lm \= dspy.LM('openai/gpt-4o-mini') dspy.configure(lm\=lm) import dspy lm = dspy.LM('openai/gpt-4o-mini') dspy.configure(lm=lm) Exploring some basic DSPy Modules.[¶](#exploring-some-basic-dspy-modules) -------------------------------------------------------------------------- You can always prompt the LM directly via `lm(prompt="prompt")` or `lm(messages=[...])`. However, DSPy gives you `Modules` as a better way to define your LM functions. The simplest module is `dspy.Predict`. It takes a [DSPy Signature](/learn/programming/signatures) , i.e. a structured input/output schema, and gives you back a callable function for the behavior you specified. Let's use the "in-line" notation for signatures to declare a module that takes a `question` (of type `str`) as input and produces a `response` as an output. In \[2\]: Copied! qa \= dspy.Predict('question: str -> response: str') response \= qa(question\="what are high memory and low memory on linux?") print(response.response) qa = dspy.Predict('question: str -> response: str') response = qa(question="what are high memory and low memory on linux?") print(response.response) In Linux, "high memory" and "low memory" refer to different regions of the system's memory address space, particularly in the context of 32-bit architectures. - \*\*Low Memory\*\*: This typically refers to the memory that is directly accessible by the kernel. In a 32-bit system, this is usually the first 896 MB of RAM (from 0 to 896 MB). The kernel can directly map this memory, making it faster for the kernel to access and manage. Low memory is used for kernel data structures and for user processes that require direct access to memory. - \*\*High Memory\*\*: This refers to the memory above the low memory limit, which is not directly accessible by the kernel in a 32-bit system. This area is typically above 896 MB. The kernel cannot directly access this memory without using special mechanisms, such as mapping it into the kernel's address space when needed. High memory is used for user processes that require more memory than what is available in low memory. In summary, low memory is directly accessible by the kernel, while high memory requires additional steps for the kernel to access it, especially in 32-bit systems. In 64-bit systems, this distinction is less significant as the kernel can address a much larger memory space directly. Notice how the variable names we specified in the signature defined our input and output argument names and their role. Now, what did DSPy do to build this `qa` module? Nothing fancy in this example, yet. The module passed your signature, LM, and inputs to an Adapter, which is a layer that handles structuring the inputs and parsing structured outputs to fit your signature. Let's see it directly. You can inspect the `n` last prompts sent by DSPy easily. In \[3\]: Copied! dspy.inspect\_history(n\=1) dspy.inspect\_history(n=1) \[2024-11-23T23:16:35.966534\] System message: Your input fields are: 1. \`question\` (str) Your output fields are: 1. \`response\` (str) All interactions will be structured in the following way, with the appropriate values filled in. \[\[ ## question ## \]\] {question} \[\[ ## response ## \]\] {response} \[\[ ## completed ## \]\] In adhering to this structure, your objective is: Given the fields \`question\`, produce the fields \`response\`. User message: \[\[ ## question ## \]\] what are high memory and low memory on linux? Respond with the corresponding output fields, starting with the field \`\[\[ ## response ## \]\]\`, and then ending with the marker for \`\[\[ ## completed ## \]\]\`. Response: \[\[ ## response ## \]\] In Linux, "high memory" and "low memory" refer to different regions of the system's memory address space, particularly in the context of 32-bit architectures. - \*\*Low Memory\*\*: This typically refers to the memory that is directly accessible by the kernel. In a 32-bit system, this is usually the first 896 MB of RAM (from 0 to 896 MB). The kernel can directly map this memory, making it faster for the kernel to access and manage. Low memory is used for kernel data structures and for user processes that require direct access to memory. - \*\*High Memory\*\*: This refers to the memory above the low memory limit, which is not directly accessible by the kernel in a 32-bit system. This area is typically above 896 MB. The kernel cannot directly access this memory without using special mechanisms, such as mapping it into the kernel's address space when needed. High memory is used for user processes that require more memory than what is available in low memory. In summary, low memory is directly accessible by the kernel, while high memory requires additional steps for the kernel to access it, especially in 32-bit systems. In 64-bit systems, this distinction is less significant as the kernel can address a much larger memory space directly. \[\[ ## completed ## \]\] DSPy has various built-in modules, e.g. `dspy.ChainOfThought`, `dspy.ProgramOfThought`, and `dspy.ReAct`. These are interchangeable with basic `dspy.Predict`: they take your signature, which is specific to your task, and they apply general-purpose prompting techniques and inference-time strategies to it. For example, `dspy.ChainOfThought` is an easy way to elicit `reasoning` out of your LM before it commits to the outputs requested in your signature. In the example below, we'll omit `str` types (as the default type is string). You should feel free to experiment with other fields and types, e.g. try `topics: list[str]` or `is_realistic: bool`. In \[4\]: Copied! cot \= dspy.ChainOfThought('question -> response') cot(question\="should curly braces appear on their own line?") cot = dspy.ChainOfThought('question -> response') cot(question="should curly braces appear on their own line?") Out\[4\]: Prediction( reasoning='The placement of curly braces on their own line depends on the coding style and conventions being followed. In some programming languages and style guides, such as the Allman style, curly braces are placed on their own line to enhance readability. In contrast, other styles, like K&R style, place the opening brace on the same line as the control statement. Ultimately, it is a matter of personal or team preference, and consistency within a project is key.', response='Curly braces can appear on their own line depending on the coding style you are following. If you prefer a style that enhances readability, such as the Allman style, then yes, they should be on their own line. However, if you are following a different style, like K&R, they may not need to be. Consistency is important, so choose a style and stick with it.' ) Interestingly, asking for reasoning can make the output `response` shorter in this case. Is this a good thing or a bad thing? It depends on what you need: there's no free lunch, but DSPy gives you the tools to experiment with different strategies extremely quickly. By the way, `dspy.ChainOfThought` is implemented in DSPy, using `dspy.Predict`. This is a good place to `dspy.inspect_history` if you're curious. Using DSPy well involves evaluation and iterative development.[¶](#using-dspy-well-involves-evaluation-and-iterative-development) ---------------------------------------------------------------------------------------------------------------------------------- You already know a lot about DSPy at this point. If all you want is quick scripting, this much of DSPy already enables a lot. Sprinkling DSPy signatures and modules into your Python control flow is a pretty ergonomic way to just get stuff done with LMs. That said, you're likely here because you want to build a high-quality system and improve it over time. The way to do that in DSPy is to iterate fast by evaluating the quality of your system and using DSPy's powerful tools, e.g. Optimizers. Manipulating Examples in DSPy.[¶](#manipulating-examples-in-dspy) ------------------------------------------------------------------ To measure the quality of your DSPy system, you need (1) a bunch of input values, like `question`s for example, and (2) a `metric` that can score the quality of an output from your system. Metrics vary widely. Some metrics need ground-truth labels of ideal outputs, e.g. for classification or question answering. Other metrics are self-supervised, e.g. checking faithfulness or lack of hallucination, perhaps using a DSPy program as a judge of these qualities. Let's load a dataset of questions and their (pretty long) gold answers. Since we started this notebook with the goal of building **a system for answering Tech questions**, we obtained a bunch of StackExchange-based questions and their correct answers from the [RAG-QA Arena](https://arxiv.org/abs/2407.13998) dataset. In \[5\]: Copied! import ujson from dspy.utils import download \# Download question--answer pairs from the RAG-QA Arena "Tech" dataset. download("https://huggingface.co/dspy/cache/resolve/main/ragqa\_arena\_tech\_examples.jsonl") with open("ragqa\_arena\_tech\_examples.jsonl") as f: data \= \[ujson.loads(line) for line in f\] import ujson from dspy.utils import download # Download question--answer pairs from the RAG-QA Arena "Tech" dataset. download("https://huggingface.co/dspy/cache/resolve/main/ragqa\_arena\_tech\_examples.jsonl") with open("ragqa\_arena\_tech\_examples.jsonl") as f: data = \[ujson.loads(line) for line in f\] In \[6\]: Copied! \# Inspect one datapoint. data\[0\] \# Inspect one datapoint. data\[0\] Out\[6\]: {'question': 'why igp is used in mpls?', 'response': "An IGP exchanges routing prefixes between gateways/routers. \\nWithout a routing protocol, you'd have to configure each route on every router and you'd have no dynamic updates when routes change because of link failures. \\nFuthermore, within an MPLS network, an IGP is vital for advertising the internal topology and ensuring connectivity for MP-BGP inside the network.", 'gold\_doc\_ids': \[2822, 2823\]} Given a simple dict like this, let's create a list of `dspy.Example`s, which is the datatype that carries training (or test) datapoints in DSPy. When you build a `dspy.Example`, you should generally specify `.with_inputs("field1", "field2", ...)` to indicate which fields are inputs. The other fields are treated as labels or metadata. In \[7\]: Copied! data \= \[dspy.Example(\*\*d).with\_inputs('question') for d in data\] \# Let's pick an \`example\` here from the data. example \= data\[2\] example data = \[dspy.Example(\*\*d).with\_inputs('question') for d in data\] # Let's pick an \`example\` here from the data. example = data\[2\] example Out\[7\]: Example({'question': 'why are my text messages coming up as maybe?', 'response': 'This is part of the Proactivity features new with iOS 9: It looks at info in emails to see if anyone with this number sent you an email and if it finds the phone number associated with a contact from your email, it will show you "Maybe". \\n\\nHowever, it has been suggested there is a bug in iOS 11.2 that can result in "Maybe" being displayed even when "Find Contacts in Other Apps" is disabled.', 'gold\_doc\_ids': \[3956, 3957, 8034\]}) (input\_keys={'question'}) Now, let's divide the data into: * Training (and with it Validation) set: * These are the splits you typically give to DSPy optimizers. * Optimizers typically learn directly from the training examples and check their progress using the validation examples. * It's good to have 30--300 examples for training and validation each. * For prompt optimizers in particular, it's often better to pass _more_ validation than training. * Below, we'll use 200 in total. MIPROv2 will split them into 20% training and 80% validation if you don't pass a valset. * Development and Test sets: The rest, typically on the order of 30--1000, can be used for: * development (i.e., you can inspect them as you iterate on your system) and * testing (final held-out evaluation). In \[8\]: Copied! import random random.Random(0).shuffle(data) trainset, devset, testset \= data\[:200\], data\[200:500\], data\[500:1000\] len(trainset), len(devset), len(testset) import random random.Random(0).shuffle(data) trainset, devset, testset = data\[:200\], data\[200:500\], data\[500:1000\] len(trainset), len(devset), len(testset) Out\[8\]: (200, 300, 500) Evaluation in DSPy.[¶](#evaluation-in-dspy) -------------------------------------------- What kind of metric can suit our question-answering task? There are many choices, but since the answers are long, we may ask: How well does the system response _cover_ all key facts in the gold response? And the other way around, how well is the system response _not saying things_ that aren't in the gold response? That metric is essentially a **semantic F1**, so let's load a `SemanticF1` metric from DSPy. This metric is actually implemented as a [very simple DSPy module](https://github.com/stanfordnlp/dspy/blob/main/dspy/evaluate/auto_evaluation.py#L21) using whatever LM we're working with. In \[9\]: Copied! from dspy.evaluate import SemanticF1 \# Instantiate the metric. metric \= SemanticF1(decompositional\=True) \# Produce a prediction from our \`cot\` module, using the \`example\` above as input. pred \= cot(\*\*example.inputs()) \# Compute the metric score for the prediction. score \= metric(example, pred) print(f"Question: \\t {example.question}\\n") print(f"Gold Response: \\t {example.response}\\n") print(f"Predicted Response: \\t {pred.response}\\n") print(f"Semantic F1 Score: {score:.2f}") from dspy.evaluate import SemanticF1 # Instantiate the metric. metric = SemanticF1(decompositional=True) # Produce a prediction from our \`cot\` module, using the \`example\` above as input. pred = cot(\*\*example.inputs()) # Compute the metric score for the prediction. score = metric(example, pred) print(f"Question: \\t {example.question}\\n") print(f"Gold Response: \\t {example.response}\\n") print(f"Predicted Response: \\t {pred.response}\\n") print(f"Semantic F1 Score: {score:.2f}") Question: why are my text messages coming up as maybe? Gold Response: This is part of the Proactivity features new with iOS 9: It looks at info in emails to see if anyone with this number sent you an email and if it finds the phone number associated with a contact from your email, it will show you "Maybe". However, it has been suggested there is a bug in iOS 11.2 that can result in "Maybe" being displayed even when "Find Contacts in Other Apps" is disabled. Predicted Response: Your text messages are showing up as "maybe" because your messaging app is uncertain about the sender's identity. This typically occurs when the sender's number is not saved in your contacts or if the message is from an unknown number. To resolve this, you can save the contact in your address book or check the message settings in your app. Semantic F1 Score: 0.33 The final DSPy module call above actually happens inside `metric`. You might be curious how it measured the semantic F1 for this example. In \[10\]: Copied! dspy.inspect\_history(n\=1) dspy.inspect\_history(n=1) \[2024-11-23T23:16:36.149518\] System message: Your input fields are: 1. \`question\` (str) 2. \`ground\_truth\` (str) 3. \`system\_response\` (str) Your output fields are: 1. \`reasoning\` (str) 2. \`ground\_truth\_key\_ideas\` (str): enumeration of key ideas in the ground truth 3. \`system\_response\_key\_ideas\` (str): enumeration of key ideas in the system response 4. \`discussion\` (str): discussion of the overlap between ground truth and system response 5. \`recall\` (float): fraction (out of 1.0) of ground truth covered by the system response 6. \`precision\` (float): fraction (out of 1.0) of system response covered by the ground truth All interactions will be structured in the following way, with the appropriate values filled in. \[\[ ## question ## \]\] {question} \[\[ ## ground\_truth ## \]\] {ground\_truth} \[\[ ## system\_response ## \]\] {system\_response} \[\[ ## reasoning ## \]\] {reasoning} \[\[ ## ground\_truth\_key\_ideas ## \]\] {ground\_truth\_key\_ideas} \[\[ ## system\_response\_key\_ideas ## \]\] {system\_response\_key\_ideas} \[\[ ## discussion ## \]\] {discussion} \[\[ ## recall ## \]\] {recall} # note: the value you produce must be a single float value \[\[ ## precision ## \]\] {precision} # note: the value you produce must be a single float value \[\[ ## completed ## \]\] In adhering to this structure, your objective is: Compare a system's response to the ground truth to compute recall and precision of key ideas. You will first enumerate key ideas in each response, discuss their overlap, and then report recall and precision. User message: \[\[ ## question ## \]\] why are my text messages coming up as maybe? \[\[ ## ground\_truth ## \]\] This is part of the Proactivity features new with iOS 9: It looks at info in emails to see if anyone with this number sent you an email and if it finds the phone number associated with a contact from your email, it will show you "Maybe". However, it has been suggested there is a bug in iOS 11.2 that can result in "Maybe" being displayed even when "Find Contacts in Other Apps" is disabled. \[\[ ## system\_response ## \]\] Your text messages are showing up as "maybe" because your messaging app is uncertain about the sender's identity. This typically occurs when the sender's number is not saved in your contacts or if the message is from an unknown number. To resolve this, you can save the contact in your address book or check the message settings in your app. Respond with the corresponding output fields, starting with the field \`\[\[ ## reasoning ## \]\]\`, then \`\[\[ ## ground\_truth\_key\_ideas ## \]\]\`, then \`\[\[ ## system\_response\_key\_ideas ## \]\]\`, then \`\[\[ ## discussion ## \]\]\`, then \`\[\[ ## recall ## \]\]\` (must be formatted as a valid Python float), then \`\[\[ ## precision ## \]\]\` (must be formatted as a valid Python float), and then ending with the marker for \`\[\[ ## completed ## \]\]\`. Response: \[\[ ## reasoning ## \]\] The system response provides a general explanation for why text messages may show up as "maybe," focusing on the uncertainty of the sender's identity. However, it does not mention the specific iOS features or the potential bug in iOS 11.2 that are highlighted in the ground truth. The ground truth explains that this feature is part of iOS 9's Proactivity features and mentions a specific bug in iOS 11.2, which is not addressed in the system response. \[\[ ## ground\_truth\_key\_ideas ## \]\] 1. Proactivity features introduced in iOS 9. 2. The system checks emails to associate phone numbers with contacts. 3. "Maybe" is displayed if the number is not saved in contacts. 4. Mention of a bug in iOS 11.2 causing "Maybe" to appear incorrectly. \[\[ ## system\_response\_key\_ideas ## \]\] 1. Text messages show up as "maybe" due to uncertainty about the sender's identity. 2. Occurs when the sender's number is not saved in contacts or is from an unknown number. 3. Suggests saving the contact or checking message settings. \[\[ ## discussion ## \]\] There is some overlap between the ground truth and the system response regarding the uncertainty of the sender's identity and the suggestion to save the contact. However, the system response lacks specific details about the iOS features and the bug mentioned in the ground truth. The ground truth provides a more comprehensive explanation of the "maybe" feature, while the system response is more general and does not address the iOS version specifics. \[\[ ## recall ## \]\] 0.25 \[\[ ## precision ## \]\] 0.5 \[\[ ## completed ## \]\] For evaluation, you could use the metric above in a simple loop and just average the score. But for nice parallelism and utilities, we can rely on `dspy.Evaluate`. In \[11\]: Copied! \# Define an evaluator that we can re-use. evaluate \= dspy.Evaluate(devset\=devset, metric\=metric, num\_threads\=24, display\_progress\=True, display\_table\=2) \# Evaluate the Chain-of-Thought program. evaluate(cot) \# Define an evaluator that we can re-use. evaluate = dspy.Evaluate(devset=devset, metric=metric, num\_threads=24, display\_progress=True, display\_table=2) # Evaluate the Chain-of-Thought program. evaluate(cot) Average Metric: 125.68 / 300 (41.9%): 100%|██████████| 300/300 \[00:00<00:00, 666.96it/s\] 2024/11/23 23:16:36 INFO dspy.evaluate.evaluate: Average Metric: 125.68228336477591 / 300 (41.9%) | | question | example\_response | gold\_doc\_ids | reasoning | pred\_response | SemanticF1 | | --- | --- | --- | --- | --- | --- | --- | | 0 | when to use c over c++, and c++ over c? | If you are equally familiar with both C++ and C, it's advisable to... | \[733\] | C and C++ are both powerful programming languages, but they serve ... | Use C when you need low-level access to memory, require high perfo... | | | 1 | should images be stored in a git repository? | One viewpoint expresses that there is no significant downside, esp... | \[6253, 6254, 6275, 6278, 8215\] | Storing images in a Git repository can be beneficial for version c... | Images can be stored in a Git repository, but it's important to co... | ✔️ \[0.444\] | ... 298 more rows not displayed ... Out\[11\]: 41.89 So far, we built a very simple chain-of-thought module for question answering and evaluated it on a small dataset. Can we do better? In the rest of this guide, we will build a retrieval-augmented generation (RAG) program in DSPy for the same task. We'll see how this can boost the score substantially, then we'll use one of the DSPy Optimizers to _compile_ our RAG program to higher-quality prompts, raising our scores even more. Basic Retrieval-Augmented Generation (RAG).[¶](#basic-retrieval-augmented-generation-rag) ------------------------------------------------------------------------------------------ First, let's download the corpus data that we will use for RAG search. An older version of this tutorial used the full (650,000 document) corpus. To make this very fast and cheap to run, we've downsampled the corpus to just 28,000 documents. In \[12\]: Copied! download("https://huggingface.co/dspy/cache/resolve/main/ragqa\_arena\_tech\_corpus.jsonl") download("https://huggingface.co/dspy/cache/resolve/main/ragqa\_arena\_tech\_corpus.jsonl") Set up your system's retriever.[¶](#set-up-your-systems-retriever) ------------------------------------------------------------------- As far as DSPy is concerned, you can plug in any Python code for calling tools or retrievers. Here, we'll just use OpenAI Embeddings and do top-K search locally, just for convenience. **Note:** The step below will require that you either do `pip install -U faiss-cpu` or pass `brute_force_threshold=30_000` to `dspy.retrievers.Embeddings` to avoid faiss. In \[13\]: Copied! \# %pip install -U faiss-cpu # or faiss-gpu if you have a GPU \# %pip install -U faiss-cpu # or faiss-gpu if you have a GPU In \[14\]: Copied! max\_characters \= 6000 \# for truncating >99th percentile of documents topk\_docs\_to\_retrieve \= 5 \# number of documents to retrieve per search query with open("ragqa\_arena\_tech\_corpus.jsonl") as f: corpus \= \[ujson.loads(line)\['text'\]\[:max\_characters\] for line in f\] print(f"Loaded {len(corpus)} documents. Will encode them below.") embedder \= dspy.Embedder('openai/text-embedding-3-small', dimensions\=512) search \= dspy.retrievers.Embeddings(embedder\=embedder, corpus\=corpus, k\=topk\_docs\_to\_retrieve) max\_characters = 6000 # for truncating >99th percentile of documents topk\_docs\_to\_retrieve = 5 # number of documents to retrieve per search query with open("ragqa\_arena\_tech\_corpus.jsonl") as f: corpus = \[ujson.loads(line)\['text'\]\[:max\_characters\] for line in f\] print(f"Loaded {len(corpus)} documents. Will encode them below.") embedder = dspy.Embedder('openai/text-embedding-3-small', dimensions=512) search = dspy.retrievers.Embeddings(embedder=embedder, corpus=corpus, k=topk\_docs\_to\_retrieve) Loaded 28436 documents. Will encode them below. Training a 32-byte FAISS index with 337 partitions, based on 28436 x 512-dim embeddings Build your first RAG Module.[¶](#build-your-first-rag-module) -------------------------------------------------------------- In the previous guide, we looked at individual DSPy modules in isolation, e.g. `dspy.Predict("question -> answer")`. What if we want to build a DSPy _program_ that has multiple steps? The syntax below with `dspy.Module` allows you to connect a few pieces together, in this case, our retriever and a generation module, so the whole system can be optimized. Concretely, in the `__init__` method, you declare any sub-module you'll need, which in this case is just a `dspy.ChainOfThought('context, question -> response')` module that takes retrieved context, a question, and produces a response. In the `forward` method, you simply express any Python control flow you like, possibly using your modules. In this case, we first invoke the `search` function defined earlier and then invoke the `self.respond` ChainOfThought module. In \[15\]: Copied! class RAG(dspy.Module): def \_\_init\_\_(self): self.respond \= dspy.ChainOfThought('context, question -> response') def forward(self, question): context \= search(question).passages return self.respond(context\=context, question\=question) class RAG(dspy.Module): def \_\_init\_\_(self): self.respond = dspy.ChainOfThought('context, question -> response') def forward(self, question): context = search(question).passages return self.respond(context=context, question=question) Let's use the RAG module. In \[16\]: Copied! rag \= RAG() rag(question\="what are high memory and low memory on linux?") rag = RAG() rag(question="what are high memory and low memory on linux?") Out\[16\]: Prediction( reasoning="High Memory and Low Memory in Linux refer to two segments of the kernel's memory space. Low Memory is the portion of memory that the kernel can access directly and is statically mapped at boot time. This area is typically used for kernel data structures and is always accessible to the kernel. High Memory, on the other hand, is not permanently mapped in the kernel's address space, meaning that the kernel cannot access it directly without first mapping it into its address space. High Memory is used for user-space applications and temporary data buffers. The distinction allows for better memory management and security, as user-space applications cannot directly access kernel-space memory.", response="In Linux, High Memory refers to the segment of memory that is not permanently mapped in the kernel's address space, which means the kernel must map it temporarily to access it. This area is typically used for user-space applications and temporary data buffers. Low Memory, in contrast, is the portion of memory that the kernel can access directly and is statically mapped at boot time. It is used for kernel data structures and is always accessible to the kernel. This separation enhances security by preventing user-space applications from accessing kernel-space memory directly." ) In \[17\]: Copied! dspy.inspect\_history() dspy.inspect\_history() \[2024-11-23T23:16:49.175612\] System message: Your input fields are: 1. \`context\` (str) 2. \`question\` (str) Your output fields are: 1. \`reasoning\` (str) 2. \`response\` (str) All interactions will be structured in the following way, with the appropriate values filled in. \[\[ ## context ## \]\] {context} \[\[ ## question ## \]\] {question} \[\[ ## reasoning ## \]\] {reasoning} \[\[ ## response ## \]\] {response} \[\[ ## completed ## \]\] In adhering to this structure, your objective is: Given the fields \`context\`, \`question\`, produce the fields \`response\`. User message: \[\[ ## context ## \]\] \[1\] «As far as I remember, High Memory is used for application space and Low Memory for the kernel. Advantage is that (user-space) applications cant access kernel-space memory.» \[2\] «HIGHMEM is a range of kernels memory space, but it is NOT memory you access but its a place where you put what you want to access. A typical 32bit Linux virtual memory map is like: 0x00000000-0xbfffffff: user process (3GB) 0xc0000000-0xffffffff: kernel space (1GB) (CPU-specific vector and whatsoever are ignored here). Linux splits the 1GB kernel space into 2 pieces, LOWMEM and HIGHMEM. The split varies from installation to installation. If an installation chooses, say, 512MB-512MB for LOW and HIGH mems, the 512MB LOWMEM (0xc0000000-0xdfffffff) is statically mapped at the kernel boot time; usually the first so many bytes of the physical memory is used for this so that virtual and physical addresses in this range have a constant offset of, say, 0xc0000000. On the other hand, the latter 512MB (HIGHMEM) has no static mapping (although you could leave pages semi-permanently mapped there, but you must do so explicitly in your driver code). Instead, pages are temporarily mapped and unmapped here so that virtual and physical addresses in this range have no consistent mapping. Typical uses of HIGHMEM include single-time data buffers.» \[3\] «This is relevant to the Linux kernel; Im not sure how any Unix kernel handles this. The High Memory is the segment of memory that user-space programs can address. It cannot touch Low Memory. Low Memory is the segment of memory that the Linux kernel can address directly. If the kernel must access High Memory, it has to map it into its own address space first. There was a patch introduced recently that lets you control where the segment is. The tradeoff is that you can take addressable memory away from user space so that the kernel can have more memory that it does not have to map before using. Additional resources: http://tldp.org/HOWTO/KernelAnalysis-HOWTO-7.html http://linux-mm.org/HighMemory» \[4\] «The first reference to turn to is Linux Device Drivers (available both online and in book form), particularly chapter 15 which has a section on the topic. In an ideal world, every system component would be able to map all the memory it ever needs to access. And this is the case for processes on Linux and most operating systems: a 32-bit process can only access a little less than 2^32 bytes of virtual memory (in fact about 3GB on a typical Linux 32-bit architecture). It gets difficult for the kernel, which needs to be able to map the full memory of the process whose system call its executing, plus the whole physical memory, plus any other memory-mapped hardware device. So when a 32-bit kernel needs to map more than 4GB of memory, it must be compiled with high memory support. High memory is memory which is not permanently mapped in the kernels address space. (Low memory is the opposite: it is always mapped, so you can access it in the kernel simply by dereferencing a pointer.) When you access high memory from kernel code, you need to call kmap first, to obtain a pointer from a page data structure (struct page). Calling kmap works whether the page is in high or low memory. There is also kmap\_atomic which has added constraints but is more efficient on multiprocessor machines because it uses finer-grained locking. The pointer obtained through kmap is a resource: it uses up address space. Once youve finished with it, you must call kunmap (or kunmap\_atomic) to free that resource; then the pointer is no longer valid, and the contents of the page cant be accessed until you call kmap again.» \[5\] «/proc/meminfo will tell you how free works, but /proc/kcore can tell you what the kernel uses. From the same page: /proc/kcore This file represents the physical memory of the system and is stored in the ELF core file format. With this pseudo-file, and an unstripped kernel (/usr/src/linux/vmlinux) binary, GDB can be used to examine the current state of any kernel data structures. The total length of the file is the size of physical memory (RAM) plus 4KB. /proc/meminfo This file reports statistics about memory usage on the system. It is used by free(1) to report the amount of free and used memory (both physical and swap) on the system as well as the shared memory and buffers used by the kernel. Each line of the file consists of a parameter name, followed by a colon, the value of the parameter, and an option unit of measurement (e.g., kB). The list below describes the parameter names and the format specifier required to read the field value. Except as noted below, all of the fields have been present since at least Linux 2.6.0. Some fileds are displayed only if the kernel was configured with various options; those dependencies are noted in the list. MemTotal %lu Total usable RAM (i.e., physical RAM minus a few reserved bits and the kernel binary code). MemFree %lu The sum of LowFree+HighFree. Buffers %lu Relatively temporary storage for raw disk blocks that shouldnt get tremendously large (20MB or so). Cached %lu In-memory cache for files read from the disk (the page cache). Doesnt include SwapCached. SwapCached %lu Memory that once was swapped out, is swapped back in but still also is in the swap file. (If memory pressure is high, these pages dont need to be swapped out again because they are already in the swap file. This saves I/O.) Active %lu Memory that has been used more recently and usually not reclaimed unless absolutely necessary. Inactive %lu Memory which has been less recently used. It is more eligible to be reclaimed for other purposes. Active(anon) %lu (since Linux 2.6.28) \[To be documented.\] Inactive(anon) %lu (since Linux 2.6.28) \[To be documented.\] Active(file) %lu (since Linux 2.6.28) \[To be documented.\] Inactive(file) %lu (since Linux 2.6.28) \[To be documented.\] Unevictable %lu (since Linux 2.6.28) (From Linux 2.6.28 to 2.6.30, CONFIG\_UNEVICTABLE\_LRU was required.) \[To be documented.\] Mlocked %lu (since Linux 2.6.28) (From Linux 2.6.28 to 2.6.30, CONFIG\_UNEVICTABLE\_LRU was required.) \[To be documented.\] HighTotal %lu (Starting with Linux 2.6.19, CONFIG\_HIGHMEM is required.) Total amount of highmem. Highmem is all memory above ~860MB of physical memory. Highmem areas are for use by user-space programs, or for the page cache. The kernel must use tricks to access this memory, making it slower to access than lowmem. HighFree %lu (Starting with Linux 2.6.19, CONFIG\_HIGHMEM is required.) Amount of free highmem. LowTotal %lu (Starting with Linux 2.6.19, CONFIG\_HIGHMEM is required.) Total amount of lowmem. Lowmem is memory which can be used for everything that highmem can be used for, but it is also available for the kernels use for its own data structures. Among many other things, it is where everything from Slab is allocated. Bad things happen when youre out of lowmem. LowFree %lu (Starting with Linux 2.6.19, CONFIG\_HIGHMEM is required.) Amount of free lowmem. MmapCopy %lu (since Linux 2.6.29) (CONFIG\_MMU is required.) \[To be documented.\] SwapTotal %lu Total amount of swap space available. SwapFree %lu Amount of swap space that is currently unused. Dirty %lu Memory which is waiting to get written back to the disk. Writeback %lu Memory which is actively being written back to the disk. AnonPages %lu (since Linux 2.6.18) Non-file backed pages mapped into user-space page tables. Mapped %lu Files which have been mmaped, such as libraries. Shmem %lu (since Linux 2.6.32) \[To be documented.\] Slab %lu In-kernel data structures cache. SReclaimable %lu (since Linux 2.6.19) Part of Slab, that might be reclaimed, such as caches. SUnreclaim %lu (since Linux 2.6.19) Part of Slab, that cannot be reclaimed on memory pressure. KernelStack %lu (since Linux 2.6.32) Amount of memory allocated to kernel stacks. PageTables %lu (since Linux 2.6.18) Amount of memory dedicated to the lowest level of page tables. Quicklists %lu (since Linux 2.6.27) (CONFIG\_QUICKLIST is required.) \[To be documented.\] NFS\_Unstable %lu (since Linux 2.6.18) NFS pages sent to the server, but not yet committed to stable storage. Bounce %lu (since Linux 2.6.18) Memory used for block device bounce buffers. WritebackTmp %lu (since Linux 2.6.26) Memory used by FUSE for temporary writeback buffers. CommitLimit %lu (since Linux 2.6.10) Based on the overcommit ratio (vm.overcommit\_ratio), this is the total amount of memory currently available to be allocated on the system. This limit is adhered to only if strict overcommit accounting is enabled (mode 2 in /proc/sys/vm/overcommit\_ratio). The CommitLimit is calculated using the following formula: CommitLimit = (\[total RAM pages\] - \[total huge TLB pages\]) \* overcommit\_ratio / 100 + \[total swap pages\] For example, on a system with 1GB of physical RAM and 7GB of swap with a overcommit\_ratio of 30, this formula yields a CommitLimit of 7.3GB. For more details, see the memory overcommit documentation in the kernel source file Documentation/vm/overcommit-accounting. Committed\_AS %lu The amount of memory presently allocated on the system. The committed memory is a sum of all of the memory which has been allocated by processes, even if it has not been used by them as of yet. A process which allocates 1GB of memory (using malloc(3) or similar), but touches only 300MB of that memory will show up as using only 300MB of memory even if it has the address space allocated for the entire 1GB. This 1GB is memory which has been committed to by the VM and can be used at any time by the allocating application. With strict overcommit enabled on the system (mode 2 /proc/sys/vm/overcommit\_memory), allocations w» \[\[ ## question ## \]\] what are high memory and low memory on linux? Respond with the corresponding output fields, starting with the field \`\[\[ ## reasoning ## \]\]\`, then \`\[\[ ## response ## \]\]\`, and then ending with the marker for \`\[\[ ## completed ## \]\]\`. Response: \[\[ ## reasoning ## \]\] High Memory and Low Memory in Linux refer to two segments of the kernel's memory space. Low Memory is the portion of memory that the kernel can access directly and is statically mapped at boot time. This area is typically used for kernel data structures and is always accessible to the kernel. High Memory, on the other hand, is not permanently mapped in the kernel's address space, meaning that the kernel cannot access it directly without first mapping it into its address space. High Memory is used for user-space applications and temporary data buffers. The distinction allows for better memory management and security, as user-space applications cannot directly access kernel-space memory. \[\[ ## response ## \]\] In Linux, High Memory refers to the segment of memory that is not permanently mapped in the kernel's address space, which means the kernel must map it temporarily to access it. This area is typically used for user-space applications and temporary data buffers. Low Memory, in contrast, is the portion of memory that the kernel can access directly and is statically mapped at boot time. It is used for kernel data structures and is always accessible to the kernel. This separation enhances security by preventing user-space applications from accessing kernel-space memory directly. \[\[ ## completed ## \]\] Earlier with a CoT module, we got around 40% in terms of semantic F1 on our `devset`. Would this `RAG` module score better? In \[18\]: Copied! evaluate(RAG()) evaluate(RAG()) Average Metric: 166.54 / 300 (55.5%): 100%|██████████| 300/300 \[00:04<00:00, 61.40it/s\] 2024/11/23 23:16:54 INFO dspy.evaluate.evaluate: Average Metric: 166.53601368289284 / 300 (55.5%) | | question | example\_response | gold\_doc\_ids | reasoning | pred\_response | SemanticF1 | | --- | --- | --- | --- | --- | --- | --- | | 0 | when to use c over c++, and c++ over c? | If you are equally familiar with both C++ and C, it's advisable to... | \[733\] | C should be used over C++ primarily in scenarios where simplicity ... | Use C over C++ when working on embedded systems, requiring low-lev... | ✔️ \[0.500\] | | 1 | should images be stored in a git repository? | One viewpoint expresses that there is no significant downside, esp... | \[6253, 6254, 6275, 6278, 8215\] | Storing images in a Git repository is generally not recommended du... | While it is technically possible to store images in a Git reposito... | ✔️ \[0.444\] | ... 298 more rows not displayed ... Out\[18\]: 55.51 Using a DSPy Optimizer to improve your RAG prompt.[¶](#using-a-dspy-optimizer-to-improve-your-rag-prompt) ---------------------------------------------------------------------------------------------------------- Off the shelf, our `RAG` module scores 55%. What are our options to make it stronger? One of the various choices DSPy offers is optimizing the prompts in our pipeline. If there are many sub-modules in your program, all of them will be optimized together. In this case, there's only one: `self.respond = dspy.ChainOfThought('context, question -> response')` Let's set up and use DSPy's MIPRO (v2) optimizer. The run below has a cost around $1.5 (for the `medium` auto setting) and may take some 20-30 minutes depending on your number of threads. In \[ \]: Copied! tp \= dspy.MIPROv2(metric\=metric, auto\="medium", num\_threads\=24) \# use fewer threads if your rate limit is small optimized\_rag \= tp.compile(RAG(), trainset\=trainset, max\_bootstrapped\_demos\=2, max\_labeled\_demos\=2, requires\_permission\_to\_run\=False) tp = dspy.MIPROv2(metric=metric, auto="medium", num\_threads=24) # use fewer threads if your rate limit is small optimized\_rag = tp.compile(RAG(), trainset=trainset, max\_bootstrapped\_demos=2, max\_labeled\_demos=2, requires\_permission\_to\_run=False) The prompt optimization process here is pretty systematic, you can learn about it for example in this paper. Importantly, it's not a magic button. It's very possible that it can overfit your training set for instance and not generalize well to a held-out set, making it essential that we iteratively validate our programs. Let's check on an example here, asking the same question to the baseline `rag = RAG()` program, which was not optimized, and to the `optimized_rag = MIPROv2(..)(..)` program, after prompt optimization. In \[20\]: Copied! baseline \= rag(question\="cmd+tab does not work on hidden or minimized windows") print(baseline.response) baseline = rag(question="cmd+tab does not work on hidden or minimized windows") print(baseline.response) You are correct that cmd+tab does not work on hidden or minimized windows. To switch back to a minimized app, you must first switch to another application and let it take focus before returning to the minimized one. In \[21\]: Copied! pred \= optimized\_rag(question\="cmd+tab does not work on hidden or minimized windows") print(pred.response) pred = optimized\_rag(question="cmd+tab does not work on hidden or minimized windows") print(pred.response) The Command + Tab shortcut on macOS is designed to switch between currently open applications, but it does not directly restore minimized or hidden windows. When you use Command + Tab, it cycles through the applications that are actively running, and minimized windows do not count as active. To manage minimized windows, you can use other shortcuts or methods. For example, you can use Command + Option + H + M to hide all other applications and minimize the most recently used one. Alternatively, you can navigate to the application you want to restore using Command + Tab and then manually click on the minimized window in the Dock to bring it back to focus. You can use `dspy.inspect_history(n=2)` to view the RAG prompt [before optimization](https://gist.github.com/okhat/5d04648f2226e72e66e26a8cb1456ee4) and [after optimization](https://gist.github.com/okhat/79405b8889b4b07da577ee19f1a3479a) . Concretely, in one of the runs of this notebook, the optimized prompt does the following (note that it may be different on a later rerun). 1. Constructs the following instruction, Using the provided \`context\` and \`question\`, analyze the information step by step to generate a comprehensive and informative \`response\`. Ensure that the response clearly explains the concepts involved, highlights key distinctions, and addresses any complexities noted in the context. 2. And includes two fully worked out RAG examples with synthetic reasoning and answers, e.g. `how to transfer whatsapp voice message to computer?`. Let's now evaluate on the overall devset. In \[22\]: Copied! evaluate(optimized\_rag) evaluate(optimized\_rag) Average Metric: 183.32 / 300 (61.1%): 100%|██████████| 300/300 \[00:02<00:00, 104.48it/s\] 2024/11/23 23:17:21 INFO dspy.evaluate.evaluate: Average Metric: 183.3194433591069 / 300 (61.1%) | | question | example\_response | gold\_doc\_ids | reasoning | pred\_response | SemanticF1 | | --- | --- | --- | --- | --- | --- | --- | | 0 | when to use c over c++, and c++ over c? | If you are equally familiar with both C++ and C, it's advisable to... | \[733\] | The context provides insights into the strengths and weaknesses of... | You should consider using C over C++ in scenarios where simplicity... | ✔️ \[0.333\] | | 1 | should images be stored in a git repository? | One viewpoint expresses that there is no significant downside, esp... | \[6253, 6254, 6275, 6278, 8215\] | The context discusses the challenges and considerations of storing... | Storing images in a Git repository is generally considered bad pra... | ✔️ \[0.500\] | ... 298 more rows not displayed ... Out\[22\]: 61.11 Keeping an eye on cost.[¶](#keeping-an-eye-on-cost) ---------------------------------------------------- DSPy allows you to track the cost of your programs, which can be used to monitor the cost of your calls. Here, we'll show you how to track the cost of your programs with DSPy. In \[23\]: Copied! cost \= sum(\[x\['cost'\] for x in lm.history if x\['cost'\] is not None\]) \# in USD, as calculated by LiteLLM for certain providers cost = sum(\[x\['cost'\] for x in lm.history if x\['cost'\] is not None\]) # in USD, as calculated by LiteLLM for certain providers Saving and loading.[¶](#saving-and-loading) -------------------------------------------- The optimized program has a pretty simple structure on the inside. Feel free to explore it. Here, we'll save `optimized_rag` so we can load it again later without having to optimize from scratch. In \[24\]: Copied! optimized\_rag.save("optimized\_rag.json") loaded\_rag \= RAG() loaded\_rag.load("optimized\_rag.json") loaded\_rag(question\="cmd+tab does not work on hidden or minimized windows") optimized\_rag.save("optimized\_rag.json") loaded\_rag = RAG() loaded\_rag.load("optimized\_rag.json") loaded\_rag(question="cmd+tab does not work on hidden or minimized windows") Out\[24\]: Prediction( reasoning='The context explains how the Command + Tab shortcut functions on macOS, particularly in relation to switching between applications. It notes that this shortcut does not bring back minimized or hidden windows directly. Instead, it cycles through applications that are currently open and visible. The information also suggests alternative methods for managing minimized windows and provides insights into how to navigate between applications effectively.', response='The Command + Tab shortcut on macOS is designed to switch between currently open applications, but it does not directly restore minimized or hidden windows. When you use Command + Tab, it cycles through the applications that are actively running, and minimized windows do not count as active. To manage minimized windows, you can use other shortcuts or methods. For example, you can use Command + Option + H + M to hide all other applications and minimize the most recently used one. Alternatively, you can navigate to the application you want to restore using Command + Tab and then manually click on the minimized window in the Dock to bring it back to focus.' ) What's next?[¶](#whats-next) ----------------------------- Improving from around 42% to approximately 61% on this task, in terms of `SemanticF1`, was pretty easy. But DSPy gives you paths to continue iterating on the quality of your system and we have barely scratched the surface. In general, you have the following tools: 1. Explore better system architectures for your program, e.g. what if we ask the LM to generate search queries for the retriever? See, e.g., the [STORM pipeline](https://arxiv.org/abs/2402.14207) built in DSPy. 2. Explore different [prompt optimizers](https://arxiv.org/abs/2406.11695) or [weight optimizers](https://arxiv.org/abs/2407.10930) . See the Optimizers Docs. 3. Scale inference time compute using DSPy Optimizers, e.g. via ensembling multiple post-optimization programs. 4. Cut cost by distilling to a smaller LM, via prompt or weight optimization. How do you decide which ones to proceed with first? The first step is to look at your system outputs, which will allow you to identify the sources of lower performance if any. While doing all of this, make sure you continue to refine your metric, e.g. by optimizing against your judgments, and to collect more (or more realistic) data, e.g. from related domains or from putting a demo of your system in front of users. Back to top --- # YouRM - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/deep-dive/retrieval_models_clients/YouRM.md "Edit this page") YouRM ===== ### Constructor Initialize an instance of the `YouRM` class that calls the [You.com APIs](https://documentation.you.com/api-reference/search) for web-based document retrieval. Options available are the "Search" and "News" APIs. `[](#__codelineno-0-1) YouRM( [](#__codelineno-0-2) ydc_api_key: Optional[str] = None, [](#__codelineno-0-3) k: int = 3, [](#__codelineno-0-4) endpoint: Literal["search", "news"] = "search", [](#__codelineno-0-5) num_web_results: Optional[int] = None, [](#__codelineno-0-6) safesearch: Optional[Literal["off", "moderate", "strict"]] = None, [](#__codelineno-0-7) country: Optional[str] = None, [](#__codelineno-0-8) search_lang: Optional[str] = None, [](#__codelineno-0-9) ui_lang: Optional[str] = None, [](#__codelineno-0-10) spellcheck: Optional[bool] = None, [](#__codelineno-0-11) )` **Parameters:** * `ydc_api_key` (Optional\[str\]): you.com API key, if `YDC_API_KEY` is not set in the environment * `k` (int): If `endpoint="search"`, the max snippets to return per search hit. If `endpoint="news"`, the max articles to return. * `endpoint` (Literal\["search", "news"\]): you.com endpoints * `num_web_results` (Optional\[int\]): The max number of web results to return, must be under 20 * `safesearch` (Optional\[Literal\["off", "moderate", "strict"\]\]): Safesearch settings, one of "off", "moderate", "strict", defaults to moderate * `country` (Optional\[str\]): Country code, ex: 'US' for United States, see API reference for more info * `search_lang` (Optional\[str\]): (News API) Language codes, ex: 'en' for English, see API reference for more info * `ui_lang` (Optional\[str\]): (News API) User interface language for the response, ex: 'en' for English. See API reference for more info * `spellcheck` (Optional\[bool\]): (News API) Whether to spell check query or not, defaults to True ### Methods #### `forward(self, query_or_queries: Union[str, List[str]], k: Optional[int] = None) -> dspy.Prediction` If `endpoint="search"`, search the web for the top `k` snippets matching the given query or queries. If `endpoint="news"`, search the web for the top `k` articles matching the given query or queries. **Parameters:** * `query_or_queries` (_Union\[str, List\[str\]\]_): The query or list of queries to search for. * `k` (_Optional\[int\]_, _optional_): The number of results to retrieve. If not specified, defaults to the value set during initialization. **Returns:** * `dspy.Prediction`: Contains the retrieved passages, each represented as a `dotdict` with schema `[{"long_text": str}]` ### Quickstart Obtain a You.com API key from [https://api.you.com/](https://api.you.com/) . Export this key to an environment variable `YDC_API_KEY`. ``[](#__codelineno-1-1) from dspy.retrieve.you_rm import YouRM [](#__codelineno-1-2) import os [](#__codelineno-1-3) [](#__codelineno-1-4) # The retriever obtains the API key from the `YDC_API_KEY` env var [](#__codelineno-1-5) retriever_model = YouRM(endpoint="search") [](#__codelineno-1-6) [](#__codelineno-1-7) results = retriever_model("Tell me about national parks in the US", k=5) [](#__codelineno-1-8) [](#__codelineno-1-9) for result in results: [](#__codelineno-1-10) print("Document:", result.long_text, "\n")`` Back to top --- # Saving and Loading - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/tutorials/saving/index.md "Edit this page") Tutorial: Saving and Loading your DSPy program ============================================== This guide demonstrates how to save and load your DSPy program. At a high level, there are two ways to save your DSPy program: 1. Save the state of the program only, similar to weights-only saving in PyTorch. 2. Save the whole program, including both the architecture and the state, which is supported by `dspy>=2.6.0`. State-only Saving ----------------- State represents the DSPy program's internal state, including the signature, demos (few-shot examples), and other informaiton like the `lm` to use for each `dspy.Predict` in the program. It also includes configurable attributes of other DSPy modules like `k` for `dspy.retrievers.Retriever`. To save the state of a program, use the `save` method and set `save_program=False`. You can choose to save the state to a JSON file or a pickle file. We recommend saving the state to a JSON file because it is safer and readable. But sometimes your program contains non-serializable objects like `dspy.Image` or `datetime.datetime`, in which case you should save the state to a pickle file. Let's say we have compiled a program with some data, and we want to save the program for future usage: `[](#__codelineno-0-1) import dspy [](#__codelineno-0-2) from dspy.datasets.gsm8k import GSM8K, gsm8k_metric [](#__codelineno-0-3) [](#__codelineno-0-4) dspy.settings.configure(lm=dspy.LM("openai/gpt-4o-mini")) [](#__codelineno-0-5) [](#__codelineno-0-6) gsm8k = GSM8K() [](#__codelineno-0-7) gsm8k_trainset = gsm8k.train[:10] [](#__codelineno-0-8) dspy_program = dspy.ChainOfThought("question -> answer") [](#__codelineno-0-9) [](#__codelineno-0-10) optimizer = dspy.BootstrapFewShot(metric=gsm8k_metric, max_bootstrapped_demos=4, max_labeled_demos=4, max_rounds=5) [](#__codelineno-0-11) compiled_dspy_program = optimizer.compile(dspy_program, trainset=gsm8k_trainset)` To save the state of your program to json file: `[](#__codelineno-1-1) compiled_dspy_program.save("./dspy_program/program.json", save_program=False)` To save the state of your program to a pickle file: `[](#__codelineno-2-1) compiled_dspy_program.save("./dspy_program/program.pkl", save_program=False)` To load your saved state, you need to **recreate the same program**, then load the state using the `load` method. `[](#__codelineno-3-1) loaded_dspy_program = dspy.ChainOfThought("question -> answer") # Recreate the same program. [](#__codelineno-3-2) loaded_dspy_program.load("./dspy_program/program.json") [](#__codelineno-3-3) [](#__codelineno-3-4) assert len(compiled_dspy_program.demos) == len(loaded_dspy_program.demos) [](#__codelineno-3-5) for original_demo, loaded_demo in zip(compiled_dspy_program.demos, loaded_dspy_program.demos): [](#__codelineno-3-6) # Loaded demo is a dict, while the original demo is a dspy.Example. [](#__codelineno-3-7) assert original_demo.toDict() == loaded_demo [](#__codelineno-3-8) assert str(compiled_dspy_program.signature) == str(loaded_dspy_program.signature)` Or load the state from a pickle file: `[](#__codelineno-4-1) loaded_dspy_program = dspy.ChainOfThought("question -> answer") # Recreate the same program. [](#__codelineno-4-2) loaded_dspy_program.load("./dspy_program/program.pkl") [](#__codelineno-4-3) [](#__codelineno-4-4) assert len(compiled_dspy_program.demos) == len(loaded_dspy_program.demos) [](#__codelineno-4-5) for original_demo, loaded_demo in zip(compiled_dspy_program.demos, loaded_dspy_program.demos): [](#__codelineno-4-6) # Loaded demo is a dict, while the original demo is a dspy.Example. [](#__codelineno-4-7) assert original_demo.toDict() == loaded_demo [](#__codelineno-4-8) assert str(compiled_dspy_program.signature) == str(loaded_dspy_program.signature)` Whole Program Saving -------------------- Starting from `dspy>=2.6.0`, DSPy supports saving the whole program, including the architecture and the state. This feature is powered by `cloudpickle`, which is a library for serializing and deserializing Python objects. To save the whole program, use the `save` method and set `save_program=True`, and specify a directory path to save the program instead of a file name. We require a directory path because we also save some metadata, e.g., the dependency versions along with the program itself. `[](#__codelineno-5-1) compiled_dspy_program.save("./dspy_program/", save_program=True)` To load the saved program, directly use `dspy.load` method: `[](#__codelineno-6-1) loaded_dspy_program = dspy.load("./dspy_program/") [](#__codelineno-6-2) [](#__codelineno-6-3) assert len(compiled_dspy_program.demos) == len(loaded_dspy_program.demos) [](#__codelineno-6-4) for original_demo, loaded_demo in zip(compiled_dspy_program.demos, loaded_dspy_program.demos): [](#__codelineno-6-5) # Loaded demo is a dict, while the original demo is a dspy.Example. [](#__codelineno-6-6) assert original_demo.toDict() == loaded_demo [](#__codelineno-6-7) assert str(compiled_dspy_program.signature) == str(loaded_dspy_program.signature)` With whole program saving, you don't need to recreate the program, but can directly load the architecture along with the state. You can pick the suitable saviing approach based on your needs. Backward Compatibility ---------------------- As of `dspy<2.7`, we don't guarantee the backward compatibility of the saved program. For example, if you save the program with `dspy==2.5.35`, at loading time please make sure to use the same version of DSPy to load the program, otherwise the program may not work as expected. Chances are that loading a saved file in a different version of DSPy will not raise an error, but the performance could be different from when the program was saved. Starting from `dspy>=2.7`, we will guarantee the backward compatibility of the saved program in major releases, i.e., programs saved in `dspy==2.7.0` should be loadeable in `dspy==2.7.10`. Back to top --- # Agents - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/tutorials/agents/index.ipynb "Edit this page") Tutorial: Agents[¶](#tutorial-agents) ====================================== Let's walk through a quick example of setting up a `dspy.ReAct` agent with a couple of tools and optimizing it to conduct advanced browsing for multi-hop search. Install the latest DSPy via `pip install -U dspy` and follow along. In this tutorial, we'll use an extremely small LM, Meta's `Llama-3.2-3B-Instruct` which has 3 billion parameters. A model like this is not very reliable out of the box for long or complex agent loops. However, it's extremely fast and cheap to host, as it needs very little RAM. You might be able to host the 3B model on your laptop with Ollama, on your GPU server with SGLang, or via a provider that hosts it for you like Databricks or Together. In the snippet below, we'll configure our main LM as `Llama-3.2-3B`. We'll also set up a larger LM, i.e. `GPT-4o`, as a teacher that we'll invoke a very small number of times to help teach the small LM. In \[1\]: Copied! import dspy llama3b \= dspy.LM('/Llama-3.2-3B-Instruct', temperature\=0.7) gpt4o \= dspy.LM('openai/gpt-4o', temperature\=0.7) dspy.configure(lm\=llama3b) import dspy llama3b = dspy.LM('/Llama-3.2-3B-Instruct', temperature=0.7) gpt4o = dspy.LM('openai/gpt-4o', temperature=0.7) dspy.configure(lm=llama3b) Let's load a dataset for our task. We'll load examples from the HoVer multi-hop task, where the input is a (really!) complex claim and the output we're seeking is the set of Wikipedia pages that are required to fact-check that claim. In \[2\]: Copied! import random from dspy.datasets import DataLoader kwargs \= dict(fields\=("claim", "supporting\_facts", "hpqa\_id", "num\_hops"), input\_keys\=("claim",)) hover \= DataLoader().from\_huggingface(dataset\_name\="hover-nlp/hover", split\="train", trust\_remote\_code\=True, \*\*kwargs) hpqa\_ids \= set() hover \= \[\ dspy.Example(claim\=x.claim, titles\=list(set(\[y\["key"\] for y in x.supporting\_facts\]))).with\_inputs("claim")\ for x in hover\ if x\["num\_hops"\] \== 3 and x\["hpqa\_id"\] not in hpqa\_ids and not hpqa\_ids.add(x\["hpqa\_id"\])\ \] random.Random(0).shuffle(hover) trainset, devset, testset \= hover\[:100\], hover\[100:200\], hover\[650:\] import random from dspy.datasets import DataLoader kwargs = dict(fields=("claim", "supporting\_facts", "hpqa\_id", "num\_hops"), input\_keys=("claim",)) hover = DataLoader().from\_huggingface(dataset\_name="hover-nlp/hover", split="train", trust\_remote\_code=True, \*\*kwargs) hpqa\_ids = set() hover = \[ dspy.Example(claim=x.claim, titles=list(set(\[y\["key"\] for y in x.supporting\_facts\]))).with\_inputs("claim") for x in hover if x\["num\_hops"\] == 3 and x\["hpqa\_id"\] not in hpqa\_ids and not hpqa\_ids.add(x\["hpqa\_id"\]) \] random.Random(0).shuffle(hover) trainset, devset, testset = hover\[:100\], hover\[100:200\], hover\[650:\] Let's view an example of this task: In \[3\]: Copied! example \= trainset\[0\] print("Claim:", example.claim) print("Pages that must be retrieved:", example.titles) example = trainset\[0\] print("Claim:", example.claim) print("Pages that must be retrieved:", example.titles) Claim: This director is known for his work on Miss Potter. The Academy of Motion Picture Arts and Sciences presents the award in which he was nominated for his work in "Babe". Pages that must be retrieved: \['Miss Potter', 'Chris Noonan', 'Academy Award for Best Director'\] Now, let's define a function to do the search in Wikipedia. We'll rely on a ColBERTv2 server that can search the "abstracts" (i.e., first paragraphs) of every article that existed in Wikipedia in 2017, which is the data used in HoVer. In \[4\]: Copied! DOCS \= {} def search(query: str, k: int) \-> list\[str\]: results \= dspy.ColBERTv2(url\='http://20.102.90.50:2017/wiki17\_abstracts')(query, k\=k) results \= \[x\['text'\] for x in results\] for result in results: title, text \= result.split(" | ", 1) DOCS\[title\] \= text return results DOCS = {} def search(query: str, k: int) -> list\[str\]: results = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17\_abstracts')(query, k=k) results = \[x\['text'\] for x in results\] for result in results: title, text = result.split(" | ", 1) DOCS\[title\] = text return results Now, let's use the `search` function to define two tools for our ReAct agent: In \[5\]: Copied! def search\_wikipedia(query: str) \-> list\[str\]: """Returns top-5 results and then the titles of the top-5 to top-30 results.""" topK \= search(query, 30) titles, topK \= \[f"\`{x.split(' | ')\[0\]}\`" for x in topK\[5:30\]\], topK\[:5\] return topK + \[f"Other retrieved pages have titles: {', '.join(titles)}."\] def lookup\_wikipedia(title: str) \-> str: """Returns the text of the Wikipedia page, if it exists.""" if title in DOCS: return DOCS\[title\] results \= \[x for x in search(title, 10) if x.startswith(title + " | ")\] if not results: return f"No Wikipedia page found for title: {title}" return results\[0\] def search\_wikipedia(query: str) -> list\[str\]: """Returns top-5 results and then the titles of the top-5 to top-30 results.""" topK = search(query, 30) titles, topK = \[f"\`{x.split(' | ')\[0\]}\`" for x in topK\[5:30\]\], topK\[:5\] return topK + \[f"Other retrieved pages have titles: {', '.join(titles)}."\] def lookup\_wikipedia(title: str) -> str: """Returns the text of the Wikipedia page, if it exists.""" if title in DOCS: return DOCS\[title\] results = \[x for x in search(title, 10) if x.startswith(title + " | ")\] if not results: return f"No Wikipedia page found for title: {title}" return results\[0\] Now, let's define the ReAct agent in DSPy. It's going to be super simple: it'll take a `claim` and produce a list `titles: list[str]`. We'll instruct it to find all Wikipedia titles that are needed to fact-check the claim. In \[6\]: Copied! instructions \= "Find all Wikipedia titles relevant to verifying (or refuting) the claim." signature \= dspy.Signature("claim -> titles: list\[str\]", instructions) react \= dspy.ReAct(signature, tools\=\[search\_wikipedia, lookup\_wikipedia\], max\_iters\=20) instructions = "Find all Wikipedia titles relevant to verifying (or refuting) the claim." signature = dspy.Signature("claim -> titles: list\[str\]", instructions) react = dspy.ReAct(signature, tools=\[search\_wikipedia, lookup\_wikipedia\], max\_iters=20) Let's try it with a really simple claim to see if our tiny 3B model can do it! In \[7\]: Copied! react(claim\="David Gregory was born in 1625.").titles\[:3\] react(claim="David Gregory was born in 1625.").titles\[:3\] Out\[7\]: \['David Gregory (physician)', 'David A. Gregory', 'David Harry Gregory'\] Great. Now let's set up an evaluation metric, `top5_recall`. It will return the fraction of the gold pages (which are always 3) that are retrieved in the top-5 titles returned by the agent. In \[9\]: Copied! def top5\_recall(example, pred, trace\=None): gold\_titles \= example.titles recall \= sum(x in pred.titles\[:5\] for x in gold\_titles) / len(gold\_titles) \# If we're "bootstrapping" for optimization, return True if and only if the recall is perfect. if trace is not None: return recall \>= 1.0 \# If we're just doing inference, just measure the recall. return recall evaluate \= dspy.Evaluate(devset\=devset, metric\=top5\_recall, num\_threads\=16, display\_progress\=True, display\_table\=5) def top5\_recall(example, pred, trace=None): gold\_titles = example.titles recall = sum(x in pred.titles\[:5\] for x in gold\_titles) / len(gold\_titles) # If we're "bootstrapping" for optimization, return True if and only if the recall is perfect. if trace is not None: return recall >= 1.0 # If we're just doing inference, just measure the recall. return recall evaluate = dspy.Evaluate(devset=devset, metric=top5\_recall, num\_threads=16, display\_progress=True, display\_table=5) Let's evaluate our off-the-shelf agent, with `Llama-3.2-8B`, to see how far we can go already. This model is tiny, so it can complain fairly often. Let's wrap it in a try/except block to hide those. In \[10\]: Copied! def safe\_react(claim: str): try: return react(claim\=claim) except Exception as e: return dspy.Prediction(titles\=\[\]) evaluate(safe\_react) def safe\_react(claim: str): try: return react(claim=claim) except Exception as e: return dspy.Prediction(titles=\[\]) evaluate(safe\_react) 0%| | 0/100 \[00:00Returns top-5 results and then the titles of the top-5 to top-30 results.. It takes arguments {'query': 'str'} in JSON format.\ (2) lookup\_wikipedia, whose description is Returns the text of the Wikipedia page, if it exists.. It takes arguments {'title': 'str'} in JSON format.\ (3) finish, whose description is Signals that the final outputs, i.e. \`titles\`, are now available and marks the task as complete.. It takes arguments {} in JSON format.\ \ User message:\ \ \[\[ ## claim ## \]\]\ 1990 Film that Khiladiyon Ka Khiladi is loosely based on stars this actor who is best known for martial arts action films.\ \ \[\[ ## trajectory ## \]\]\ \[\[ ## thought\_0 ## \]\]\ To verify the claim, I need to identify the 1990 film that "Khiladiyon Ka Khiladi" is loosely based on and the actor known for martial arts action films who starred in it. I will start by searching for information on "Khiladiyon Ka Khiladi" to find details about its inspiration.\ \ \[\[ ## tool\_name\_0 ## \]\]\ search\_wikipedia\ \ \[\[ ## tool\_args\_0 ## \]\]\ {"query": "Khiladiyon Ka Khiladi"}\ \ \[\[ ## observation\_0 ## \]\]\ \[1\] «Khiladiyon Ka Khiladi | Khiladiyon Ka Khiladi (English: Player of Players) is a 1996 Indian action film starring Rekha in her first villain role, Akshay Kumar, Raveena Tandon and former WWE wrestlers "Crush" and Brian Lee as "The Undertaker". It was the 5th highest grossing movie of the year 1996 and was declared 'SuperHit' by Box Office India. It was the fourth installment in the Khiladi (film series). The movie is loosely based based on Hollywood film Lionheart»\ \[2\] «Khiladi 420 | Khiladi 420 (English: "Con Player") is an Indian Hindi action film directed by Neeraj Vora and starring Akshay Kumar and Mahima Chaudhry. The film was written by Uttam Gudda and released on 29 December 2000. It is the seventh installment in the "Khiladi" series starring Kumar, which included "Khiladi" (1992), "Main Khiladi Tu Anari" (1994), "Sabse Bada Khiladi" (1995), "Khiladiyon Ka Khiladi" (1996), "Mr. and Mrs. Khiladi" (1997) and "International Khiladi" (1999).»\ \[3\] «Khiladi (1992 film) | Khiladi (English: "Player" ) is a 1992 Indian suspense thriller film directed by Abbas Mustan. The film was Akshay Kumar's breakthrough role and also stars Ayesha Jhulka, Deepak Tijori, Sabeeha. While Prem Chopra, Shakti Kapoor, Anant Mahadevan and Johnny Lever played supporting roles. "Khiladi" was the first installment in the Khiladi (film series) which had "Khiladi" in the title and Akshay Kumar in the leading role. It was followed by "Main Khiladi Tu Anari" (1994), "Sabse Bada Khiladi" (1995), "Khiladiyon Ka Khiladi" (1996), "Mr. and Mrs. Khiladi" (1997), "International Khiladi" (1999), "Khiladi 420"(2000) and "Khiladi 786" (2012). Khiladi was critically and commercially success at the box-office and the tenth highest grossing film of 1992. It was Akshay Kumar's first successful movie and was declared a "Super Hit" at the box office. The basic premise of the story is similar to 1975 released movie Khel Khel Mein starring Rishi Kapoor and Neetu Singh. The film was remade in Kannada as "Aata Hudugaata".»\ \[4\] «Khiladi (film series) | Khiladi series is a Bollywood action film series starring Akshay Kumar in the lead role. However, unlike other film series, other than having Akshay Kumar in lead role, and other than having the word "Khiladi" in the title, these films have nothing in common. The producers, directors and stories of these films are totally different. " Khiladi" (1992) was the first in a series of films which had Akshay Kumar in the title role and gave it his first breakthrough role. It was followed by "Main Khiladi Tu Anari" (1994), "Sabse Bada Khiladi" (1995), "Khiladiyon Ka Khiladi" (1996), "Mr. and Mrs. Khiladi" (1997), "International Khiladi" (1999) and "Khiladi 420" (2000), all featuring Kumar in the lead role. The latest film in the franchise is "Khiladi 786" (2012).»\ \[5\] «Khiladi 786 | Khiladi 786 (खिलाड़ी 786) is a 2012 Indian Hindi Punjabi action comedy film directed by Ashish R Mohan, featuring Akshay Kumar in the title role alongside Asin playing the female lead. It features Himesh Reshammiya, Mithun Chakraborty, Raj Babbar and Mukesh Rishi in supporting roles. The film marks the return of Akshay Kumar to his "Khiladi" series after 12 years. It is mostly shot in Mumbai and Punjab. Khiladi786 is AkshayKumar 100th film.»\ \[6\] «Other retrieved pages have titles: \`Khiladi (2013 film)\`, \`Khiladi (1968 film)\`, \`International Khiladi\`, \`Khatron Ke Khiladi (1988 film)\`, \`Khatron Ke Khiladi (2001 film)\`, \`Umesh Mehra\`, \`Shatranj ke Khiladi\`, \`Main Khiladi Tu Anari\`, \`Fear Factor: Khatron Ke Khiladi (season 7)\`, \`Fear Factor: Khatron Ke Khiladi\`, \`Mr. and Mrs. Khiladi\`, \`Sabse Bada Khiladi\`, \`Ek Khiladi Bawan Pattey\`, \`Ek Khiladi Ek Haseena (TV series)\`, \`Do Khiladi\`, \`Ek Khiladi Ek Haseena (film)\`, \`Khel Khilari Ka\`, \`Fear Factor: Khatron Ke Khiladi (season 8)\`, \`Fear Factor: Khatron Ke Khiladi Darr Ka Blockbuster\`, \`Khilona (1942 film)\`, \`Fear Factor: Khatron Ke Khiladi Darr Ka Blockbuster Returns\`, \`Chhota Bheem: Dholakpur ke Khiladi\`, \`Khilani\`, \`Kandireega\`, \`Khilaaf\`.»\ \ \[\[ ## thought\_1 ## \]\]\ The search results indicate that "Khiladiyon Ka Khiladi" is loosely based on the Hollywood film "Lionheart". I should now verify which actor, known for martial arts action films, starred in "Lionheart".\ \ \[\[ ## tool\_name\_1 ## \]\]\ search\_wikipedia\ \ \[\[ ## tool\_args\_1 ## \]\]\ {"query": "Lionheart 1990 film"}\ \ \[\[ ## observation\_1 ## \]\]\ \[1\] «Lionheart (1990 film) | Lionheart (also known as Wrong Bet, A.W.O.L.: Absent Without Leave, Leon and Full Contact) is a 1990 action film, directed by Sheldon Lettich, starring Jean-Claude Van Damme and co-starring Brian Thompson, along with Harrison Page, Deborah Rennard, Lisa Pelikan, and Ashley Johnson.»\ \[2\] «Truly, Madly, Deeply | Truly, Madly, Deeply is a 1990 British fantasy drama film made for the BBC's "Screen Two" series, by BBC Films, Lionheart and Winston Pictures. The film, written and directed by Anthony Minghella, stars Juliet Stevenson and Alan Rickman.»\ \[3\] «Lionheart (1987 film) | Lionheart, also known as Lionheart: The Children's Crusade, is a 1987 adventure film directed by Franklin J. Schaffner and produced by Talia Shire and Stanley O'Toole. Shire's brother, Francis Ford Coppola, initially planned to direct the film but instead opted to be executive producer along with Shire's husband, Jack Schwartzman. The screenplay was written by Menno Meyjes and Richard Outten from a story by Meyjes. The composer Jerry Goldsmith wrote the score. The film was released in August 1987. It was distributed by Orion Pictures.»\ \[4\] «Lionheart (2016 film) | Lionheart is a 2016 American boxing film short written and produced by Oscar DeRosa and Orlando Cicilia III. The film stars Oscar DeRosa and Marc Macaulay. The film portrays struggling professional boxer Max Rossi who is finally presented with the fight he's been waiting for that will launch his career to the next level but when he is suddenly confronted with a life-changing opportunity he must decide which path to take.»\ \[5\] «Richard the Lionheart (2013 film) | Richard The Lionheart is a 2013 film, starring Chandler Maness as Richard the Lionheart, Malcolm McDowell as King Henry II and Andrea Zirio as Henry the Young.»\ \[6\] «Other retrieved pages have titles: \`Richard the Lionheart: Rebellion\`, \`The Brothers Lionheart (1977 film)\`, \`Kodama Simham\`, \`Jean-Claude Van Damme\`, \`Lionsheart\`, \`A Day for Lionhearts\`, \`The Lion in Winter (1968 film)\`, \`Lion Jagapathi Rao\`, \`Theatre of Blood\`, \`Young Lions (album)\`, \`Richard the Lion-Hearted (1923 film)\`, \`Bullseye! (1990 film)\`, \`Fear (1990 film)\`, \`The Guard (1990 film)\`, \`Ironheart\`, \`The Guardian (1990 film)\`, \`Tiger Cage 2\`, \`Lion Rampant (game publisher)\`, \`Success (1991 film)\`, \`Lion in the Streets\`, \`The King of the Kickboxers\`, \`Lionheart (video game)\`, \`Richard the Lionheart (TV series)\`, \`Dermot Walsh\`, \`Lars Söderdahl\`.»\ \ Respond with the corresponding output fields, starting with the field \`\[\[ ## next\_thought ## \]\]\`, then \`\[\[ ## next\_tool\_name ## \]\]\` (must be formatted as a valid Python Literal\[search\_wikipedia, lookup\_wikipedia, finish\]), then \`\[\[ ## next\_tool\_args ## \]\]\` (must be formatted as a valid Python dict\[str, Any\]), and then ending with the marker for \`\[\[ ## completed ## \]\]\`.\ \ Assistant message:\ \ \[\[ ## next\_thought ## \]\]\ The search results confirm that the 1990 film "Lionheart" stars Jean-Claude Van Damme, who is well-known for his martial arts action films. This information aligns with what is needed to verify the claim. I now have the relevant titles related to the claim: "Khiladiyon Ka Khiladi" and "Lionheart."\ \ \[\[ ## next\_tool\_name ## \]\]\ finish\ \ \[\[ ## next\_tool\_args ## \]\]\ {}\ \ \[\[ ## completed ## \]\]\ \ User message:\ \ \[\[ ## claim ## \]\]\ The figure who currently holds the role Michael Noonan held from 2001 to 2002, was the Minister for Social Protection from 2016 until 2017.\ \ \[\[ ## trajectory ## \]\]\ \ \ Respond with the corresponding output fields, starting with the field \`\[\[ ## next\_thought ## \]\]\`, then \`\[\[ ## next\_tool\_name ## \]\]\` (must be formatted as a valid Python Literal\[search\_wikipedia, lookup\_wikipedia, finish\]), then \`\[\[ ## next\_tool\_args ## \]\]\` (must be formatted as a valid Python dict\[str, Any\]), and then ending with the marker for \`\[\[ ## completed ## \]\]\`.\ \ Assistant message:\ \ \[\[ ## next\_thought ## \]\]\ To verify the claim, I need to identify the role Michael Noonan held from 2001 to 2002. Then, I can find out who currently holds this role and check if they were the Minister for Social Protection from 2016 to 2017.\ \ \[\[ ## next\_tool\_name ## \]\]\ search\_wikipedia\ \ \[\[ ## next\_tool\_args ## \]\]\ {"query": "Michael Noonan role 2001 to 2002"}\ \ \[\[ ## completed ## \]\]\ \ User message:\ \ \[\[ ## claim ## \]\]\ An actor in a musical film was also a United States Navy Combat veteran in World War II. This film was awarded The Golden Bear at the 50th Berlin International Film Festival.\ \ \[\[ ## trajectory ## \]\]\ \[\[ ## thought\_0 ## \]\]\ To verify the claim, I need to find out which musical film was awarded The Golden Bear at the 50th Berlin International Film Festival and check if an actor from that film was a United States Navy Combat veteran in World War II. I will start by searching for the musical films that won the Golden Bear around the time of the 50th Berlin International Film Festival.\ \ \[\[ ## tool\_name\_0 ## \]\]\ search\_wikipedia\ \ \[\[ ## tool\_args\_0 ## \]\]\ {"query": "Golden Bear 50th Berlin International Film Festival musical film"}\ \ \[\[ ## observation\_0 ## \]\]\ \[1\] «53rd Berlin International Film Festival | The 54th annual Berlin International Film Festival was held from February 6–16, 2003. The festival opened with musical film "Chicago" by Rob Marshall and closed with Martin Scorsese's "Gangs of New York", both films played out of competition at the festival. The Golden Bear was awarded to British film "In This World" directed by Michael Winterbottom.»\ \[2\] «50th Berlin International Film Festival | The 50th annual Berlin International Film Festival was held from February 9 to 20, 2000. The festival opened with "The Million Dollar Hotel" by Wim Wenders. " Bossa Nova" by Bruno Barreto, screened out of competition was the closing film of the festival. The Golden Bear was awarded to American film "Magnolia" directed by Paul Thomas Anderson.»\ \[3\] «40th Berlin International Film Festival | The 40th annual Berlin International Film Festival was held from 9 to 20 February 1990. The festival opened with "Steel Magnolias" by Herbert Ross, which was shown out of competition. The Golden Bear was awarded to the American film "Music Box" directed by Costa-Gavras and Czech film "Skřivánci na niti" directed by Jiří Menzel.»\ \[4\] «66th Berlin International Film Festival | The 66th Berlin International Film Festival was held from 11 to 21 February 2016, with American actress Meryl Streep as the President of the Jury. The Honorary Golden Bear for lifetime achievement was presented to German cinematographer Michael Ballhaus. " Hail, Caesar! ", directed by Joel and Ethan Coen, was selected to open the festival. The Golden Bear was awarded to the Italian documentary "Fire at Sea", directed by Gianfranco Rosi, which also serves as closing night film.»\ \[5\] «49th Berlin International Film Festival | The 49th annual Berlin International Film Festival was held from February 10 to 21, 1999. The festival opened with "Aimée & Jaguar" by Max Färberböck. The Golden Bear was awarded to Canadian-American film "The Thin Red Line" directed by Terrence Malick.»\ \[6\] «Other retrieved pages have titles: \`38th Berlin International Film Festival\`, \`48th Berlin International Film Festival\`, \`45th Berlin International Film Festival\`, \`47th Berlin International Film Festival\`, \`64th Berlin International Film Festival\`, \`65th Berlin International Film Festival\`, \`46th Berlin International Film Festival\`, \`54th Berlin International Film Festival\`, \`62nd Berlin International Film Festival\`, \`25th Berlin International Film Festival\`, \`35th Berlin International Film Festival\`, \`43rd Berlin International Film Festival\`, \`67th Berlin International Film Festival\`, \`Golden Bear\`, \`44th Berlin International Film Festival\`, \`60th Berlin International Film Festival\`, \`13th Berlin International Film Festival\`, \`34th Berlin International Film Festival\`, \`51st Berlin International Film Festival\`, \`1st Berlin International Film Festival\`, \`30th Berlin International Film Festival\`, \`42nd Berlin International Film Festival\`, \`27th Berlin International Film Festival\`, \`31st Berlin International Film Festival\`, \`32nd Berlin International Film Festival\`.»\ \ \[\[ ## thought\_1 ## \]\]\ The Golden Bear at the 50th Berlin International Film Festival was awarded to the American film "Magnolia" directed by Paul Thomas Anderson. I need to verify if any actor in "Magnolia" was a United States Navy Combat veteran in World War II.\ \ \[\[ ## tool\_name\_1 ## \]\]\ search\_wikipedia\ \ \[\[ ## tool\_args\_1 ## \]\]\ {"query": "Magnolia film cast"}\ \ \[\[ ## observation\_1 ## \]\]\ \[1\] «Magnolia (film) | Magnolia is a 1999 American ensemble drama film written, co-produced and directed by Paul Thomas Anderson. The film stars Jeremy Blackman, Tom Cruise, Melinda Dillon, Philip Baker Hall, Philip Seymour Hoffman, Ricky Jay, William H. Macy, Alfred Molina, Julianne Moore, John C. Reilly, Jason Robards and Melora Walters, and is a mosaic of interrelated characters in search of happiness, forgiveness and meaning in the San Fernando Valley.»\ \[2\] «Steel Magnolias (2012 film) | Steel Magnolias is an American comedy-drama television film directed by Kenny Leon that premiered at Lifetime Network on October 7, 2012. It is a contemporary retelling of the play "Steel Magnolias" and its 1989 film adaptation. The new film stars an all-Black American cast, including Queen Latifah as M'Lynn, Jill Scott as Truvy, Condola Rashād as Shelby, Adepero Oduye as Annelle, with Phylicia Rashād as Clairee and Alfre Woodard as Ouiser.»\ \[3\] «Baller Blockin' | Baller Blockin' is a 2000 drama film set in New Orleans's Magnolia Projects. It stars Bryan “Birdman” Williams, Ronald "Slim" Williams, Juvenile, Manuel Vazquez, B.G., Lil Wayne, Turk, and Mannie Fresh with cameos by comedians Anthony Johnson and T.K. Kirkland.»\ \[4\] «360 (film) | 360 is a 2011 ensemble drama film starring Anthony Hopkins, Ben Foster, Rachel Weisz, Jude Law and other international actors. The film, directed by Fernando Meirelles, opened the 2011 London Film Festival. Magnolia Pictures released the film on video on demand on 29 June 2012 and was released in United States theaters on 3 August 2012.»\ \[5\] «River of Romance | River of Romance is a 1929 American drama film directed by Richard Wallace and written by Ethel Doherty, Joseph L. Mankiewicz, Dan Totheroh and John V.A. Weaver. The film stars Charles 'Buddy' Rogers, Mary Brian, June Collyer, Henry B. Walthall, Wallace Beery, Fred Kohler and Natalie Kingston. It is based on the play "Magnolia" by Booth Tarkington. The film was released on June 29, 1929, by Paramount Pictures.»\ \[6\] «Other retrieved pages have titles: \`Ira & Abby\`, \`The Seat Filler\`, \`Serious Moonlight (2009 film)\`, \`Experimenter (film)\`, \`Humpday\`, \`Roadie (2011 film)\`, \`A Kind of Murder\`, \`The Magic of Belle Isle\`, \`The Hero of Color City\`, \`The War Within (film)\`, \`Claudia Lee\`, \`Steel Magnolias\`, \`Dina Bonnevie\`, \`Deadline (1987 film)\`, \`Creative Control (film)\`, \`Humboldt County (film)\`, \`Temptation Island (1980 film)\`, \`Gina Beck\`, \`Magnolia (score)\`, \`Cashback (film)\`, \`Cynthia Zamora\`, \`Straight Talk\`, \`Magnolia (soundtrack)\`, \`The Great Buck Howard\`, \`Binta and the Great Idea\`.»\ \ \[\[ ## thought\_2 ## \]\]\ The cast of the film "Magnolia" includes several well-known actors. I need to investigate whether any of these actors, such as Tom Cruise, Philip Baker Hall, or Jason Robards, were United States Navy Combat veterans in World War II.\ \ \[\[ ## tool\_name\_2 ## \]\]\ search\_wikipedia\ \ \[\[ ## tool\_args\_2 ## \]\]\ {"query": "Jason Robards military service"}\ \ \[\[ ## observation\_2 ## \]\]\ \[1\] «Jason Robards | Jason Nelson Robards Jr. (July 26, 1922 – December 26, 2000) was an American stage, film, and television actor. He was a winner of the Tony Award, two Academy Awards and an Emmy Award. He was also a United States Navy combat veteran of World War II.»\ \[2\] «Jason Robards Sr. | Jason Nelson Robards Sr. (December 31, 1892 – April 4, 1963) was an American stage and screen actor, and the father of Oscar-winning actor Jason Robards Jr. Robards appeared in many films, initially as a leading man, then in character roles and occasional bits. Most of his final roles were in television.»\ \[3\] «Jason Wingard | Jason Wingard, Ph.D. (born December 13, 1971) is Dean and Professor of the School of Professional Studies (SPS) at Columbia University. He is a leading academic and executive in the areas of leadership development, professional learning, and human capital management. As part of this work, he has 1) authored several books; 2) provided keynote, speaking, and consulting services to global Fortune 500 organizations, professional events, and conferences; 3) served as a board member and facilitator; and 4) held senior executive roles in corporate, academic, and non-profit organizations.»\ \[4\] «Jason Torpy | Jason Torpy is president of the Military Association of Atheists and Freethinkers (MAAF), an advocacy group focused on non-religious service members and veterans. He is a veteran of the U.S. Army and a Humanist Celebrant who works to increase the visibility of "atheists in foxholes".»\ \[5\] «Jason Dunkerley | Jason Joseph Dunkerley (born August 21, 1977, Newtownards, Northern Ireland) is a Canadian Paralympian athlete competing mainly in category T11 and T12 middle distance events.»\ \[6\] «Other retrieved pages have titles: \`Jase Daniels\`, \`Jason Dunham\`, \`Jason Schauble\`, \`Frederick C. Billard\`, \`Wilfred C. Menard, Jr.\`, \`Jason Millard\`, \`George C. Rickards\`, \`Jason Amerine\`, \`USS Jason (AR-8)\`, \`Rob (dog)\`, \`USS Jason Dunham\`, \`Robert Lee Bullard\`, \`Jason Chee (para-athlete)\`, \`The War Between Men and Women\`, \`Thomas Byard\`, \`Ralph Baker (general)\`, \`Jason Lamont\`, \`Jason Rother incident\`, \`William H. G. Bullard\`, \`Sidney Gerald Burrard\`, \`Jason Gilardi\`, \`Reunion (1989 film)\`, \`JASON (advisory group)\`, \`The Great American West\`, \`Samuel Robison\`.»\ \ Respond with the corresponding output fields, starting with the field \`\[\[ ## next\_thought ## \]\]\`, then \`\[\[ ## next\_tool\_name ## \]\]\` (must be formatted as a valid Python Literal\[search\_wikipedia, lookup\_wikipedia, finish\]), then \`\[\[ ## next\_tool\_args ## \]\]\` (must be formatted as a valid Python dict\[str, Any\]), and then ending with the marker for \`\[\[ ## completed ## \]\]\`.\ \ Assistant message:\ \ \[\[ ## next\_thought ## \]\]\ I have confirmed that Jason Robards, an actor in the film "Magnolia," was a United States Navy combat veteran of World War II. Since "Magnolia" was awarded The Golden Bear at the 50th Berlin International Film Festival, this satisfies the claim. I can now finish with the relevant Wikipedia titles.\ \ \[\[ ## next\_tool\_name ## \]\]\ finish\ \ \[\[ ## next\_tool\_args ## \]\]\ {}\ \ \[\[ ## completed ## \]\]\ \ User message:\ \ \[\[ ## claim ## \]\]\ The author of the 1960s unproduced script written for The Beatles, Up Against It, and Bernard-Marie Koltès are both playwrights.\ \ \[\[ ## trajectory ## \]\]\ \[\[ ## thought\_0 ## \]\]\ To verify the claim, I need to identify the playwrights who wrote the unproduced script for "Up Against It" and "The Beatles".\ \ \[\[ ## tool\_name\_0 ## \]\]\ search\_wikipedia\ \ \[\[ ## tool\_args\_0 ## \]\]\ {"query": "Up Against It script writer Bernard-Marie Koltès"}\ \ \[\[ ## observation\_0 ## \]\]\ \[1\] «Bernard-Marie Koltès | Bernard-Marie Koltès (\] ; 9 April 1948 – 15 April 1989) was a French playwright and theatre director best known for his plays "La Nuit juste avant les Forêts" ("The Night Just Before the Forests", 1976), "Sallinger" (1977) and "Dans la Solitude des Champs de Coton" ("In the Solitude of Cotton Fields", 1986).»\ \[2\] «In the Solitude of Cotton Fields | In the Solitude of Cotton Fields is a play written by Bernard Marie Koltès in 1985. It is a two-person play involving The Client and Dealer. They negotiate a deal on the streets late at night. The play moves through mutual solitude with the Dealer unable to sell and the Client unable to buy. It’s never clear what each character has to offer each other. Koltès is interested in negotiation, power and violence and the way we treat each other.»\ \[3\] «Bernard Toublanc-Michel | Bernard Toublanc-Michel (born 6 December 1927) is a French film director and screenwriter. His 1964 film "La Difficulté d'être infidèle" was entered into the 14th Berlin International Film Festival.»\ \[4\] «Not For, or Against | Not For, or Against (Quite the Contrary) (original title: Ni pour, ni contre (bien au contraire)) is a 2003 French crime drama film directed and co-written by Cédric Klapisch. It stars Marie Gillain and Vincent Elbaz.»\ \[5\] «Bernard Comment | Bernard Comment (born April 20, 1960) is a Swiss writer, translator, scriptwriter, and publisher of books.»\ \[6\] «Other retrieved pages have titles: \`Wrong (film)\`, \`Bernard Bellefroid\`, \`Too Beautiful for You\`, \`Bernard Paul\`, \`Jean-Jacques Bernard\`, \`Anahareo\`, \`Bernard Émond\`, \`Bernard Nabonne\`, \`Marie-Christine Bernard\`, \`Bernard Giraudeau\`, \`Bernard Kops\`, \`Bernard Charlès\`, \`Bernard Assiniwi\`, \`Bernard C. Schoenfeld\`, \`Jean Bernard-Luc\`, \`Bernard Stora\`, \`Bernard Quiriny\`, \`De contemptu mundi\`, \`Marie Françoise Bernard\`, \`Marc Bernard\`, \`Up Against It\`, \`Bernard Gui\`, \`Bernard Lecache\`, \`Bernard Le Nail\`, \`Bernard Zimmer\`.»\ \ \[\[ ## thought\_1 ## \]\]\ I have found that Bernard-Marie Koltès wrote the unproduced script for "Up Against It". I need to verify if the author of "Up Against It" is also a playwright.\ \ \[\[ ## tool\_name\_1 ## \]\]\ search\_wikipedia\ \ \[\[ ## tool\_args\_1 ## \]\]\ {"query": "Up Against It author"}\ \ \[\[ ## observation\_1 ## \]\]\ \[1\] «Up Against It | Up Against It is an unproduced script by Joe Orton, written in 1967 for The Beatles at the height of their fame.»\ \[2\] «Up Against It! (Todd Rundgren album) | Up Against It! is a 1997 album by Todd Rundgren, essentially consisting of Rundgren's song demos for the Off Broadway show that were Written and Recorded by Todd from 1986-88. The project was inspired by the never-produced Up Against It which was a play originally written by Joe Orton for The Beatles.»\ \[3\] «Up Against It (album) | Up Against It is the fifth album by West London Post punk and Indie band The Times released in 1986.»\ \[4\] «Up Against It (1912 film) | The 1912 American short film Up Against It is a romantic comedy directed by Otis Turner and starring King Baggot. It was produced by the Independent Moving Pictures (IMP) Company of New York.»\ \[5\] «Against the Odds (novel) | Against the Odds is a science fiction novel by Elizabeth Moon. It is her seventh and last novel set in the Familias Regnant fictional universe. It does not fall in either informal trilogy (the Heris Serrano and the Esmay Suiza trilogies); fittingly it does not focus on any particular character, instead a more general, almost kaleidoscopic perspective of the upheaval in the Familias Regnant, and the rise to power of a new and more capable Speaker. It can be seen as a conclusion to the series, resolving or at least making a good start at resolving many issues and peoples and ending as it does on a memorializing elegiac note.»\ \[6\] «Other retrieved pages have titles: \`Against All Things Ending\`, \`Mark Greif\`, \`The Man Who Wouldn't Stand Up\`, \`Against the Day\`, \`Against a Dark Background\`, \`Against the Day (1999)\`, \`Against Happiness\`, \`Seventeen Against the Dealer\`, \`Against Our Will\`, \`Fredy Perlman\`, \`The Case Against Barack Obama\`, \`Against the Fall of Night\`, \`Against Therapy\`, \`The War Against the Assholes\`, \`Thirteen Against Fate\`, \`Hollywood, Interrupted\`, \`The Opposing Shore\`, \`Tim Weiner\`, \`Against Democracy\`, \`Against Equality of Opportunity\`, \`H. P. Lovecraft: Against the World, Against Life\`, \`Against All Enemies\`, \`Against All Enemies (novel)\`, \`The Plot Against America\`, \`Resistance (creativity)\`.»\ \ \[\[ ## thought\_2 ## \]\]\ I have confirmed that the author of the unproduced script for "Up Against It" is Joe Orton, a playwright.\ \ \[\[ ## tool\_name\_2 ## \]\]\ lookup\_wikipedia\ \ \[\[ ## tool\_args\_2 ## \]\]\ {"title": "Joe Orton"}\ \ \[\[ ## observation\_2 ## \]\]\ John Kingsley "Joe" Orton (1 January 1933 – 9 August 1967) was an English playwright and author. His public career was short but prolific, lasting from 1964 until his death three years later. During this brief period he shocked, outraged, and amused audiences with his scandalous black comedies. The adjective "Ortonesque" is sometimes used to refer to work characterised by a similarly dark yet farcical cynicism.\ \ Respond with the corresponding output fields, starting with the field \`\[\[ ## next\_thought ## \]\]\`, then \`\[\[ ## next\_tool\_name ## \]\]\` (must be formatted as a valid Python Literal\[search\_wikipedia, lookup\_wikipedia, finish\]), then \`\[\[ ## next\_tool\_args ## \]\]\` (must be formatted as a valid Python dict\[str, Any\]), and then ending with the marker for \`\[\[ ## completed ## \]\]\`.\ \ Response:\ \ \[\[ ## next\_thought ## \]\]\ I have identified both playwrights as Bernard-Marie Koltès and Joe Orton, who both wrote the unproduced script for "Up Against It".\ \ \[\[ ## next\_tool\_name ## \]\]\ finish\ \ \[\[ ## next\_tool\_args ## \]\]\ {}\ \ \[\[ ## completed ## \]\]\ \ \[2024-12-17T15:13:27.562128\]\ \ System message:\ \ Your input fields are:\ 1. \`claim\` (str)\ 2. \`trajectory\` (str)\ \ Your output fields are:\ 1. \`reasoning\` (str)\ 2. \`titles\` (list\[str\])\ \ All interactions will be structured in the following way, with the appropriate values filled in.\ \ \[\[ ## claim ## \]\]\ {claim}\ \ \[\[ ## trajectory ## \]\]\ {trajectory}\ \ \[\[ ## reasoning ## \]\]\ {reasoning}\ \ \[\[ ## titles ## \]\]\ {titles} # note: the value you produce must be pareseable according to the following JSON schema: {"type": "array", "items": {"type": "string"}}\ \ \[\[ ## completed ## \]\]\ \ In adhering to this structure, your objective is: \ You are a fact-checking assistant tasked with verifying or refuting claims using Wikipedia as your primary source. Your goal is to identify all relevant Wikipedia titles that can help substantiate or invalidate the given claim. Approach the task by reasoning through the claim step-by-step, using your knowledge to determine the best tools for gathering evidence. Utilize the available tools to search for and look up Wikipedia articles, and compile a list of titles that are pertinent to the claim. Finish the process by ensuring the list of titles accurately reflects the information needed to assess the claim's validity.\ \ User message:\ \ \[\[ ## claim ## \]\]\ 1990 Film that Khiladiyon Ka Khiladi is loosely based on stars this actor who is best known for martial arts action films.\ \ \[\[ ## trajectory ## \]\]\ \[\[ ## thought\_0 ## \]\]\ To verify the claim, I need to identify the 1990 film that "Khiladiyon Ka Khiladi" is loosely based on and the actor known for martial arts action films who starred in it. I will start by searching for information on "Khiladiyon Ka Khiladi" to find details about its inspiration.\ \ \[\[ ## tool\_name\_0 ## \]\]\ search\_wikipedia\ \ \[\[ ## tool\_args\_0 ## \]\]\ {"query": "Khiladiyon Ka Khiladi"}\ \ \[\[ ## observation\_0 ## \]\]\ \[1\] «Khiladiyon Ka Khiladi | Khiladiyon Ka Khiladi (English: Player of Players) is a 1996 Indian action film starring Rekha in her first villain role, Akshay Kumar, Raveena Tandon and former WWE wrestlers "Crush" and Brian Lee as "The Undertaker". It was the 5th highest grossing movie of the year 1996 and was declared 'SuperHit' by Box Office India. It was the fourth installment in the Khiladi (film series). The movie is loosely based based on Hollywood film Lionheart»\ \[2\] «Khiladi 420 | Khiladi 420 (English: "Con Player") is an Indian Hindi action film directed by Neeraj Vora and starring Akshay Kumar and Mahima Chaudhry. The film was written by Uttam Gudda and released on 29 December 2000. It is the seventh installment in the "Khiladi" series starring Kumar, which included "Khiladi" (1992), "Main Khiladi Tu Anari" (1994), "Sabse Bada Khiladi" (1995), "Khiladiyon Ka Khiladi" (1996), "Mr. and Mrs. Khiladi" (1997) and "International Khiladi" (1999).»\ \[3\] «Khiladi (1992 film) | Khiladi (English: "Player" ) is a 1992 Indian suspense thriller film directed by Abbas Mustan. The film was Akshay Kumar's breakthrough role and also stars Ayesha Jhulka, Deepak Tijori, Sabeeha. While Prem Chopra, Shakti Kapoor, Anant Mahadevan and Johnny Lever played supporting roles. "Khiladi" was the first installment in the Khiladi (film series) which had "Khiladi" in the title and Akshay Kumar in the leading role. It was followed by "Main Khiladi Tu Anari" (1994), "Sabse Bada Khiladi" (1995), "Khiladiyon Ka Khiladi" (1996), "Mr. and Mrs. Khiladi" (1997), "International Khiladi" (1999), "Khiladi 420"(2000) and "Khiladi 786" (2012). Khiladi was critically and commercially success at the box-office and the tenth highest grossing film of 1992. It was Akshay Kumar's first successful movie and was declared a "Super Hit" at the box office. The basic premise of the story is similar to 1975 released movie Khel Khel Mein starring Rishi Kapoor and Neetu Singh. The film was remade in Kannada as "Aata Hudugaata".»\ \[4\] «Khiladi (film series) | Khiladi series is a Bollywood action film series starring Akshay Kumar in the lead role. However, unlike other film series, other than having Akshay Kumar in lead role, and other than having the word "Khiladi" in the title, these films have nothing in common. The producers, directors and stories of these films are totally different. " Khiladi" (1992) was the first in a series of films which had Akshay Kumar in the title role and gave it his first breakthrough role. It was followed by "Main Khiladi Tu Anari" (1994), "Sabse Bada Khiladi" (1995), "Khiladiyon Ka Khiladi" (1996), "Mr. and Mrs. Khiladi" (1997), "International Khiladi" (1999) and "Khiladi 420" (2000), all featuring Kumar in the lead role. The latest film in the franchise is "Khiladi 786" (2012).»\ \[5\] «Khiladi 786 | Khiladi 786 (खिलाड़ी 786) is a 2012 Indian Hindi Punjabi action comedy film directed by Ashish R Mohan, featuring Akshay Kumar in the title role alongside Asin playing the female lead. It features Himesh Reshammiya, Mithun Chakraborty, Raj Babbar and Mukesh Rishi in supporting roles. The film marks the return of Akshay Kumar to his "Khiladi" series after 12 years. It is mostly shot in Mumbai and Punjab. Khiladi786 is AkshayKumar 100th film.»\ \[6\] «Other retrieved pages have titles: \`Khiladi (2013 film)\`, \`Khiladi (1968 film)\`, \`International Khiladi\`, \`Khatron Ke Khiladi (1988 film)\`, \`Khatron Ke Khiladi (2001 film)\`, \`Umesh Mehra\`, \`Shatranj ke Khiladi\`, \`Main Khiladi Tu Anari\`, \`Fear Factor: Khatron Ke Khiladi (season 7)\`, \`Fear Factor: Khatron Ke Khiladi\`, \`Mr. and Mrs. Khiladi\`, \`Sabse Bada Khiladi\`, \`Ek Khiladi Bawan Pattey\`, \`Ek Khiladi Ek Haseena (TV series)\`, \`Do Khiladi\`, \`Ek Khiladi Ek Haseena (film)\`, \`Khel Khilari Ka\`, \`Fear Factor: Khatron Ke Khiladi (season 8)\`, \`Fear Factor: Khatron Ke Khiladi Darr Ka Blockbuster\`, \`Khilona (1942 film)\`, \`Fear Factor: Khatron Ke Khiladi Darr Ka Blockbuster Returns\`, \`Chhota Bheem: Dholakpur ke Khiladi\`, \`Khilani\`, \`Kandireega\`, \`Khilaaf\`.»\ \ \[\[ ## thought\_1 ## \]\]\ The search results indicate that "Khiladiyon Ka Khiladi" is loosely based on the Hollywood film "Lionheart". I should now verify which actor, known for martial arts action films, starred in "Lionheart".\ \ \[\[ ## tool\_name\_1 ## \]\]\ search\_wikipedia\ \ \[\[ ## tool\_args\_1 ## \]\]\ {"query": "Lionheart 1990 film"}\ \ \[\[ ## observation\_1 ## \]\]\ \[1\] «Lionheart (1990 film) | Lionheart (also known as Wrong Bet, A.W.O.L.: Absent Without Leave, Leon and Full Contact) is a 1990 action film, directed by Sheldon Lettich, starring Jean-Claude Van Damme and co-starring Brian Thompson, along with Harrison Page, Deborah Rennard, Lisa Pelikan, and Ashley Johnson.»\ \[2\] «Truly, Madly, Deeply | Truly, Madly, Deeply is a 1990 British fantasy drama film made for the BBC's "Screen Two" series, by BBC Films, Lionheart and Winston Pictures. The film, written and directed by Anthony Minghella, stars Juliet Stevenson and Alan Rickman.»\ \[3\] «Lionheart (1987 film) | Lionheart, also known as Lionheart: The Children's Crusade, is a 1987 adventure film directed by Franklin J. Schaffner and produced by Talia Shire and Stanley O'Toole. Shire's brother, Francis Ford Coppola, initially planned to direct the film but instead opted to be executive producer along with Shire's husband, Jack Schwartzman. The screenplay was written by Menno Meyjes and Richard Outten from a story by Meyjes. The composer Jerry Goldsmith wrote the score. The film was released in August 1987. It was distributed by Orion Pictures.»\ \[4\] «Lionheart (2016 film) | Lionheart is a 2016 American boxing film short written and produced by Oscar DeRosa and Orlando Cicilia III. The film stars Oscar DeRosa and Marc Macaulay. The film portrays struggling professional boxer Max Rossi who is finally presented with the fight he's been waiting for that will launch his career to the next level but when he is suddenly confronted with a life-changing opportunity he must decide which path to take.»\ \[5\] «Richard the Lionheart (2013 film) | Richard The Lionheart is a 2013 film, starring Chandler Maness as Richard the Lionheart, Malcolm McDowell as King Henry II and Andrea Zirio as Henry the Young.»\ \[6\] «Other retrieved pages have titles: \`Richard the Lionheart: Rebellion\`, \`The Brothers Lionheart (1977 film)\`, \`Kodama Simham\`, \`Jean-Claude Van Damme\`, \`Lionsheart\`, \`A Day for Lionhearts\`, \`The Lion in Winter (1968 film)\`, \`Lion Jagapathi Rao\`, \`Theatre of Blood\`, \`Young Lions (album)\`, \`Richard the Lion-Hearted (1923 film)\`, \`Bullseye! (1990 film)\`, \`Fear (1990 film)\`, \`The Guard (1990 film)\`, \`Ironheart\`, \`The Guardian (1990 film)\`, \`Tiger Cage 2\`, \`Lion Rampant (game publisher)\`, \`Success (1991 film)\`, \`Lion in the Streets\`, \`The King of the Kickboxers\`, \`Lionheart (video game)\`, \`Richard the Lionheart (TV series)\`, \`Dermot Walsh\`, \`Lars Söderdahl\`.»\ \ \[\[ ## thought\_2 ## \]\]\ The search results confirm that the 1990 film "Lionheart" stars Jean-Claude Van Damme, who is well-known for his martial arts action films. This information aligns with what is needed to verify the claim. I now have the relevant titles related to the claim: "Khiladiyon Ka Khiladi" and "Lionheart."\ \ \[\[ ## tool\_name\_2 ## \]\]\ finish\ \ \[\[ ## tool\_args\_2 ## \]\]\ {}\ \ \[\[ ## observation\_2 ## \]\]\ Completed.\ \ Respond with the corresponding output fields, starting with the field \`\[\[ ## reasoning ## \]\]\`, then \`\[\[ ## titles ## \]\]\` (must be formatted as a valid Python list\[str\]), and then ending with the marker for \`\[\[ ## completed ## \]\]\`.\ \ Assistant message:\ \ \[\[ ## reasoning ## \]\]\ The claim suggests that the 1990 film which "Khiladiyon Ka Khiladi" is loosely based on stars an actor known for martial arts action films. Upon investigation, "Khiladiyon Ka Khiladi" is indeed loosely based on the Hollywood film "Lionheart". The 1990 film "Lionheart" stars Jean-Claude Van Damme, who is recognized for his roles in martial arts action films. This information supports the claim.\ \ \[\[ ## titles ## \]\]\ \["Khiladiyon Ka Khiladi", "Lionheart (1990 film)", "Jean-Claude Van Damme"\]\ \ \[\[ ## completed ## \]\]\ \ User message:\ \ \[\[ ## claim ## \]\]\ Both the Chengiopanax sciadophylloides and Amaryllis are genus.\ \ \[\[ ## trajectory ## \]\]\ \[\[ ## thought\_0 ## \]\]\ To verify the claim, I need to search for information regarding both "Chengiopanax sciadophylloides" and "Amaryllis" to see if they are indeed classified as genus.\ \ \[\[ ## tool\_name\_0 ## \]\]\ search\_wikipedia\ \ \[\[ ## tool\_args\_0 ## \]\]\ {"query": "Chengiopanax sciadophylloides"}\ \ \[\[ ## observation\_0 ## \]\]\ \[1\] «Chengiopanax sciadophylloides | Chengiopanax sciadophylloides is a flowering tree in the family Araliaceae native to Japan. Previously included in the genus "Eleutherococcus", it is distinguished from other members of that genus by not having spines or prickles and ITS sequence data confirmed the separation.»\ \[2\] «Hunaniopanax hypoglaucus | Hunaniopanax hypoglaucus is a species of flowering plant of family Araliaceae, and the only species of genus Hunanioglaucus, named after the Chinese province of Hunan. Some authorities suggest merging this species into the genus "Aralia".»\ \[3\] «Cortinarius caesiophylloides | Cortinarius caesiophylloides is a species of fungus in the large mushroom genus "Cortinarius" (subgenus "Phlegmacium"). Found in Fennoscandia, where it grows on the ground in mesic coniferous forests, it was described as new to science in 2014. The specific epithet "caesiophylloides" alludes to both its similarity to "Cortinarius multiformis" var. "caesiophyllus" (now named "C. caesiolamellatus"), and the bluish tints in the gills. It has since been found in Slovakia.»\ \[4\] «Merrilliopanax | Merrilliopanax is a genus of flowering plant of family Araliaceae, comprising 3 species of the sub-genus Airampora. They are found in western China, Myanmar, northeast India, Bhutan, and Nepal.»\ \[5\] «Schefflera morototoni | Schefflera morototoni (yagrumo macho; syn. "Didymopanax morototoni" (Aubl. ) Decne. & Planch. , "Didymopanax morototoni" var. "angustipetalum" March; "Panax morototoni" Aublet; "Sciadophyllum paniculatum" Britton ) is a timber tree native to southern Mexico, the Greater Antilles, Central America, and South America. It grows in a variety of habitats, such as the Caatinga, Cerrado, and Amazon Rainforest of Brazil.»\ \[6\] «Other retrieved pages have titles: \`Merrilliopanax chinensis\`, \`Harmsiopanax ingens\`, \`Gunniopsis zygophylloides\`, \`Merrilliopanax alpinus\`, \`Sciadophyton\`, \`Sinopanax formosanus\`, \`Tabelliscolex\`, \`Tetragonia tetragonioides\`, \`Chengjiangocaris\`, \`Wan Chun Cheng\`, \`Cheng Weigao\`, \`Chengguan, Chamdo\`, \`Nocardioides panzhihuensis\`, \`Cardiodictyon\`, \`Scyphiphora\`, \`Puschkinia scilloides\`, \`Liao Chengzhi\`, \`Lilaea scilloides\`, \`Chenggao\`, \`Chengjiao Subdistrict, Ningxiang\`, \`Seioglobal\`, \`Chengjiangaspis\`, \`Cheng Shikai\`, \`Xenocyprioides\`, \`Chenggong fire belly newt\`.»\ \ \[\[ ## thought\_1 ## \]\]\ The search results indicate that "Chengiopanax sciadophylloides" is described as a flowering tree and doesn't specifically mention it being a genus. It was previously included in the genus "Eleutherococcus". To verify the claim, I need to ensure "Amaryllis" is classified as a genus.\ \ \[\[ ## tool\_name\_1 ## \]\]\ search\_wikipedia\ \ \[\[ ## tool\_args\_1 ## \]\]\ {"query": "Amaryllis"}\ \ \[\[ ## observation\_1 ## \]\]\ \[1\] «Amaryllis (disambiguation) | Amaryllis is a small genus of flowering bulbs, with two species.»\ \[2\] «Amaryllis | Amaryllis ( ) is the only genus in the subtribe Amaryllidinae (tribe Amaryllideae). It is a small genus of flowering bulbs, with two species. The better known of the two, "Amaryllis belladonna", is a native of the Western Cape region of South Africa, particularly the rocky southwest area between the Olifants River Valley to Knysna. For many years there was confusion among botanists over the generic names "Amaryllis" and "Hippeastrum", one result of which is that the common name "amaryllis" is mainly used for cultivars of the genus "Hippeastrum", widely sold in the winter months for their ability to bloom indoors. Plants of the genus "Amaryllis" are known as belladonna lily, Jersey lily, naked lady, amarillo, Easter lily in Southern Australia or, in South Africa, March lily due to its propensity to flower around March. This is one of numerous genera with the common name "lily" due to their flower shape and growth habit. However, they are only distantly related to the true lily, "Lilium".»\ \[3\] «Amaryllis (given name) | Amaryllis (Αμαρυλλίς) is a female ancient Greek name and means "sparkling". According the mythology, the name of the beautiful flower Amaryllis derived from the nymph Amaryllis.»\ \[4\] «Amaryllidaceae | The Amaryllidaceae are a family of herbaceous, mainly perennial and bulbous (rarely rhizomatous) flowering plants in the monocot order Asparagales. The family takes its name from the genus "Amaryllis" and is commonly known as the amaryllis family. The leaves are usually linear, and the flowers are usually bisexual and symmetrical, arranged in umbels on the stem. The petals and sepals are undifferentiated as tepals, which may be fused at the base into a floral tube. Some also display a corona. Allyl sulfide compounds produce the characteristic odour of the onion subfamily (Allioideae).»\ \[5\] «Amaryllis paradisicola | Amaryllis paradisicola is a species of bulbous perennial plant from South Africa.»\ \[6\] «Other retrieved pages have titles: \`Amaryllis (catamaran)\`, \`Amaryllis (yacht)\`, \`Amaryllis belladonna\`, \`Schinia amaryllis\`, \`Hippeastrum reticulatum\`, \`Pirangoclytus amaryllis\`, \`Amaryllis (ship)\`, \`Hippeastrum psittacinum\`, \`Ogyris amaryllis\`, \`Amaryllis (Shinedown album)\`, \`Amaryllis (restaurant)\`, \`1085 Amaryllis\`, \`Apatelodes amaryllis\`, \`Premolis amaryllis\`, \`Amaryllideae\`, \`Amaryllis Garnett\`, \`Amaryllis Chamber Ensemble\`, \`Amaryllis Fleming\`, \`Brithys crini\`, \`Amaryllis (Marilyn Crispell, Gary Peacock and Paul Motian album)\`, \`Amarylloidinae\`, \`Amaryllidoideae\`, \`Amaryllis Night and Day\`, \`Amaryllididae\`, \`Aecidium amaryllidis\`.»\ \ \[\[ ## thought\_2 ## \]\]\ The search results confirm that "Amaryllis" is a genus, specifically a small genus of flowering bulbs. However, the earlier results for "Chengiopanax sciadophylloides" do not confirm it as a genus; it is described more as a species of flowering tree. Therefore, Chengiopanax sciadophylloides might not be a genus itself but belongs to a different classification. I will now verify if "Chengiopanax" itself is recognized as a genus.\ \ \[\[ ## tool\_name\_2 ## \]\]\ search\_wikipedia\ \ \[\[ ## tool\_args\_2 ## \]\]\ {"query": "Chengiopanax"}\ \ \[\[ ## observation\_2 ## \]\]\ \[1\] «Chengiopanax sciadophylloides | Chengiopanax sciadophylloides is a flowering tree in the family Araliaceae native to Japan. Previously included in the genus "Eleutherococcus", it is distinguished from other members of that genus by not having spines or prickles and ITS sequence data confirmed the separation.»\ \[2\] «Hunaniopanax hypoglaucus | Hunaniopanax hypoglaucus is a species of flowering plant of family Araliaceae, and the only species of genus Hunanioglaucus, named after the Chinese province of Hunan. Some authorities suggest merging this species into the genus "Aralia".»\ \[3\] «Merrilliopanax | Merrilliopanax is a genus of flowering plant of family Araliaceae, comprising 3 species of the sub-genus Airampora. They are found in western China, Myanmar, northeast India, Bhutan, and Nepal.»\ \[4\] «Xiaxue | Cheng Yan Yan Wendy (born Cheng Yan Yan; 28 April 1984), better known by her pseudonym Xiaxue, is a Singaporean blogger and online television personality who writes about her life, fashion and local issues in a provocative style. Her main blog, which attracts about 50,000 readers daily, has won prestigious blog awards and earned her sponsorship deals, as well as stints as a columnist and TV show host, but some of her posts have sparked national controversies. She is married to American engineer Mike Sayre and they have one child.»\ \[5\] «Sinopanax formosanus | Sinopanax formosanus is a species of flowering plant of family Araliaceae and the only species of genus Sinopanax. It is endemic to Taiwan. It is a small, evergreen tree, up to 12 m in height, that grows in open areas in mountainous forests at altitudes between 2300 - .»\ \[6\] «Other retrieved pages have titles: \`Merrilliopanax chinensis\`, \`Linpan in Chengdu Plain\`, \`Pan Chengdong\`, \`Merrilliopanax alpinus\`, \`Pan Cheng-tsung\`, \`Pan Chengyou\`, \`Seioglobal\`, \`Harmsiopanax ingens\`, \`Kalopanax\`, \`Cheng Jiao\`, \`Chengo\`, \`Chengguan, Chamdo\`, \`Chenguang\`, \`Chenggao\`, \`Chengdu University of Information Technology\`, \`Chengai Sivam\`, \`Chênggo\`, \`Cheng Weigao\`, \`Cheng Xunzhao\`, \`Gan Peck Cheng\`, \`Cheng Hong\`, \`Chengdu Qianbao F.C.\`, \`Cheng I\`, \`Cheng Yu-tung\`, \`Sun Chengyao\`.»\ \ \[\[ ## thought\_3 ## \]\]\ The search results do not definitively confirm "Chengiopanax" as a genus, rather they focus on the species "Chengiopanax sciadophylloides" which was previously part of the "Eleutherococcus" genus. There is no clear indication in the search results that "Chengiopanax" is recognized as a genus. To ensure a comprehensive understanding, I will look up the "Chengiopanax sciadophylloides" page directly to see if there is further clarification on its classification as a genus.\ \ \[\[ ## tool\_name\_3 ## \]\]\ lookup\_wikipedia\ \ \[\[ ## tool\_args\_3 ## \]\]\ {"title": "Chengiopanax sciadophylloides"}\ \ \[\[ ## observation\_3 ## \]\]\ Chengiopanax sciadophylloides is a flowering tree in the family Araliaceae native to Japan. Previously included in the genus "Eleutherococcus", it is distinguished from other members of that genus by not having spines or prickles and ITS sequence data confirmed the separation.\ \ \[\[ ## thought\_4 ## \]\]\ The lookup did not provide any new information suggesting that "Chengiopanax" is a genus. It seems that "Chengiopanax sciadophylloides" is a species and not a genus. On the other hand, "Amaryllis" is confirmed to be a genus. Therefore, the claim that both "Chengiopanax sciadophylloides" and "Amaryllis" are genus is only partially correct. I will now finish with the titles that I have verified.\ \ \[\[ ## tool\_name\_4 ## \]\]\ finish\ \ \[\[ ## tool\_args\_4 ## \]\]\ {}\ \ \[\[ ## observation\_4 ## \]\]\ Completed.\ \ Respond with the corresponding output fields, starting with the field \`\[\[ ## reasoning ## \]\]\`, then \`\[\[ ## titles ## \]\]\` (must be formatted as a valid Python list\[str\]), and then ending with the marker for \`\[\[ ## completed ## \]\]\`.\ \ Assistant message:\ \ \[\[ ## reasoning ## \]\]\ The claim states that both "Chengiopanax sciadophylloides" and "Amaryllis" are genus. Upon investigation, it was found that "Chengiopanax sciadophylloides" is a species of flowering tree, and there is no confirmation that "Chengiopanax" itself is a recognized genus. It was previously included in the genus "Eleutherococcus". On the other hand, "Amaryllis" is confirmed to be a genus, specifically a small genus of flowering bulbs with two species. Therefore, the claim is only partially correct as "Chengiopanax sciadophylloides" is not a genus.\ \ \[\[ ## titles ## \]\]\ \["Amaryllis", "Eleutherococcus", "Chengiopanax sciadophylloides"\]\ \ \[\[ ## completed ## \]\]\ \ User message:\ \ \[\[ ## claim ## \]\]\ 200,000 people go through the airport where the headquarters of the airline that acquired Baboo Airlines are located.\ \ \[\[ ## trajectory ## \]\]\ \[\[ ## thought\_0 ## \]\]\ I need to identify which airline acquired Baboo Airlines to verify the claim. Once I have the name of the airline, I can find out where its headquarters are located and then look for information on the airport's passenger traffic.\ \ \[\[ ## tool\_name\_0 ## \]\]\ search\_wikipedia\ \ \[\[ ## tool\_args\_0 ## \]\]\ {"query": "Baboo Airlines acquisition"}\ \ \[\[ ## observation\_0 ## \]\]\ \[1\] «Baboo (airline) | Fly Baboo SA, marketed as Baboo, was a regional airline based on the grounds of Geneva International Airport and in Grand-Saconnex, Switzerland. In November 2010, the company was saved from bankruptcy and acquired by Darwin Airline Group from Lugano.»\ \[2\] «Okada Air | Okada Air was an airline based in Benin City, Nigeria. The carrier was established in 1983 with a fleet of BAC-One Eleven 300s. and started charter operations in the same year. In 1984, a Boeing 707-355C was acquired for cargo operations. By 1990, ten BAC One-Elevens were bought, and eight more were acquired in 1991. The company was granted the right of operating international flights in 1992.»\ \[3\] «Go (airline) | Go Fly (styled and trading as Go) was the name of a British low-cost airline, founded by British Airways (BA) in 1998. It operated flights between London Stansted Airport and destinations in Europe. The airline was purchased from BA in a management buy-out backed by the private equity firm 3i in 2001. In 2002 it was bought by its rival EasyJet, and was merged into the airline's operations. Its head office was in the Enterprise House in London Stansted Airport in Stansted Mountfitchet, Essex.»\ \[4\] «BAX Global | BAX Global is an American international shipping company that is headquartered in Irvine, California, and has other major offices in Australia, Singapore, London, the Netherlands and Toledo, Ohio. The company which was founded in 1971 also operates an airline. After having been acquired by DB Logistics in January 2006, Bax Global is currently being integrated with Schenker, the land, sea, and air freight branch of DB Logistics. DB Logistics is the freight and logistics division of Deutsche Bahn, the State-owned German railway company.»\ \[5\] «Iloilo-Negros Air Express | Iloilo-Negros Air Express Company, Inc., which operated as Iloilo-Negros Air Express or INAEC and later as FEATI, was an airline based in the Philippines. The airline was folded into the operation of Philippine Airlines (PAL) which was re-nationalized by the Philippine government and became the national flag carrier. The company built the Bacolod International Airport in Bacolod City, which was bought by PAL after the acquisition of the airline.»\ \[6\] «Other retrieved pages have titles: \`Babool (brand)\`, \`British Aircraft Corporation\`, \`Boeing Aircraft Holding Company\`, \`Bakhtar Afghan Airlines\`, \`Baboo destinations\`, \`Baboo (band)\`, \`Baboom\`, \`Bond Aviation Group\`, \`Mark Darby\`, \`AGS Airports\`, \`BA Connect\`, \`AirMall\`, \`BAT Baboon\`, \`National Shipping Company of Saudi Arabia\`, \`Aviesa\`, \`British Airways Ltd (2012–15)\`, \`BA CityFlyer\`, \`Baboon (album)\`, \`Qingdao Haili Helicopters\`, \`Business Express Airlines\`, \`Babo Airport\`, \`Baboon lymphocryptovirus\`, \`BxB\`, \`British Airways\`, \`Manchester Airports Group\`.»\ \ \[\[ ## thought\_1 ## \]\]\ The observation indicates that Baboo Airlines was acquired by Darwin Airline Group. Next, I should search for the headquarters location of Darwin Airline Group to identify the associated airport and find out the passenger traffic.\ \ \[\[ ## tool\_name\_1 ## \]\]\ search\_wikipedia\ \ \[\[ ## tool\_args\_1 ## \]\]\ {"query": "Darwin Airline Group headquarters"}\ \ \[\[ ## observation\_1 ## \]\]\ \[1\] «Darwin Airline | Darwin Airline SA is a Swiss regional airline with its head office in Bioggio, Lugano flying under the brand name Adria Airways Switzerland. It has been operating scheduled domestic and international services throughout some western European countries using the brand name Etihad Regional from January 2014 until July 2017 when it was sold from Etihad Airways to Adria Airways as well as formerly also on behalf of Alitalia. Its bases are Geneva Airport and Lugano Airport.»\ \[2\] «Airnorth | Capiteq Limited, trading as Airnorth, is a regional airline based at Darwin International Airport in Darwin, Northern Territory, Australia. It operates scheduled and charter services in the Northern Territory, Queensland, Victoria, Western Australia, and East Timor.»\ \[3\] «Fly Tiwi | Fly Tiwi is an Australian airline based in Darwin, Northern Territory, offering scheduled passenger services between the Northern Territory capital and communities located on the Tiwi, South Goulburn and Croker islands, as well as a number of remote Arnhem Land communities and the town of Tennant Creek. The company is wholly owned by the Hardy Aviation group, Australia's largest general aviation company and was founded in 2008 in association with the Tiwi Land Council and now operates over 50 flights per week between 9 destinations.»\ \[4\] «RAAF Base Darwin | RAAF Base Darwin (IATA: DRW, ICAO: YPDN) is a Royal Australian Air Force (RAAF) military air base located in the city of Darwin, in the Northern Territory, Australia. The base shares its runway with Darwin International Airport, for civil aviation purposes. The heritage-listed RAAF Base Darwin is a forward operating base with year round activity with approximately 400 personnel.»\ \[5\] «Lugano Airport | Lugano Airport (IATA: LUG, ICAO: LSZA) is a regional airport located 4 km west of the Swiss city of Lugano, approximately 80 km north of Milan, in the municipalities of Agno, Bioggio and Muzzano. It lies closer to the village of Agno than to Lugano itself, and is sometimes known as Lugano-Agno. Around 200,000 passengers use the airport each year, using some 2,400 flights. There are also limited freight operations, a flying club and flight school. The headquarters of Darwin Airline are located at the airport.»\ \[6\] «Other retrieved pages have titles: \`No. 76 Wing RAAF\`, \`No. 13 Squadron RAAF\`, \`Mount Darwin Airport\`, \`Darwin International Airport\`, \`Baboo (airline)\`, \`Virgin Australia Regional Airlines\`, \`No. 1 Wing RAAF\`, \`No. 114 Mobile Control and Reporting Unit RAAF\`, \`Air Austral\`, \`Tigerair Australia destinations\`, \`Marine Rotational Force – Darwin\`, \`Austral Líneas Aéreas\`, \`Australian Aviation Heritage Centre\`, \`AirAsia\`, \`Flight Centre Australia\`, \`Flight Centre\`, \`North-Western Area Command (RAAF)\`, \`No. 5 Fighter Sector RAAF\`, \`McMillans Road\`, \`Cobham Aviation Services Australia\`, \`Asia Pacific Transport Consortium\`, \`1st Aviation Regiment (Australia)\`, \`Charles Darwin Foundation\`, \`Brindabella Airlines\`, \`Kulula.com\`.»\ \ \[\[ ## thought\_2 ## \]\]\ The headquarters of Darwin Airline are located at Lugano Airport, which has around 200,000 passengers using the airport each year. This aligns with the claim about the passenger traffic at the airport where the headquarters of the airline that acquired Baboo Airlines are located.\ \ \[\[ ## tool\_name\_2 ## \]\]\ finish\ \ \[\[ ## tool\_args\_2 ## \]\]\ {}\ \ \[\[ ## observation\_2 ## \]\]\ Completed.\ \ Respond with the corresponding output fields, starting with the field \`\[\[ ## reasoning ## \]\]\`, then \`\[\[ ## titles ## \]\]\` (must be formatted as a valid Python list\[str\]), and then ending with the marker for \`\[\[ ## completed ## \]\]\`.\ \ Assistant message:\ \ \[\[ ## reasoning ## \]\]\ The claim states that 200,000 people go through the airport where the headquarters of the airline that acquired Baboo Airlines are located. Through the information found, Baboo Airlines was acquired by Darwin Airline Group. The headquarters of Darwin Airline Group are located at Lugano Airport. Further research revealed that Lugano Airport handles approximately 200,000 passengers per year. This confirms the accuracy of the claim regarding passenger traffic at the airport associated with Darwin Airline Group's headquarters.\ \ \[\[ ## titles ## \]\]\ \["Baboo (airline)", "Darwin Airline", "Lugano Airport"\]\ \ \[\[ ## completed ## \]\]\ \ User message:\ \ \[\[ ## claim ## \]\]\ The author of the 1960s unproduced script written for The Beatles, Up Against It, and Bernard-Marie Koltès are both playwrights.\ \ \[\[ ## trajectory ## \]\]\ \[\[ ## thought\_0 ## \]\]\ To verify the claim, I need to identify the playwrights who wrote the unproduced script for "Up Against It" and "The Beatles".\ \ \[\[ ## tool\_name\_0 ## \]\]\ search\_wikipedia\ \ \[\[ ## tool\_args\_0 ## \]\]\ {"query": "Up Against It script writer Bernard-Marie Koltès"}\ \ \[\[ ## observation\_0 ## \]\]\ \[1\] «Bernard-Marie Koltès | Bernard-Marie Koltès (\] ; 9 April 1948 – 15 April 1989) was a French playwright and theatre director best known for his plays "La Nuit juste avant les Forêts" ("The Night Just Before the Forests", 1976), "Sallinger" (1977) and "Dans la Solitude des Champs de Coton" ("In the Solitude of Cotton Fields", 1986).»\ \[2\] «In the Solitude of Cotton Fields | In the Solitude of Cotton Fields is a play written by Bernard Marie Koltès in 1985. It is a two-person play involving The Client and Dealer. They negotiate a deal on the streets late at night. The play moves through mutual solitude with the Dealer unable to sell and the Client unable to buy. It’s never clear what each character has to offer each other. Koltès is interested in negotiation, power and violence and the way we treat each other.»\ \[3\] «Bernard Toublanc-Michel | Bernard Toublanc-Michel (born 6 December 1927) is a French film director and screenwriter. His 1964 film "La Difficulté d'être infidèle" was entered into the 14th Berlin International Film Festival.»\ \[4\] «Not For, or Against | Not For, or Against (Quite the Contrary) (original title: Ni pour, ni contre (bien au contraire)) is a 2003 French crime drama film directed and co-written by Cédric Klapisch. It stars Marie Gillain and Vincent Elbaz.»\ \[5\] «Bernard Comment | Bernard Comment (born April 20, 1960) is a Swiss writer, translator, scriptwriter, and publisher of books.»\ \[6\] «Other retrieved pages have titles: \`Wrong (film)\`, \`Bernard Bellefroid\`, \`Too Beautiful for You\`, \`Bernard Paul\`, \`Jean-Jacques Bernard\`, \`Anahareo\`, \`Bernard Émond\`, \`Bernard Nabonne\`, \`Marie-Christine Bernard\`, \`Bernard Giraudeau\`, \`Bernard Kops\`, \`Bernard Charlès\`, \`Bernard Assiniwi\`, \`Bernard C. Schoenfeld\`, \`Jean Bernard-Luc\`, \`Bernard Stora\`, \`Bernard Quiriny\`, \`De contemptu mundi\`, \`Marie Françoise Bernard\`, \`Marc Bernard\`, \`Up Against It\`, \`Bernard Gui\`, \`Bernard Lecache\`, \`Bernard Le Nail\`, \`Bernard Zimmer\`.»\ \ \[\[ ## thought\_1 ## \]\]\ I have found that Bernard-Marie Koltès wrote the unproduced script for "Up Against It". I need to verify if the author of "Up Against It" is also a playwright.\ \ \[\[ ## tool\_name\_1 ## \]\]\ search\_wikipedia\ \ \[\[ ## tool\_args\_1 ## \]\]\ {"query": "Up Against It author"}\ \ \[\[ ## observation\_1 ## \]\]\ \[1\] «Up Against It | Up Against It is an unproduced script by Joe Orton, written in 1967 for The Beatles at the height of their fame.»\ \[2\] «Up Against It! (Todd Rundgren album) | Up Against It! is a 1997 album by Todd Rundgren, essentially consisting of Rundgren's song demos for the Off Broadway show that were Written and Recorded by Todd from 1986-88. The project was inspired by the never-produced Up Against It which was a play originally written by Joe Orton for The Beatles.»\ \[3\] «Up Against It (album) | Up Against It is the fifth album by West London Post punk and Indie band The Times released in 1986.»\ \[4\] «Up Against It (1912 film) | The 1912 American short film Up Against It is a romantic comedy directed by Otis Turner and starring King Baggot. It was produced by the Independent Moving Pictures (IMP) Company of New York.»\ \[5\] «Against the Odds (novel) | Against the Odds is a science fiction novel by Elizabeth Moon. It is her seventh and last novel set in the Familias Regnant fictional universe. It does not fall in either informal trilogy (the Heris Serrano and the Esmay Suiza trilogies); fittingly it does not focus on any particular character, instead a more general, almost kaleidoscopic perspective of the upheaval in the Familias Regnant, and the rise to power of a new and more capable Speaker. It can be seen as a conclusion to the series, resolving or at least making a good start at resolving many issues and peoples and ending as it does on a memorializing elegiac note.»\ \[6\] «Other retrieved pages have titles: \`Against All Things Ending\`, \`Mark Greif\`, \`The Man Who Wouldn't Stand Up\`, \`Against the Day\`, \`Against a Dark Background\`, \`Against the Day (1999)\`, \`Against Happiness\`, \`Seventeen Against the Dealer\`, \`Against Our Will\`, \`Fredy Perlman\`, \`The Case Against Barack Obama\`, \`Against the Fall of Night\`, \`Against Therapy\`, \`The War Against the Assholes\`, \`Thirteen Against Fate\`, \`Hollywood, Interrupted\`, \`The Opposing Shore\`, \`Tim Weiner\`, \`Against Democracy\`, \`Against Equality of Opportunity\`, \`H. P. Lovecraft: Against the World, Against Life\`, \`Against All Enemies\`, \`Against All Enemies (novel)\`, \`The Plot Against America\`, \`Resistance (creativity)\`.»\ \ \[\[ ## thought\_2 ## \]\]\ I have confirmed that the author of the unproduced script for "Up Against It" is Joe Orton, a playwright.\ \ \[\[ ## tool\_name\_2 ## \]\]\ lookup\_wikipedia\ \ \[\[ ## tool\_args\_2 ## \]\]\ {"title": "Joe Orton"}\ \ \[\[ ## observation\_2 ## \]\]\ John Kingsley "Joe" Orton (1 January 1933 – 9 August 1967) was an English playwright and author. His public career was short but prolific, lasting from 1964 until his death three years later. During this brief period he shocked, outraged, and amused audiences with his scandalous black comedies. The adjective "Ortonesque" is sometimes used to refer to work characterised by a similarly dark yet farcical cynicism.\ \ \[\[ ## thought\_3 ## \]\]\ I have identified both playwrights as Bernard-Marie Koltès and Joe Orton, who both wrote the unproduced script for "Up Against It".\ \ \[\[ ## tool\_name\_3 ## \]\]\ finish\ \ \[\[ ## tool\_args\_3 ## \]\]\ {}\ \ \[\[ ## observation\_3 ## \]\]\ Completed.\ \ Respond with the corresponding output fields, starting with the field \`\[\[ ## reasoning ## \]\]\`, then \`\[\[ ## titles ## \]\]\` (must be formatted as a valid Python list\[str\]), and then ending with the marker for \`\[\[ ## completed ## \]\]\`.\ \ Response:\ \ \[\[ ## reasoning ## \]\]\ The claim states that the authors of the unproduced script for "Up Against It" are both playwrights. Investigation revealed that Bernard-Marie Koltès wrote the unproduced script, and further research confirmed that Joe Orton was also a playwright who wrote the script. Therefore, both playwrights are indeed the authors of the unproduced script for "Up Against It".\ \ \[\[ ## titles ## \]\]\ \["Bernard-Marie Koltès", "Joe Orton"\]\ \ \[\[ ## completed ## \]\]\ \ Finally, let's save our optimized program so we can use it again later.\ \ In \[18\]:\ \ Copied!\ \ optimized\_react.save("optimized\_react.json")\ \ loaded\_react \= dspy.ReAct("claim -> titles: list\[str\]", tools\=\[search\_wikipedia, lookup\_wikipedia\], max\_iters\=20)\ loaded\_react.load("optimized\_react.json")\ \ loaded\_react(claim\="The author of the 1960s unproduced script written for The Beatles, Up Against It, and Bernard-Marie Koltès are both playwrights.").titles\ \ optimized\_react.save("optimized\_react.json") loaded\_react = dspy.ReAct("claim -> titles: list\[str\]", tools=\[search\_wikipedia, lookup\_wikipedia\], max\_iters=20) loaded\_react.load("optimized\_react.json") loaded\_react(claim="The author of the 1960s unproduced script written for The Beatles, Up Against It, and Bernard-Marie Koltès are both playwrights.").titles\ \ Out\[18\]:\ \ \['Bernard-Marie Koltès', 'Joe Orton'\]\ \ Back to top --- # Multi-Hop Search - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/tutorials/multihop_search/index.ipynb "Edit this page") Tutorial: Multi-Hop Retrieval[¶](#tutorial-multi-hop-retrieval) ================================================================ Let's walk through a quick example of building a `dspy.Module` with multiple sub-modules. We'll do this for the task for multi-hop search. Install the latest DSPy via `pip install -U dspy` and follow along. In this tutorial, we'll use a small local LM, Meta's `Llama-3.1-8B-Instruct` which has 8 billion parameters. You might be able to host the 8B model on your laptop with Ollama, on your GPU server with SGLang, or via a provider that hosts it for you like Databricks or Together. In the snippet below, we'll configure this small model as our main LM. We'll also set up a larger LM, i.e. `GPT-4o`, as a teacher that we'll invoke a very small number of times to help teach the small LM. This is technically not necessary; the small model can typically teach itself tasks like this in DSPy. But using a larger teacher will give us some peace of mind, where the initial system or optimizer configuration doesn't matter as much. In \[1\]: Copied! import dspy lm \= dspy.LM('/Llama-3.1-8B-Instruct', max\_tokens\=3000) gpt4o \= dspy.LM('openai/gpt-4o', max\_tokens\=3000) dspy.configure(lm\=lm) import dspy lm = dspy.LM('/Llama-3.1-8B-Instruct', max\_tokens=3000) gpt4o = dspy.LM('openai/gpt-4o', max\_tokens=3000) dspy.configure(lm=lm) ### Install dependencies and download data[¶](#install-dependencies-and-download-data) To do the retrieval, we'll use the cool BM25S library, as it's pretty lightweight. You can replace this components with whatever you like. \> pip install \-U bm25s PyStemmer "jax\[cpu\]" Next, we'll download a snapshot abstracts (i.e., first paragraphs) of all 5,000,000 Wikipedia pages as of 2017. We'll use this as our retrieval corpus. This is 500MB compressed, so the download and decompression may take 2-3 minutes. from dspy.utils import download download("https://huggingface.co/dspy/cache/resolve/main/wiki.abstracts.2017.tar.gz") !tar \-xzvf wiki.abstracts.2017.tar.gz Let's now load the corpus. In \[2\]: Copied! import ujson corpus \= \[\] with open("wiki.abstracts.2017.jsonl") as f: for line in f: line \= ujson.loads(line) corpus.append(f"{line\['title'\]} | {' '.join(line\['text'\])}") len(corpus) import ujson corpus = \[\] with open("wiki.abstracts.2017.jsonl") as f: for line in f: line = ujson.loads(line) corpus.append(f"{line\['title'\]} | {' '.join(line\['text'\])}") len(corpus) Out\[2\]: 5233330 And then let's index it for BM25 retrieval! This will take 2-3 minutes. In \[ \]: Copied! import bm25s import Stemmer stemmer \= Stemmer.Stemmer("english") corpus\_tokens \= bm25s.tokenize(corpus, stopwords\="en", stemmer\=stemmer) retriever \= bm25s.BM25(k1\=0.9, b\=0.4) retriever.index(corpus\_tokens) import bm25s import Stemmer stemmer = Stemmer.Stemmer("english") corpus\_tokens = bm25s.tokenize(corpus, stopwords="en", stemmer=stemmer) retriever = bm25s.BM25(k1=0.9, b=0.4) retriever.index(corpus\_tokens) ### Load the HoVer dataset.[¶](#load-the-hover-dataset) Let's load a dataset for our task. We'll load examples from the HoVer multi-hop task, where the input is a (really!) complex claim and the output we're seeking is the set of Wikipedia pages that are required to fact-check that claim. In \[4\]: Copied! import random from dspy.datasets import DataLoader kwargs \= dict(fields\=("claim", "supporting\_facts", "hpqa\_id", "num\_hops"), input\_keys\=("claim",)) hover \= DataLoader().from\_huggingface(dataset\_name\="hover-nlp/hover", split\="train", trust\_remote\_code\=True, \*\*kwargs) hpqa\_ids \= set() hover \= \[\ dspy.Example(claim\=x.claim, titles\=list(set(\[y\["key"\] for y in x.supporting\_facts\]))).with\_inputs("claim")\ for x in hover\ if x\["num\_hops"\] \== 3 and x\["hpqa\_id"\] not in hpqa\_ids and not hpqa\_ids.add(x\["hpqa\_id"\])\ \] random.Random(0).shuffle(hover) trainset, devset, testset \= hover\[:200\], hover\[200:500\], hover\[650:\] import random from dspy.datasets import DataLoader kwargs = dict(fields=("claim", "supporting\_facts", "hpqa\_id", "num\_hops"), input\_keys=("claim",)) hover = DataLoader().from\_huggingface(dataset\_name="hover-nlp/hover", split="train", trust\_remote\_code=True, \*\*kwargs) hpqa\_ids = set() hover = \[ dspy.Example(claim=x.claim, titles=list(set(\[y\["key"\] for y in x.supporting\_facts\]))).with\_inputs("claim") for x in hover if x\["num\_hops"\] == 3 and x\["hpqa\_id"\] not in hpqa\_ids and not hpqa\_ids.add(x\["hpqa\_id"\]) \] random.Random(0).shuffle(hover) trainset, devset, testset = hover\[:200\], hover\[200:500\], hover\[650:\] Let's view an example of this task: In \[5\]: Copied! example \= trainset\[0\] print("Claim:", example.claim) print("Pages that must be retrieved:", example.titles) example = trainset\[0\] print("Claim:", example.claim) print("Pages that must be retrieved:", example.titles) Claim: This director is known for his work on Miss Potter. The Academy of Motion Picture Arts and Sciences presents the award in which he was nominated for his work in "Babe". Pages that must be retrieved: \['Miss Potter', 'Chris Noonan', 'Academy Award for Best Director'\] Now, let's define a function to do the search in Wikipedia. This will use our BM25 index. In \[6\]: Copied! def search(query: str, k: int) \-> list\[str\]: tokens \= bm25s.tokenize(query, stopwords\="en", stemmer\=stemmer, show\_progress\=False) results, scores \= retriever.retrieve(tokens, k\=k, n\_threads\=1, show\_progress\=False) run \= {corpus\[doc\]: float(score) for doc, score in zip(results\[0\], scores\[0\])} return run def search(query: str, k: int) -> list\[str\]: tokens = bm25s.tokenize(query, stopwords="en", stemmer=stemmer, show\_progress=False) results, scores = retriever.retrieve(tokens, k=k, n\_threads=1, show\_progress=False) run = {corpus\[doc\]: float(score) for doc, score in zip(results\[0\], scores\[0\])} return run Now, let's define the multi-hop program in DSPy. It's going to be super simple: it'll take a `claim` and produce a list `titles: list[str]`. It will do this via two sub-modules: `generate_query` and `append_notes`. In \[7\]: Copied! class Hop(dspy.Module): def \_\_init\_\_(self, num\_docs\=10, num\_hops\=4): self.num\_docs, self.num\_hops \= num\_docs, num\_hops self.generate\_query \= dspy.ChainOfThought('claim, notes -> query') self.append\_notes \= dspy.ChainOfThought('claim, notes, context -> new\_notes: list\[str\], titles: list\[str\]') def forward(self, claim: str) \-> list\[str\]: notes \= \[\] titles \= \[\] for \_ in range(self.num\_hops): query \= self.generate\_query(claim\=claim, notes\=notes).query context \= search(query, k\=self.num\_docs) prediction \= self.append\_notes(claim\=claim, notes\=notes, context\=context) notes.extend(prediction.new\_notes) titles.extend(prediction.titles) return dspy.Prediction(notes\=notes, titles\=list(set(titles))) class Hop(dspy.Module): def \_\_init\_\_(self, num\_docs=10, num\_hops=4): self.num\_docs, self.num\_hops = num\_docs, num\_hops self.generate\_query = dspy.ChainOfThought('claim, notes -> query') self.append\_notes = dspy.ChainOfThought('claim, notes, context -> new\_notes: list\[str\], titles: list\[str\]') def forward(self, claim: str) -> list\[str\]: notes = \[\] titles = \[\] for \_ in range(self.num\_hops): query = self.generate\_query(claim=claim, notes=notes).query context = search(query, k=self.num\_docs) prediction = self.append\_notes(claim=claim, notes=notes, context=context) notes.extend(prediction.new\_notes) titles.extend(prediction.titles) return dspy.Prediction(notes=notes, titles=list(set(titles))) Great. Now let's set up an evaluation metric, `top5_recall`. It will return the fraction of the gold pages (which are always 3) that are retrieved in the top-5 titles returned by the program. In \[8\]: Copied! def top5\_recall(example, pred, trace\=None): gold\_titles \= example.titles recall \= sum(x in pred.titles\[:5\] for x in gold\_titles) / len(gold\_titles) \# If we're "bootstrapping" for optimization, return True if and only if the recall is perfect. if trace is not None: return recall \>= 1.0 \# If we're just doing inference, just measure the recall. return recall evaluate \= dspy.Evaluate(devset\=devset, metric\=top5\_recall, num\_threads\=16, display\_progress\=True, display\_table\=5) def top5\_recall(example, pred, trace=None): gold\_titles = example.titles recall = sum(x in pred.titles\[:5\] for x in gold\_titles) / len(gold\_titles) # If we're "bootstrapping" for optimization, return True if and only if the recall is perfect. if trace is not None: return recall >= 1.0 # If we're just doing inference, just measure the recall. return recall evaluate = dspy.Evaluate(devset=devset, metric=top5\_recall, num\_threads=16, display\_progress=True, display\_table=5) Let's evaluate our off-the-shelf program! In \[9\]: Copied! evaluate(Hop()) evaluate(Hop()) Average Metric: 27.67 / 98 (28.2%): 32%|███▏ | 97/300 \[00:02<00:04, 49.34it/s\] 2024/12/25 12:18:00 ERROR dspy.utils.parallelizer: Error processing item Example({'claim': "All That is the show that the co-creator with the host of Vibe and Wild 'N Out had a debut on.", 'titles': \['Chris Spencer (actor)', 'Nick Cannon', 'Vibe (talk show)'\]}) (input\_keys={'claim'}): Expected dict\_keys(\['reasoning', 'new\_notes', 'titles'\]) but got dict\_keys(\['reasoning', 'new\_notes'\]). Set \`provide\_traceback=True\` to see the stack trace. Average Metric: 59.33 / 186 (31.9%): 62%|██████▏ | 186/300 \[00:03<00:02, 51.84it/s\] 2024/12/25 12:18:02 ERROR dspy.utils.parallelizer: Error processing item Example({'claim': 'The song, which Billie Anthony is best known for her Top 10 hit version, topped the UK chart in 1981 in a recording by a platinum-selling British rock and roll singer whose recording and performing career began in the late 1960s.', 'titles': \["Shakin' Stevens", 'This Ole House', 'Billie Anthony'\]}) (input\_keys={'claim'}): Expected dict\_keys(\['reasoning', 'new\_notes', 'titles'\]) but got dict\_keys(\['reasoning'\]). Set \`provide\_traceback=True\` to see the stack trace. Average Metric: 94.00 / 298 (31.5%): 100%|██████████| 300/300 \[00:06<00:00, 48.56it/s\] 2024/12/25 12:18:04 INFO dspy.evaluate.evaluate: Average Metric: 93.99999999999993 / 300 (31.3%) | | claim | example\_titles | notes | pred\_titles | top5\_recall | titles | | --- | --- | --- | --- | --- | --- | --- | | 0 | Nike football team has had a player endorse the football boot Nike... | \[Nike Hypervenom, Nike Total 90, Marcus Rashford\] | \['The Nike Total 90 has been replaced by the Nike Hypervenom.', 'T... | \['Nike Mercurial Vapor \| The Mercurial Vapor is a football boot ma... | ✔️ \[0.333\] | NaN |\ | 1 | Bill Boyd is the chairman of the appliance company that operates t... | \[Suncoast Hotel and Casino, Boyd Gaming, Thomas Eje\] | \['Bill Boyd is not mentioned as the chairman of an appliance compa... | \[Suncoast Casino, Thomas Eje, Boyd Gaming Corporation, Bill Boyd, ... | ✔️ \[0.333\] | NaN |\ | 2 | The president of South Korea was born 24 January 1953. The group t... | \[Presidential Council on Nation Branding, Korea, Moon Jae-in, Euh ... | \['The president of South Korea was likely born before 1945', 'Euh ... | \['Yi Cheol-seung', 'List of Presidents of South Korea', 'Lifespan ... | | NaN |\ | 3 | The movie Khan Kluay was released 2 months before the 2009 movie t... | \[Fantastic Mr. Fox (film), Jason Schwartzman, Khan Kluay\] | \['The movie Khan Kluay was released in 2006.', 'The 2009 movie tha... | \[Khan Kluay, The Darjeeling Limited\] | ✔️ \[0.333\] | NaN |\ | 4 | The director of Finding Dory co-directed the film A Bug's Life. | \[Andrew Stanton, Finding Dory, A Bug's Life\] | \['The director of Finding Dory is Andrew Stanton and Angus MacLane... | \[Finding Dory, A Bug's Life\] | ✔️ \[0.667\] | NaN |\ \ ... 295 more rows not displayed ...\ \ Out\[9\]:\ \ 31.33\ \ Let's now optimize the two prompts inside the `Hop()` program jointly to maximize the recall of our program. This may take around 35 minutes and make some $5 worth of calls to GPT-4o to optimize Llama-3.1-8B.\ \ In \[10\]:\ \ Copied!\ \ models \= dict(prompt\_model\=gpt4o, teacher\_settings\=dict(lm\=gpt4o))\ tp \= dspy.MIPROv2(metric\=top5\_recall, auto\="medium", num\_threads\=16, \*\*models)\ \ kwargs \= dict(minibatch\_size\=40, minibatch\_full\_eval\_steps\=4, requires\_permission\_to\_run\=False)\ optimized \= tp.compile(Hop(), trainset\=trainset, max\_bootstrapped\_demos\=4, max\_labeled\_demos\=4, \*\*kwargs)\ \ models = dict(prompt\_model=gpt4o, teacher\_settings=dict(lm=gpt4o)) tp = dspy.MIPROv2(metric=top5\_recall, auto="medium", num\_threads=16, \*\*models) kwargs = dict(minibatch\_size=40, minibatch\_full\_eval\_steps=4, requires\_permission\_to\_run=False) optimized = tp.compile(Hop(), trainset=trainset, max\_bootstrapped\_demos=4, max\_labeled\_demos=4, \*\*kwargs)\ \ Let's now evaluate again, after optimization.\ \ In \[11\]:\ \ Copied!\ \ evaluate(optimized)\ \ evaluate(optimized)\ \ Average Metric: 38.67 / 64 (60.4%): 21%|██ | 63/300 \[00:01<00:06, 38.13it/s\]\ \ 2024/12/25 12:18:09 ERROR dspy.utils.parallelizer: Error processing item Example({'claim': 'Eliot Hyman co-founded Seven Arts Productions in 1957. His co-founder produced the American-American black comedy-drama film directed by Stanley Kubrick.', 'titles': \['Ray Stark', 'Seven Arts Productions', 'Lolita (1962 film)'\]}) (input\_keys={'claim'}): Expected dict\_keys(\['reasoning', 'query'\]) but got dict\_keys(\['reasoning'\]). Set \`provide\_traceback=True\` to see the stack trace.\ \ Average Metric: 177.33 / 299 (59.3%): 100%|██████████| 300/300 \[00:08<00:00, 36.01it/s\]\ \ 2024/12/25 12:18:16 INFO dspy.evaluate.evaluate: Average Metric: 177.33333333333334 / 300 (59.1%)\ \ | | claim | example\_titles | notes | pred\_titles | top5\_recall | titles |\ | --- | --- | --- | --- | --- | --- | --- |\ | 0 | Nike football team has had a player endorse the football boot Nike... | \[Nike Hypervenom, Nike Total 90, Marcus Rashford\] | \[\] | \[Nike Hypervenom, Nike Total 90, Kylian Mbappé, Marcus Rashford\] | ✔️ \[1.000\] | NaN |\ | 1 | Bill Boyd is the chairman of the appliance company that operates t... | \[Suncoast Hotel and Casino, Boyd Gaming, Thomas Eje\] | \[\] | \[Bill Boyd, Suncoast Casino, Las Vegas, Thomas Eje\] | ✔️ \[0.333\] | NaN |\ | 2 | The president of South Korea was born 24 January 1953. The group t... | \[Presidential Council on Nation Branding, Korea, Moon Jae-in, Euh ... | \['Euh Yoon-Dae is a South Korean professor, financier, and advisor... | \[Euh Yoon-Dae, KB Financial Group, Chang Dae-hwan, Maeil Business ... | | NaN |\ | 3 | The movie Khan Kluay was released 2 months before the 2009 movie t... | \[Fantastic Mr. Fox (film), Jason Schwartzman, Khan Kluay\] | \["Jason Schwartzman collaborated with Wes Anderson on the 2009 mov... | \[Wes Anderson, Fantastic Mr. Fox, Khan Kluay 2, Jason Schwartzman,... | ✔️ \[0.667\] | NaN |\ | 4 | The director of Finding Dory co-directed the film A Bug's Life. | \[Andrew Stanton, Finding Dory, A Bug's Life\] | \["Andrew Stanton co-directed A Bug's Life", "John Lasseter directe... | \[John Lasseter, Andrew Stanton, Finding Dory, A Bug's Life\] | ✔️ \[1.000\] | NaN |\ \ ... 295 more rows not displayed ...\ \ Out\[11\]:\ \ 59.11\ \ Awesome. It looks like the system improved drastically from around 30% recall to a little below 60% recall. That was a pretty straightforward approach, but DSPy gives you many tools to continue iterating on this from here.\ \ Next, let's inspect the optimized prompts to understand what it has learned. We'll run one query and then inspect the last two prompts, which will show us the prompts used for both sub-modules, in the later iteration inside the `Hop()` program.\ \ In \[13\]:\ \ Copied!\ \ optimized(claim\="The author of the 1960s unproduced script written for The Beatles, Up Against It, and Bernard-Marie Koltès are both playwrights.").titles\ \ optimized(claim="The author of the 1960s unproduced script written for The Beatles, Up Against It, and Bernard-Marie Koltès are both playwrights.").titles\ \ Out\[13\]:\ \ \['Up Against It', 'Bernard-Marie Koltès', 'The Beatles', 'Joe Orton'\]\ \ In \[15\]:\ \ Copied!\ \ dspy.inspect\_history(n\=2)\ \ dspy.inspect\_history(n=2)\ \ \[2024-12-25T12:18:16.177899\]\ \ System message:\ \ Your input fields are:\ 1. \`claim\` (str)\ 2. \`notes\` (str)\ \ Your output fields are:\ 1. \`reasoning\` (str)\ 2. \`query\` (str)\ \ All interactions will be structured in the following way, with the appropriate values filled in.\ \ \[\[ ## claim ## \]\]\ {claim}\ \ \[\[ ## notes ## \]\]\ {notes}\ \ \[\[ ## reasoning ## \]\]\ {reasoning}\ \ \[\[ ## query ## \]\]\ {query}\ \ \[\[ ## completed ## \]\]\ \ In adhering to this structure, your objective is: \ Given a claim and a set of notes, generate a query that can be used to gather additional evidence or context to support or refute the claim. Think step by step to ensure the query is specific and relevant to the information provided in the notes.\ \ User message:\ \ \[\[ ## claim ## \]\]\ Danyang, Jiangusu and this city are both cities in China. This city was the birthplace of Chen Xiuke.\ \ \[\[ ## notes ## \]\]\ \[1\] «Chen Xiuke was born in Dongfang, Hainan.»\ \[2\] «Danyang is a city in Jiangsu province, China.»\ \[3\] «Chen Xiuke was born in Dongfang, Hainan.»\ \[4\] «Danyang is a city in Jiangsu province, China.»\ \[5\] «Dongfang is a city in Hainan province, China.»\ \[6\] «Chen Xiuke was born in Dongfang, Hainan.»\ \[7\] «Danyang is a city in Jiangsu province, China.»\ \[8\] «Dongfang is a city in Hainan province, China.»\ \[9\] «Chen Xiuke was born in Dongfang, Hainan.»\ \[10\] «Danyang is a city in Jiangsu province, China.»\ \[11\] «Dongfang is a city in Hainan province, China.»\ \ Respond with the corresponding output fields, starting with the field \`\[\[ ## reasoning ## \]\]\`, then \`\[\[ ## query ## \]\]\`, and then ending with the marker for \`\[\[ ## completed ## \]\]\`.\ \ Assistant message:\ \ \[\[ ## reasoning ## \]\]\ The claim states that Danyang, Jiangsu, and another city are both cities in China, and that this other city was the birthplace of Chen Xiuke. According to the notes, Chen Xiuke was born in Dongfang, Hainan. Therefore, the city referred to in the claim as "this city" is Dongfang, which is indeed a city in Hainan province, China. The claim is consistent with the information provided in the notes.\ \ \[\[ ## query ## \]\]\ What is the birthplace of Chen Xiuke?\ \ \[\[ ## completed ## \]\]\ \ User message:\ \ \[\[ ## claim ## \]\]\ The star of Nothing to Report and Gary Barlow have a profession in common.\ \ \[\[ ## notes ## \]\]\ \[1\] «Chris Jericho is a musician and professional wrestler.»\ \[2\] «Gary Barlow is a singer and songwriter, known for being a member of Take That.»\ \[3\] «Chris Jericho is a musician and professional wrestler.»\ \[4\] «Gary Barlow is a singer and songwriter, known for being a member of Take That.»\ \[5\] «Chris Jericho stars in 'Nothing to Report'.»\ \[6\] «Chris Jericho is a musician and professional wrestler.»\ \[7\] «Gary Barlow is a singer and songwriter, known for being a member of Take That.»\ \[8\] «Chris Jericho stars in 'Nothing to Report'.»\ \[9\] «Chris Jericho is a musician and professional wrestler.»\ \[10\] «Gary Barlow is a singer and songwriter, known for being a member of Take That.»\ \[11\] «Chris Jericho stars in 'Nothing to Report'.»\ \ Respond with the corresponding output fields, starting with the field \`\[\[ ## reasoning ## \]\]\`, then \`\[\[ ## query ## \]\]\`, and then ending with the marker for \`\[\[ ## completed ## \]\]\`.\ \ Assistant message:\ \ \[\[ ## reasoning ## \]\]\ The claim suggests that the star of "Nothing to Report" and Gary Barlow share a profession. From the notes, we know that Chris Jericho stars in "Nothing to Report" and is a musician and professional wrestler. Gary Barlow is a singer and songwriter. The common profession between Chris Jericho and Gary Barlow is that they are both musicians.\ \ \[\[ ## query ## \]\]\ What is the profession that Chris Jericho and Gary Barlow have in common?\ \ \[\[ ## completed ## \]\]\ \ User message:\ \ \[\[ ## claim ## \]\]\ The author of the 1960s unproduced script written for The Beatles, Up Against It, and Bernard-Marie Koltès are both playwrights.\ \ \[\[ ## notes ## \]\]\ N/A\ \ Respond with the corresponding output fields, starting with the field \`\[\[ ## reasoning ## \]\]\`, then \`\[\[ ## query ## \]\]\`, and then ending with the marker for \`\[\[ ## completed ## \]\]\`.\ \ Response:\ \ \[\[ ## reasoning ## \]\]\ The claim states that the author of the 1960s unproduced script written for The Beatles, Up Against It, and Bernard-Marie Koltès are both playwrights. However, there is no information in the notes to support or refute this claim. The notes are empty, which means we cannot verify the author of the script or Bernard-Marie Koltès' profession.\ \ \[\[ ## query ## \]\]\ Who is the author of the 1960s unproduced script written for The Beatles, Up Against It?\ \ \[\[ ## completed ## \]\]\ \ \[2024-12-25T12:18:16.191896\]\ \ System message:\ \ Your input fields are:\ 1. \`claim\` (str)\ 2. \`notes\` (str)\ 3. \`context\` (str)\ \ Your output fields are:\ 1. \`reasoning\` (str)\ 2. \`new\_notes\` (list\[str\])\ 3. \`titles\` (list\[str\])\ \ All interactions will be structured in the following way, with the appropriate values filled in.\ \ \[\[ ## claim ## \]\]\ {claim}\ \ \[\[ ## notes ## \]\]\ {notes}\ \ \[\[ ## context ## \]\]\ {context}\ \ \[\[ ## reasoning ## \]\]\ {reasoning}\ \ \[\[ ## new\_notes ## \]\]\ {new\_notes} # note: the value you produce must be pareseable according to the following JSON schema: {"type": "array", "items": {"type": "string"}}\ \ \[\[ ## titles ## \]\]\ {titles} # note: the value you produce must be pareseable according to the following JSON schema: {"type": "array", "items": {"type": "string"}}\ \ \[\[ ## completed ## \]\]\ \ In adhering to this structure, your objective is: \ Analyze the provided \`claim\`, \`notes\`, and \`context\` to generate \`new\_notes\` that further support or refute the claim, and extract relevant \`titles\` that represent key topics or entities from the context.\ \ User message:\ \ This is an example of the task, though some input or output fields are not supplied.\ \ \[\[ ## claim ## \]\]\ The figure who currently holds the role Michael Noonan held from 2001 to 2002, was the Minister for Social Protection from 2016 until 2017.\ \ \[\[ ## notes ## \]\]\ Not supplied for this particular example.\ \ \[\[ ## context ## \]\]\ Not supplied for this particular example.\ \ Respond with the corresponding output fields, starting with the field \`\[\[ ## reasoning ## \]\]\`, then \`\[\[ ## new\_notes ## \]\]\` (must be formatted as a valid Python list\[str\]), then \`\[\[ ## titles ## \]\]\` (must be formatted as a valid Python list\[str\]), and then ending with the marker for \`\[\[ ## completed ## \]\]\`.\ \ Assistant message:\ \ \[\[ ## reasoning ## \]\]\ Not supplied for this particular example.\ \ \[\[ ## new\_notes ## \]\]\ Not supplied for this particular example.\ \ \[\[ ## titles ## \]\]\ \["Michael Noonan", "Leader of Fine Gael", "Leo Varadkar"\]\ \ \[\[ ## completed ## \]\]\ \ User message:\ \ This is an example of the task, though some input or output fields are not supplied.\ \ \[\[ ## claim ## \]\]\ The type of area Wiliwili are typically found in have a dominant tree species of Acacia koa. They are of the pea family.\ \ \[\[ ## notes ## \]\]\ Not supplied for this particular example.\ \ \[\[ ## context ## \]\]\ Not supplied for this particular example.\ \ Respond with the corresponding output fields, starting with the field \`\[\[ ## reasoning ## \]\]\`, then \`\[\[ ## new\_notes ## \]\]\` (must be formatted as a valid Python list\[str\]), then \`\[\[ ## titles ## \]\]\` (must be formatted as a valid Python list\[str\]), and then ending with the marker for \`\[\[ ## completed ## \]\]\`.\ \ Assistant message:\ \ \[\[ ## reasoning ## \]\]\ Not supplied for this particular example.\ \ \[\[ ## new\_notes ## \]\]\ Not supplied for this particular example.\ \ \[\[ ## titles ## \]\]\ \["Acacia koa", "Wiliwili", "Hawaiian tropical dry forests"\]\ \ \[\[ ## completed ## \]\]\ \ User message:\ \ \[\[ ## claim ## \]\]\ The father of Zak Ové and A. Edward Sutherland are not both photographers.\ \ \[\[ ## notes ## \]\]\ \[1\] «Horace Ové is a photographer, filmmaker, and writer.»\ \[2\] «A. Edward Sutherland is a film director, not a photographer.»\ \[3\] «Horace Ové is a photographer, filmmaker, and writer.»\ \[4\] «A. Edward Sutherland is a film director, not a photographer.»\ \[5\] «Horace Ové is a photographer, filmmaker, and writer.»\ \[6\] «A. Edward Sutherland is a film director, not a photographer.»\ \[7\] «Horace Ové is a photographer, filmmaker, and writer.»\ \[8\] «A. Edward Sutherland is a film director, not a photographer.»\ \ \[\[ ## context ## \]\]\ {"Horace Ové | Horace Ové, CBE (born 1939), is a British filmmaker, photographer, painter and writer, one of the leading black independent film-makers to emerge in Britain since the post-war period. Ové holds the \\"Guinness World Record\\" for being the first black British film-maker to direct a feature-length film, \\"Pressure\\" (1975). In its retrospective history, \\"100 Years of Cinema\\", the British Film Institute (BFI) declared: \\"Horace Ové is undoubtedly a pioneer in Black British history and his work provides a perspective on the Black experience in Britain.\\"": 15.836545944213867, "Zak Ové | Zak Ové (born 1966) is a British visual artist who works between sculpture, film and photography, living in London, UK, and Trinidad. His themes reflect \\"his documentation of and anthropological interest in diasporic and African history, specifically that which is explored through Trinidadian carnival.\\" In work that is \\"filtered through his own personal and cultural upbringing, with a black Trinidadian father and white Irish mother\\", he has exhibited widely in Europe, the United States and Africa, participating in international museum shows in London, Dakar, Paris, Dubai, Prague, Berlin, Johannesburg, Bamako and New York City. His father is the filmmaker Horace Ové and his sister is the actress Indra Ové.": 13.145259857177734, "Playing Away | Playing Away is a 1987 TV comedy film directed by Horace Ové, from a screenplay by Caryl Phillips. In the story, an English cricket team, fictitiously named \\"Sneddington\\" (based in Lavenham, Suffolk), invites a team of West Indian heritage based in Brixton (South London) to play a charity game in support of their \\"Third World Week.\\" According to Screenonline, \\"The gentle comedy of manners and unexpected reversal of white and black stereotypes in \\"Playing Away\\" contrasts sharply with the stylistic experimentation and the militant denunciations of racial prejudice in director Horace Ové's earlier feature, \\"Pressure\\" (1975).\\" \\" New York Times\\" reviewer Vincent Canby called it \\"witty and wise without being seriously disturbing for a minute\\".": 12.445182800292969, "Pressure (film) | Pressure is a 1976 British drama film and the first feature-length fiction film directed by a Black film-maker in Britain. Directed by Horace Ové, and co-written by him with Samuel Selvon, \\"Pressure\\" is a powerful portrait of inter-generational tensions between first- and second-generation West Indian migrants in London's Notting Hill area. According to Julia Toppin,": 10.526924133300781, "What a Night! (1928 film) | What a Night! (1928) is an American silent film directed by A. Edward Sutherland. The romantic comedy was written by Louise Long, from a story by Lloyd Corrigan and Grover Jones. The film stars Bebe Daniels, Neil Hamilton, and William Austin.": 10.453304290771484, "The Orchid House (TV serial) | The Orchid House is a four-part television serial that first aired on British television's Channel 4 from 21 February to 14 March 1991, directed by Horace Ové. Its cast featured Diana Quick, Madge Sinclair, Nigel Terry, Elizabeth Hurley, Kate Buffery and Frances Barber, and was based on Phyllis Shand Allfrey's only novel of the same name.": 10.413471221923828, "Indra Ové | Indra Ové (born 1968) is a British movie and stage actress.": 10.15088176727295, "Henry Sutherland Edwards | Henry Sutherland Edwards (1828–1906) was a British journalist.": 9.868606567382812, "Burning an Illusion | Burning an Illusion is a 1981 British film written and directed by Menelik Shabazz, about a young British-born black woman's love life, mostly shot in London's Notting Hill and Ladbroke Grove communities. It was only the second British feature to have been made by a black director, following Horace Ové’s 1975 \\"Pressure\\", and is described by Stephen Bourne as \\"the first British film to give a black woman a voice of any kind.\\" Imruh Bakari worked with Shabazz and co-founded Kumba productions with him.": 9.634703636169434, "Horace Hall Edwards | Horace Hall Edwards (August 21, 1902 – January 27, 1987) was an American politician, who was mayor of Richmond, Virginia, and a candidate for governor in 1949.": 9.451831817626953}\ \ Respond with the corresponding output fields, starting with the field \`\[\[ ## reasoning ## \]\]\`, then \`\[\[ ## new\_notes ## \]\]\` (must be formatted as a valid Python list\[str\]), then \`\[\[ ## titles ## \]\]\` (must be formatted as a valid Python list\[str\]), and then ending with the marker for \`\[\[ ## completed ## \]\]\`.\ \ Assistant message:\ \ \[\[ ## reasoning ## \]\]\ The claim states that the father of Zak Ové and A. Edward Sutherland are not both photographers. From the context, we know that Zak Ové's father is Horace Ové, who is indeed a photographer among other professions. However, A. Edward Sutherland is not mentioned as a photographer in the context; he is known as a film director. Therefore, the claim is correct because Horace Ové is a photographer, but A. Edward Sutherland is not.\ \ \[\[ ## new\_notes ## \]\]\ \["Horace Ové is a photographer, filmmaker, and writer.", "A. Edward Sutherland is a film director, not a photographer."\]\ \ \[\[ ## titles ## \]\]\ \["Horace Ové", "A. Edward Sutherland", "Zak Ové"\]\ \ \[\[ ## completed ## \]\]\ \ User message:\ \ \[\[ ## claim ## \]\]\ The summer 2016 romantic drama "Me Before You" is directed by Thea Sharrock. The star of the film The Lost Future (who also appears in The Hunger Games) stars as the character Will Traynor.\ \ \[\[ ## notes ## \]\]\ \[1\] «Sam Claflin stars as Will Traynor in 'Me Before You'.»\ \[2\] «Sam Claflin is known for his role as Finnick Odair in 'The Hunger Games'.»\ \[3\] «Thea Sharrock directed 'Me Before You'.»\ \[4\] «Sam Claflin stars as Will Traynor in 'Me Before You'.»\ \[5\] «Sam Claflin is known for his role as Finnick Odair in 'The Hunger Games'.»\ \[6\] «Thea Sharrock directed 'Me Before You'.»\ \[7\] «Sam Claflin starred in 'The Lost Future'.»\ \[8\] «Sam Claflin stars as Will Traynor in 'Me Before You'.»\ \[9\] «Sam Claflin is known for his role as Finnick Odair in 'The Hunger Games'.»\ \[10\] «Thea Sharrock directed 'Me Before You'.»\ \[11\] «Sam Claflin starred in 'The Lost Future'.»\ \[12\] «Sam Claflin stars as Will Traynor in 'Me Before You'.»\ \[13\] «Sam Claflin is known for his role as Finnick Odair in 'The Hunger Games'.»\ \[14\] «Thea Sharrock directed 'Me Before You'.»\ \[15\] «Sam Claflin starred in 'The Lost Future'.»\ \ \[\[ ## context ## \]\]\ {"Sam Claflin | Samuel George Claflin (born 27 June 1986) is an English actor. He is known for portraying Finnick Odair in \\"The Hunger Games\\" film series, Philip Swift in \\"\\", and Will Traynor in \\"Me Before You\\".": 19.94539451599121, "Me Before You (film) | Me Before You is a 2016 romantic drama film directed by Thea Sharrock in her directorial debut and adapted by English author Jojo Moyes from her 2012 novel of the same name. The film stars Emilia Clarke, Sam Claflin, Steve Peacocke, Jenna Coleman, Charles Dance, Matthew Lewis, Janet McTeer, Vanessa Kirby and Joanna Lumley.": 18.48834228515625, "Look What You Did to Me | Look What You Did to Me is the debut studio album by American rapper and singer Z-Ro. It was released on June 16, 1998, by this independently distributed label Fisherboy Records. The album is entirely produced by Z-Ro alongside Rakish Jacob (aka Roc), while the guest appearances was from T.A.Z., Bam, Trae, The Fakkulty, Chris Ward and Al-D.": 14.100790977478027, "I Still Know What You Did Last Summer | I Still Know What You Did Last Summer is a 1998 American slasher film and a sequel to the 1997 film \\"I Know What You Did Last Summer\\". Directed by Danny Cannon, the film was written by Trey Callaway, and features characters originally created in Lois Duncan's 1973 novel \\"I Know What You Did Last Summer\\". Jennifer Love Hewitt, Freddie Prinze, Jr. and Muse Watson reprise their roles, with Brandy, Mekhi Phifer, Jennifer Esposito, and Matthew Settle joining the cast. \\"I Still Know What You Did Last Summer\\" continues after the events of the first film.": 13.990736961364746, "Tell Me What You Dream | \\"Tell Me What You Dream\\" is a song written by Timothy B. Schmit, Josh Leo and Vince Melamed and performed by country group Restless Heart along with saxophonist Warren Hill. The single was the group's only number one on the adult contemporary chart and despite previous country chart success, the song did not make the country top 40. \\"Tell Me What You Dream\\" spent two weeks at number one and peaked at number forty-three on the \\"Billboard\\" Hot 100.": 13.774335861206055, "Geraldine Jones (character) | Geraldine Jones was a fictional African American character, the most famous recurring persona of comedian Flip Wilson. Geraldine was played as a sassy liberated Southern woman who was coarsely flirty yet faithful to her (unseen) boyfriend \\"Killer\\". Poorly educated, she was nevertheless confident; she did not change her behavior to suit anyone. Several of Geraldine's sayings entered U.S. popular culture as catchphrases, especially \\"When you're hot, you're hot; when you're not, you're not,\\" \\"The Devil made me do it,\\" and \\"What you see is what you get! \\"": 13.149161338806152, "When Did You Stop Loving Me, When Did I Stop Loving You | When Did You Stop Loving Me, When Did I Stop Loving You": 12.946025848388672, "Pajama Sam 3: You Are What You Eat from Your Head to Your Feet | Pajama Sam 3: You Are What You Eat from Your Head to Your Feet": 12.687528610229492, "Sam Quartin | Sam Quartin is a musician, producer and actress best known for role as June Glass, starring alongside Marilyn Manson in the American movie Let Me Make You a Martyr\\"(2017), also for \\"By the Rivers of Babylon)\\" and \\"Aimy in a Cage\\".": 12.639335632324219, "Why You Wanna Treat Me So Bad? | \\"Why You Wanna Treat Me So Bad?\\" is the US follow-up single to Prince's first big hit, \\"I Wanna Be Your Lover\\". It is also Prince's first rock and roll-inspired single release. It did not make the top 40 of the Billboard Hot 100 charts, although it did reach #13 on the R&B Singles charts. The lyrics are self-explanatory from the title, and deal with a cruel lover. The song prominently features guitar and bass, with the keyboard featured in a less prominent role. A highlight of the song is a soaring guitar solo at the end, played by Prince himself.": 12.574298858642578}\ \ Respond with the corresponding output fields, starting with the field \`\[\[ ## reasoning ## \]\]\`, then \`\[\[ ## new\_notes ## \]\]\` (must be formatted as a valid Python list\[str\]), then \`\[\[ ## titles ## \]\]\` (must be formatted as a valid Python list\[str\]), and then ending with the marker for \`\[\[ ## completed ## \]\]\`.\ \ Assistant message:\ \ \[\[ ## reasoning ## \]\]\ The claim states that the film "Me Before You" is directed by Thea Sharrock and that the actor who stars as Will Traynor also appeared in "The Lost Future" and "The Hunger Games". According to the notes, Sam Claflin stars as Will Traynor in "Me Before You" and is known for his role as Finnick Odair in "The Hunger Games". Additionally, Sam Claflin starred in "The Lost Future". The context confirms that "Me Before You" is directed by Thea Sharrock and that Sam Claflin plays Will Traynor. Therefore, the claim is supported by the provided notes and context.\ \ \[\[ ## new\_notes ## \]\]\ \["Sam Claflin stars as Will Traynor in 'Me Before You'.", "Sam Claflin is known for his role as Finnick Odair in 'The Hunger Games'.", "Thea Sharrock directed 'Me Before You'.", "Sam Claflin starred in 'The Lost Future'."\]\ \ \[\[ ## titles ## \]\]\ \["Me Before You (film)", "Sam Claflin", "Thea Sharrock", "The Lost Future", "The Hunger Games"\]\ \ \[\[ ## completed ## \]\]\ \ User message:\ \ \[\[ ## claim ## \]\]\ The author of the 1960s unproduced script written for The Beatles, Up Against It, and Bernard-Marie Koltès are both playwrights.\ \ \[\[ ## notes ## \]\]\ N/A\ \ \[\[ ## context ## \]\]\ {"Up Against It | Up Against It is an unproduced script by Joe Orton, written in 1967 for The Beatles at the height of their fame.": 23.741416931152344, "The Alien (unproduced film) | The Alien was an unproduced Indian-American science fiction film in development in the late 1960s which was eventually cancelled. It was to be directed by celebrated Indian filmmaker Satyajit Ray and co-produced by Columbia Pictures. The script was written by Ray in 1967, loosely based on Bankubabur Bandhu (\\"Banku Babu's Friend\\" or \\"Mr. Banku's Friend\\"), a Bengali science fiction story he had written in 1962 for \\"Sandesh\\", the Ray family magazine, which gained popularity among Bengalis in the early 1960s. \\"Bankubabur Bandhu\\" was eventually adapted into a television film by Satyajit Ray's son Sandip Ray, and a play by the theatre group Swapnasandhani Kaushik Sen, in 2006.": 14.683004379272461, "Marsha Albert | Marsha Albert (born 1948) is credited with being the person who jump-started the early 1960s phenomena known as Beatlemania in the United States when as a 15-year-old girl, on 17 December 1963, she introduced for the first time on American radio a song written and recorded by The Beatles titled I Want to Hold Your Hand (the Beatles' best-selling single worldwide), and that Beatles historian and author Bruce Spizer noted, in 2004, by his stating \\"Marsha Albert's actions forced a major record company to push up the release date of a debut single from an unknown band during the holiday season, a time when record companies traditionally released no new product.\\"": 13.236483573913574, "Up Against It! (Todd Rundgren album) | Up Against It! is a 1997 album by Todd Rundgren, essentially consisting of Rundgren's song demos for the Off Broadway show that were Written and Recorded by Todd from 1986-88. The project was inspired by the never-produced Up Against It which was a play originally written by Joe Orton for The Beatles.": 12.989564895629883, "Blood and Fire (Star Trek: The Next Generation) | \\"Blood and Fire\\" is an episode written by David Gerrold for possible use on \\"\\". The script was commissioned and written, but never actually filmed. According to Gerrold, some of the production staff, including Rick Berman, had a negative reaction to its positive depiction of an openly gay couple. Herbert Wright rewrote the script under the name \\"Blood and Ice\\", which also was left unproduced.": 11.980508804321289, "Cedar Rapids (film) | Cedar Rapids is a 2011 American comedy film directed by Miguel Arteta. The script, written by Phil Johnston, was included on the 2009 Black List, a Hollywood list of the most popular unproduced screenplays of the year.": 11.593443870544434, "The Beatles: The Biography | The Beatles: The Biography is the name of a 2005 biography of the 1960s rock band The Beatles written by Bob Spitz. It was first published by Little, Brown and Company on November 1, 2005.": 11.583497047424316, "The Illusionist (2010 film) | The Illusionist (French: L'Illusionniste ) is a 2010 French-British animated film directed by Sylvain Chomet. The film is based on an unproduced script written by French mime, director and actor Jacques Tati in 1956. Controversy surrounds Tati's motivation for the script, which was written as a personal letter to his estranged eldest daughter, Helga Marie-Jeanne Schiel in collaboration with his long-term writing partner Henri Marquet, between writing for the films \\"Mon Oncle\\" and \\"Play Time\\".": 11.528533935546875, "Bobby Florsheim | Robert Florsheim (born December 19, 1969) is an American screenwriter, best known for co-writing \\"The Passion Of The Ark\\" with Josh Stolberg, (the basis for the film \\"Evan Almighty\\"). Their original script was reported by Daily Variety as the highest priced spec script ever sold by unproduced screenwriters (April, 2004). In refashioning the script into \\"Evan Almighty\\", Universal Studios discarded the script and then hired screenwriter Steve Oedekerk, who received sole writing credit on the finished film.": 11.23757553100586, "The Hook (screenplay) | The Hook is an unproduced screenplay by American playwright, Arthur Miller. It was written in 1947 and was intended to be produced by Columbia Pictures Studio, Hollywood, and to be directed by Elia Kazan. The screenplay was inspired by the true story of Pete Panto, a young dockworker who stood up against the corrupt Mafia-connected union leadership. Panto was discovered dead in a pit outside New York eighteen months after his disappearance. Set in the Red Hook district of Brooklyn, \\"The Hook\\" is the story of Marty Ferrara, a longshoreman who is ‘ready to lay down his life, if need be, to secure one thing – his sense of personal dignity.’": 11.222980499267578}\ \ Respond with the corresponding output fields, starting with the field \`\[\[ ## reasoning ## \]\]\`, then \`\[\[ ## new\_notes ## \]\]\` (must be formatted as a valid Python list\[str\]), then \`\[\[ ## titles ## \]\]\` (must be formatted as a valid Python list\[str\]), and then ending with the marker for \`\[\[ ## completed ## \]\]\`.\ \ Response:\ \ \[\[ ## reasoning ## \]\]\ The claim states that the author of the 1960s unproduced script written for The Beatles, "Up Against It", and Bernard-Marie Koltès are both playwrights. The context confirms that Joe Orton wrote the script "Up Against It" for The Beatles, and Bernard-Marie Koltès was a French playwright. Therefore, the claim is supported by the provided context.\ \ \[\[ ## new\_notes ## \]\]\ \[\]\ \ \[\[ ## titles ## \]\]\ \["Up Against It", "Bernard-Marie Koltès", "Joe Orton", "The Beatles"\]\ \ \[\[ ## completed ## \]\]\ \ Finally, let's save our optimized program so we can use it again later.\ \ In \[16\]:\ \ Copied!\ \ optimized.save("optimized\_hop.json")\ \ loaded\_program \= Hop()\ loaded\_program.load("optimized\_hop.json")\ \ loaded\_program(claim\="The author of the 1960s unproduced script written for The Beatles, Up Against It, and Bernard-Marie Koltès are both playwrights.").titles\ \ optimized.save("optimized\_hop.json") loaded\_program = Hop() loaded\_program.load("optimized\_hop.json") loaded\_program(claim="The author of the 1960s unproduced script written for The Beatles, Up Against It, and Bernard-Marie Koltès are both playwrights.").titles\ \ Out\[16\]:\ \ \['Up Against It', 'Bernard-Marie Koltès', 'The Beatles', 'Joe Orton'\]\ \ Back to top --- # Finetuning Agents - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/tutorials/games/index.ipynb "Edit this page") Tutorial: Fine-tuning Agents[¶](#tutorial-fine-tuning-agents) ============================================================== Let's walk through a quick example of optimizing the _language model weights_ (i.e., fine-tuning) inside a DSPy module that represents a ReAct agent playing a game with 50-step tasks. ### Install dependencies and download data[¶](#install-dependencies-and-download-data) Install the latest DSPy via `pip install -U --pre dspy` and follow along. This tutorial uses the AlfWorld dataset, which depends on DSPy 2.6.0 (pre-release). You will also need the following dependencies: \> pip install \-U alfworld\==0.3.5 multiprocess > alfworld-download ### Set up the language models[¶](#set-up-the-language-models) Our goal is to allow `gpt-4o-mini` to play the AlfWorld household game proficiently, without tinkering with string prompts or example trajectories by hand. Though it's not strictly necessary, we'll make our job a little easier by using the larger `gpt-4o` for prompt optimization and fine-tuning, building our small `gpt-4o-mini` agent. In \[1\]: Copied! import dspy gpt4o\_mini \= dspy.LM('gpt-4o-mini-2024-07-18') gpt4o \= dspy.LM('openai/gpt-4o') dspy.configure(experimental\=True) import dspy gpt4o\_mini = dspy.LM('gpt-4o-mini-2024-07-18') gpt4o = dspy.LM('openai/gpt-4o') dspy.configure(experimental=True) Let's load 200 training and 200 development tasks from AlfWorld. The dataset is much larger, but a small number of examples will help keep this tutorial run in 1-2 hours, including fine-tuning. With just 100 training tasks, we'll teach 4o-mini to go from 19% (can barely play the game) to 72%. If you use 500 tasks and retain the demonstrations during fine-tuning, you can push that easily to 82%. In \[2\]: Copied! from dspy.datasets.alfworld import AlfWorld alfworld \= AlfWorld() trainset, devset \= alfworld.trainset\[:200\], alfworld.devset\[\-200:\] len(trainset), len(devset) from dspy.datasets.alfworld import AlfWorld alfworld = AlfWorld() trainset, devset = alfworld.trainset\[:200\], alfworld.devset\[-200:\] len(trainset), len(devset) Out\[2\]: (200, 200) Before we proceed, let's view an example of this task. In \[3\]: Copied! example \= trainset\[0\] with alfworld.POOL.session() as env: task, info \= env.init(\*\*example.inputs()) print(task) example = trainset\[0\] with alfworld.POOL.session() as env: task, info = env.init(\*\*example.inputs()) print(task) \-= Welcome to TextWorld, ALFRED! =- You are in the middle of a room. Looking quickly around you, you see a countertop 1, a drawer 8, a drawer 7, a drawer 6, a drawer 5, a drawer 4, a drawer 3, a drawer 2, a drawer 1, a garbagecan 1, a handtowelholder 1, a sinkbasin 2, a sinkbasin 1, a toilet 1, a toiletpaperhanger 1, and a towelholder 1. Your task is to: put a clean soapbar in garbagecan. ### Defining the Agent program[¶](#defining-the-agent-program) The agent is a pretty simple `dspy.Module` with one sub-module called `self.react`. This sub-module consumes a definition of a specific `task`, sees its previous `trajectory`, and sees a list of `possible_actions` it can take. It responds simply with the next action. In the `forward` method, we just initialize an environment for the given task `idx`. And we loop up to `self.max_iters`, repeatedly invoking the `self.react` module to take the next action. In \[4\]: Copied! class Agent(dspy.Module): def \_\_init\_\_(self, max\_iters\=50, verbose\=False): self.max\_iters \= max\_iters self.verbose \= verbose self.react \= dspy.Predict("task, trajectory, possible\_actions: list\[str\] -> action") def forward(self, idx): with alfworld.POOL.session() as env: trajectory \= \[\] task, info \= env.init(idx) if self.verbose: print(f"Task: {task}") for \_ in range(self.max\_iters): trajectory\_ \= "\\n".join(trajectory) possible\_actions \= info\["admissible\_commands"\]\[0\] + \["think: ${...thoughts...}"\] prediction \= self.react(task\=task, trajectory\=trajectory\_, possible\_actions\=possible\_actions) trajectory.append(f"> {prediction.action}") if prediction.action.startswith("think:"): trajectory.append("OK.") continue obs, reward, done, info \= env.step(prediction.action) obs, reward, done \= obs\[0\], reward\[0\], done\[0\] trajectory.append(obs) if self.verbose: print("\\n".join(trajectory\[\-2:\])) if done: break assert reward \== int(info\["won"\]\[0\]), (reward, info\["won"\]\[0\]) return dspy.Prediction(trajecotry\=trajectory, success\=reward) class Agent(dspy.Module): def \_\_init\_\_(self, max\_iters=50, verbose=False): self.max\_iters = max\_iters self.verbose = verbose self.react = dspy.Predict("task, trajectory, possible\_actions: list\[str\] -> action") def forward(self, idx): with alfworld.POOL.session() as env: trajectory = \[\] task, info = env.init(idx) if self.verbose: print(f"Task: {task}") for \_ in range(self.max\_iters): trajectory\_ = "\\n".join(trajectory) possible\_actions = info\["admissible\_commands"\]\[0\] + \["think: ${...thoughts...}"\] prediction = self.react(task=task, trajectory=trajectory\_, possible\_actions=possible\_actions) trajectory.append(f"> {prediction.action}") if prediction.action.startswith("think:"): trajectory.append("OK.") continue obs, reward, done, info = env.step(prediction.action) obs, reward, done = obs\[0\], reward\[0\], done\[0\] trajectory.append(obs) if self.verbose: print("\\n".join(trajectory\[-2:\])) if done: break assert reward == int(info\["won"\]\[0\]), (reward, info\["won"\]\[0\]) return dspy.Prediction(trajecotry=trajectory, success=reward) #### Aside: If you wanted to include instructions for your agent...[¶](#aside-if-you-wanted-to-include-instructions-for-your-agent) Above, we opted to keep the agent super simple, without even providing short instructions that describe the task. In principle, you can copy a short definition of the AlfWorld task (based on Yao et al., 2022) and use that as the instruction for your agent. This is not inherently essential, but it helps illustrate the role that instructions play in DSPy: they're not for coercing the model to exhibit a certain behavior, but they're there to describe the fundamentals of the task in a straightforward, human-readable way. If you want to do that, you can simply replace this: self.react \= dspy.Predict("task, trajectory, possible\_actions: list\[str\] -> action") with this: INSTRUCTIONS \= """ Interact with a simulated household to achieve a high-level goal. Make sure to plan, track subgoals, determine likely locations for common household items (e.g. desklamps will likely be on desks, shelfs, or dressers), and explore systematically (e.g. check all desks one by one for desklamp). """.strip() self.react \= dspy.Predict(dspy.Signature("task, trajectory, possible\_actions: list\[str\] -> action", INSTRUCTIONS)) ### Zero-shot evaluation[¶](#zero-shot-evaluation) Now, let's try this simple program, prior to any optimization work. In \[5\]: Copied! agent\_4o \= Agent() agent\_4o.set\_lm(gpt4o) agent\_4o.verbose \= True agent\_4o(\*\*example.inputs()) agent\_4o = Agent() agent\_4o.set\_lm(gpt4o) agent\_4o.verbose = True agent\_4o(\*\*example.inputs()) Task: -= Welcome to TextWorld, ALFRED! =- You are in the middle of a room. Looking quickly around you, you see a countertop 1, a drawer 8, a drawer 7, a drawer 6, a drawer 5, a drawer 4, a drawer 3, a drawer 2, a drawer 1, a garbagecan 1, a handtowelholder 1, a sinkbasin 2, a sinkbasin 1, a toilet 1, a toiletpaperhanger 1, and a towelholder 1. Your task is to: put a clean soapbar in garbagecan. > go to countertop 1 You arrive at countertop 1. On the countertop 1, you see a candle 1, a soapbar 1, a soapbottle 2, a soapbottle 1, and a spraybottle 1. > take soapbar 1 from countertop 1 You pick up the soapbar 1 from the countertop 1. > go to garbagecan 1 You arrive at garbagecan 1. On the garbagecan 1, you see nothing. > move soapbar 1 to garbagecan 1 You move the soapbar 1 to the garbagecan 1. > examine garbagecan 1 On the garbagecan 1, you see a soapbar 1. > examine garbagecan 1 On the garbagecan 1, you see a soapbar 1. > examine garbagecan 1 On the garbagecan 1, you see a soapbar 1. > examine garbagecan 1 On the garbagecan 1, you see a soapbar 1. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > take soapbar 1 from garbagecan 1 You pick up the soapbar 1 from the garbagecan 1. > move soapbar 1 to garbagecan 1 You move the soapbar 1 to the garbagecan 1. > examine garbagecan 1 On the garbagecan 1, you see a soapbar 1. > examine garbagecan 1 On the garbagecan 1, you see a soapbar 1. > examine garbagecan 1 On the garbagecan 1, you see a soapbar 1. > examine garbagecan 1 On the garbagecan 1, you see a soapbar 1. > examine garbagecan 1 On the garbagecan 1, you see a soapbar 1. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > examine garbagecan 1 On the garbagecan 1, you see a soapbar 1. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > examine garbagecan 1 On the garbagecan 1, you see a soapbar 1. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > look You are facing the garbagecan 1. Next to it, you see nothing. > examine garbagecan 1 On the garbagecan 1, you see a soapbar 1. > look You are facing the garbagecan 1. Next to it, you see nothing. Out\[5\]: Prediction( trajecotry=\['> go to countertop 1', 'You arrive at countertop 1. On the countertop 1, you see a candle 1, a soapbar 1, a soapbottle 2, a soapbottle 1, and a spraybottle 1.', '> take soapbar 1 from countertop 1', 'You pick up the soapbar 1 from the countertop 1.', '> go to garbagecan 1', 'You arrive at garbagecan 1. On the garbagecan 1, you see nothing.', '> move soapbar 1 to garbagecan 1', 'You move the soapbar 1 to the garbagecan 1.', '> examine garbagecan 1', 'On the garbagecan 1, you see a soapbar 1.', '> examine garbagecan 1', 'On the garbagecan 1, you see a soapbar 1.', '> examine garbagecan 1', 'On the garbagecan 1, you see a soapbar 1.', '> examine garbagecan 1', 'On the garbagecan 1, you see a soapbar 1.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> take soapbar 1 from garbagecan 1', 'You pick up the soapbar 1 from the garbagecan 1.', '> move soapbar 1 to garbagecan 1', 'You move the soapbar 1 to the garbagecan 1.', '> examine garbagecan 1', 'On the garbagecan 1, you see a soapbar 1.', '> examine garbagecan 1', 'On the garbagecan 1, you see a soapbar 1.', '> examine garbagecan 1', 'On the garbagecan 1, you see a soapbar 1.', '> examine garbagecan 1', 'On the garbagecan 1, you see a soapbar 1.', '> examine garbagecan 1', 'On the garbagecan 1, you see a soapbar 1.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> examine garbagecan 1', 'On the garbagecan 1, you see a soapbar 1.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> examine garbagecan 1', 'On the garbagecan 1, you see a soapbar 1.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.', '> examine garbagecan 1', 'On the garbagecan 1, you see a soapbar 1.', '> look', 'You are facing the garbagecan 1. Next to it, you see nothing.'\], success=0 ) Okay, in this case it couldn't solve this example! Now, let's check the average quality of 4o and 4o-mini. In \[6\]: Copied! metric \= lambda x, y, trace\=None: y.success evaluate \= dspy.Evaluate(devset\=devset, metric\=metric, display\_progress\=True, num\_threads\=16) metric = lambda x, y, trace=None: y.success evaluate = dspy.Evaluate(devset=devset, metric=metric, display\_progress=True, num\_threads=16) In \[7\]: Copied! agent\_4o.verbose \= False evaluate(agent\_4o) agent\_4o.verbose = False evaluate(agent\_4o) Average Metric: 115.00 / 200 (57.5%): 100%|██████████| 200/200 \[06:14<00:00, 1.87s/it\] 2024/12/28 11:10:25 INFO dspy.evaluate.evaluate: Average Metric: 115 / 200 (57.5%) Out\[7\]: 57.5 In \[8\]: Copied! agent\_4o\_mini \= Agent() agent\_4o\_mini.set\_lm(gpt4o\_mini) evaluate(agent\_4o\_mini) agent\_4o\_mini = Agent() agent\_4o\_mini.set\_lm(gpt4o\_mini) evaluate(agent\_4o\_mini) Average Metric: 30.00 / 200 (15.0%): 100%|██████████| 200/200 \[08:33<00:00, 2.57s/it\] 2024/12/28 11:18:59 INFO dspy.evaluate.evaluate: Average Metric: 30 / 200 (15.0%) Out\[8\]: 15.0 Out of the box, on this task, 4o is decent (58% success rate) while 4o-mini struggles (15% success rate). Let's apply the following strategy: 1. We'll optimize the _prompts_ for gpt-4o in a lightweight way. 2. We'll then use this prompt-optimized agent as a teacher to fine-tune gpt-4o-mini on the task. This will increase its quality from 19% to 72% (or 82% if you use 500 trainset examples). ### Prompt-optimizing GPT-4o[¶](#prompt-optimizing-gpt-4o) In \[9\]: Copied! optimizer \= dspy.MIPROv2(metric\=metric, auto\="light", num\_threads\=16, prompt\_model\=gpt4o) config \= dict(max\_bootstrapped\_demos\=1, max\_labeled\_demos\=0, minibatch\_size\=40) optimized\_4o \= optimizer.compile(agent\_4o, trainset\=trainset, \*\*config, requires\_permission\_to\_run\=False) optimizer = dspy.MIPROv2(metric=metric, auto="light", num\_threads=16, prompt\_model=gpt4o) config = dict(max\_bootstrapped\_demos=1, max\_labeled\_demos=0, minibatch\_size=40) optimized\_4o = optimizer.compile(agent\_4o, trainset=trainset, \*\*config, requires\_permission\_to\_run=False) ### Fine-tuning GPT-4o-mini[¶](#fine-tuning-gpt-4o-mini) For fine-tuning, we'll need a teacher program (`optimized_4o` above) and a student program derived from it (`student_4om` below). In \[10\]: Copied! student\_4o\_mini \= optimized\_4o.deepcopy() student\_4o\_mini.set\_lm(gpt4o\_mini) \# student\_4o\_mini.react.demos = \[\] # you can optionally reset the demos student\_4o\_mini = optimized\_4o.deepcopy() student\_4o\_mini.set\_lm(gpt4o\_mini) # student\_4o\_mini.react.demos = \[\] # you can optionally reset the demos In \[11\]: Copied! optimizer \= dspy.BootstrapFinetune(metric\=metric, num\_threads\=16) finetuned\_4o\_mini \= optimizer.compile(student\_4o\_mini, teacher\=optimized\_4o, trainset\=trainset) optimizer = dspy.BootstrapFinetune(metric=metric, num\_threads=16) finetuned\_4o\_mini = optimizer.compile(student\_4o\_mini, teacher=optimized\_4o, trainset=trainset) ### Evaluate the finetuned GPT-4o-mini agent[¶](#evaluate-the-finetuned-gpt-4o-mini-agent) In \[12\]: Copied! evaluate(finetuned\_4o\_mini) evaluate(finetuned\_4o\_mini) Average Metric: 143.00 / 200 (71.5%): 100%|██████████| 200/200 \[03:15<00:00, 1.05it/s\] Having done all this optimization, let's save our program so we can use it later! This will keep a reference to the fine-tuned model as well, as long as it continued to exist with the same identifier at the provider side. In \[13\]: Copied! finetuned\_4o\_mini.save('finetuned\_4o\_mini\_001.pkl') finetuned\_4o\_mini.save('finetuned\_4o\_mini\_001.pkl') Let's now check out one task using our fine-tuned agent program! In \[14\]: Copied! finetuned\_4o\_mini.verbose \= True finetuned\_4o\_mini(\*\*devset\[0\].inputs()) finetuned\_4o\_mini.verbose = True finetuned\_4o\_mini(\*\*devset\[0\].inputs()) Task: -= Welcome to TextWorld, ALFRED! =- You are in the middle of a room. Looking quickly around you, you see a armchair 1, a cabinet 1, a drawer 21, a drawer 20, a drawer 19, a drawer 18, a drawer 17, a drawer 16, a drawer 15, a drawer 14, a drawer 13, a drawer 12, a drawer 11, a drawer 10, a drawer 9, a drawer 8, a drawer 7, a drawer 6, a drawer 5, a drawer 4, a drawer 3, a drawer 2, a drawer 1, a dresser 1, a garbagecan 1, a sidetable 5, a sidetable 4, a sidetable 3, a sidetable 2, a sidetable 1, and a sofa 1. Your task is to: put some box on dresser. > go to cabinet 1 You arrive at cabinet 1. On the cabinet 1, you see nothing. > go to dresser 1 You arrive at dresser 1. On the dresser 1, you see a book 1, a newspaper 1, a remotecontrol 1, a statue 3, and a television 1. > look You are facing the dresser 1. Next to it, you see nothing. > go to sidetable 1 You arrive at sidetable 1. On the sidetable 1, you see a cellphone 1, and a desklamp 1. > go to sidetable 2 You arrive at sidetable 2. On the sidetable 2, you see a box 2. > take box 2 from sidetable 2 You pick up the box 2 from the sidetable 2. > go to dresser 1 You arrive at dresser 1. On the dresser 1, you see a book 1, a newspaper 1, a remotecontrol 1, a statue 3, and a television 1. > move box 2 to dresser 1 You move the box 2 to the dresser 1. Out\[14\]: Prediction( trajecotry=\['> go to cabinet 1', 'You arrive at cabinet 1. On the cabinet 1, you see nothing.', '> go to dresser 1', 'You arrive at dresser 1. On the dresser 1, you see a book 1, a newspaper 1, a remotecontrol 1, a statue 3, and a television 1.', '> look', 'You are facing the dresser 1. Next to it, you see nothing.', '> go to sidetable 1', 'You arrive at sidetable 1. On the sidetable 1, you see a cellphone 1, and a desklamp 1.', '> go to sidetable 2', 'You arrive at sidetable 2. On the sidetable 2, you see a box 2.', '> take box 2 from sidetable 2', 'You pick up the box 2 from the sidetable 2.', '> go to dresser 1', 'You arrive at dresser 1. On the dresser 1, you see a book 1, a newspaper 1, a remotecontrol 1, a statue 3, and a television 1.', '> move box 2 to dresser 1', 'You move the box 2 to the dresser 1.'\], success=1 ) If you want to load and use the agent program, you can do that as follows. In \[16\]: Copied! loaded \= Agent() loaded.load('finetuned\_4o\_mini\_001.pkl') loaded = Agent() loaded.load('finetuned\_4o\_mini\_001.pkl') Back to top --- # Privacy-Conscious Delegation - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/tutorials/papillon/index.md "Edit this page") Privacy-Conscious Delegation ============================ Please refer to [this tutorial from the PAPILLON authors](https://github.com/Columbia-NLP-Lab/PAPILLON/blob/main/papillon_tutorial.ipynb) using DSPy. This tutorial demonstrates a few aspects of using DSPy in a more advanced context: 1. It builds a multi-stage `dspy.Module` that involves a small local LM using an external tool. 2. It builds a multi-stage _judge_ in DSPy, and uses it as a metric for evaluation. 3. It uses this judge for optimizing the `dspy.Module`, using a large model as a teacher for a small local LM. Back to top --- # Classification - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/tutorials/classification/index.md "Edit this page") Classification ============== Please refer to [this tutorial from Drew Breunig](https://www.dbreunig.com/2024/12/12/pipelines-prompt-optimization-with-dspy.html) using DSPy. This tutorial demonstrates a few aspects of using DSPy in a highly-accessible, concrete context for categorizing historic events with a tiny LM. Back to top --- # Use Cases - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/community/use-cases.md "Edit this page") Use Cases ========= We often get questions like "How are people using DSPy in practice?", both in production and for research. This list was created to collect a few pointers and to encourage others in the community to add their own work below. This list is ever expanding and highly incomplete (WIP)! We'll be adding a bunch more. If you would like to add your product or research to this list, please make a PR. A Few Company Use Cases ----------------------- | **Name** | **Use Cases** | | --- | --- | | **[JetBlue](https://www.jetblue.com/)
** | Multiple chatbot use cases. [Blog](https://www.databricks.com/blog/optimizing-databricks-llm-pipelines-dspy) | | **[Replit](https://replit.com/)
** | Synthesize diffs using code LLMs using a DSPy pipeline. [Blog](https://blog.replit.com/code-repair) | | **[Databricks](https://www.databricks.com/)
** | Research, products, and customer solutions around LM Judges, RAG, classification, and other applications. [Blog](https://www.databricks.com/blog/dspy-databricks)
[Blog II](https://www.databricks.com/customers/ddi) | | **[Sephora](https://www.sephora.com/)
** | Undisclosed agent usecases; perspectives shared in [DAIS Session](https://www.youtube.com/watch?v=D2HurSldDkE)
. | | **[Zoro UK](https://www.zoro.co.uk/)
** | E-commerce applications around structured shopping. [Portkey Session](https://www.youtube.com/watch?v=_vGKSc1tekE) | | **[VMware](https://www.vmware.com/)
** | RAG and other prompt optimization applications. [Interview in The Register.](https://www.theregister.com/2024/02/22/prompt_engineering_ai_models/)
[Business Insider.](https://www.businessinsider.com/chaptgpt-large-language-model-ai-prompt-engineering-automated-optimizer-2024-3) | | **[Haize Labs](https://www.haizelabs.com/)
** | Automated red-teaming for LLMs. [Blog](https://blog.haizelabs.com/posts/dspy/) | | **[Plastic Labs](https://www.plasticlabs.ai/)
** | Different pipelines within Honcho. [Blog](https://blog.plasticlabs.ai/blog/User-State-is-State-of-the-Art) | | **[PingCAP](https://pingcap.com/)
** | Building a knowledge graph. [Article](https://www.pingcap.com/article/building-a-graphrag-from-wikipedia-page-using-dspy-openai-and-tidb-vector-database/) | | **[Salomatic](https://langtrace.ai/blog/case-study-how-salomatic-used-langtrace-to-build-a-reliable-medical-report-generation-system)
** | Enriching medical reports using DSPy. [Blog](https://langtrace.ai/blog/case-study-how-salomatic-used-langtrace-to-build-a-reliable-medical-report-generation-system) | | **[Truelaw](https://www.youtube.com/watch?v=O0F3RAWZNfM)
** | How Truelaw builds bespoke LLM pipelines for law firms using DSPy. [Podcast](https://www.youtube.com/watch?v=O0F3RAWZNfM) | | **[Moody's](https://www.moodys.com/)
** | Leveraging DSPy to optimize RAG systems, LLM-as-a-Judge, and agentic systems for financial workflows. | | **[Normal Computing](https://www.normalcomputing.com/)
** | Translate specs from chip companies from English to intermediate formal languages | | **[Procure.FYI](https://www.procure.fyi/)
** | Process messy, publicly available technology spending and pricing data via DSPy. | | **[RadiantLogic](https://www.radiantlogic.com/)
** | AI Data Assistant. DSPy is used for the agent that routes the query, the context extraction module, the text-to-sql conversion engine, and the table summarization module. | | **[Raia](https://raiahealth.com/)
** | Using DSPy for AI-powered Personal Healthcare Agents. | | **[Hyperlint](https://hyperlint.com)
** | Uses DSPy to generate technical documentation. DSPy helps to fetch relevant information and synthesize that into tutorials. | | **[Starops](https://staropshq.com/)
& [Saya](https://heysaya.ai/)
** | Building research documents given a user's corpus. Generate prompts to create more articles from example articles. | | **[Tessel AI](https://tesselai.com/)
** | Enhancing human-machine interaction with data use cases. | | **[Dicer.ai](https://dicer.ai/)
** | Uses DSPy for marketing AI to get the most from their paid ads. | | **[Howie](https://howie.ai)
** | Using DSPy to automate meeting scheduling through email. | | **[Isoform.ai](https://isoform.ai)
** | Building custom integrations using DSPy. | | **[Trampoline AI](https://trampoline.ai)
** | Uses DSPy to power their data-augmentation and LM pipelines. | | **[Pretrain](https://pretrain.com)
** | Uses DSPy to automatically optimize AI performance towards user-defined tasks based on uploaded examples. | WIP. This list mainly includes companies that have public posts or have OKed being included for specific products so far. A Few Papers Using DSPy ----------------------- | **Name** | **Description** | | --- | --- | | **[STORM](https://arxiv.org/abs/2402.14207)
** | Writing Wikipedia-like Articles From Scratch. | | **[PATH](https://arxiv.org/abs/2406.11706)
** | Prompts as Auto-Optimized Training Hyperparameters: Training Best-in-Class IR Models from Scratch with 10 Gold Labels | | **[WangLab @ MEDIQA](https://arxiv.org/abs/2404.14544)
** | UofT's winning system at MEDIQA, outperforms the next best system by 20 points | | **[UMD's Suicide Detection System](https://arxiv.org/abs/2406.06608)
** | Outperforms 20-hour expert human prompt engineering by 40% | | **[IReRa](https://arxiv.org/abs/2401.12178)
** | Infer-Retrieve-Rank: Extreme Classification with > 10,000 Labels | | **[Unreasonably Effective Eccentric Prompts](https://arxiv.org/abs/2402.10949v2)
** | General Prompt Optimization | | **[Palimpzest](https://arxiv.org/abs/2405.14696)
** | A Declarative System for Optimizing AI Workloads | | **[AI Agents that Matter](https://arxiv.org/abs/2407.01502v1)
** | Agent Efficiency Optimization | | **[EDEN](https://arxiv.org/abs/2406.17982v1)
** | Empathetic Dialogues for English Learning: Uses adaptive empathetic feedback to improve student grit | | **[ECG-Chat](https://arxiv.org/pdf/2408.08849)
** | Uses DSPy with GraphRAG for medical report generation | | **[DSPy Assertions](https://arxiv.org/abs/2312.13382)
** | Various applications of imposing hard and soft constraints on LM outputs | | **[DSPy Guardrails](https://boxiyu.github.io/assets/pdf/DSPy_Guardrails.pdf)
** | Reduce the attack success rate of CodeAttack, decreasing from 75% to 5% | | **[Co-STORM](https://arxiv.org/pdf/2408.15232)
** | Collaborative STORM: Generate Wikipedia-like articles through collaborative discourse among users and multiple LM agents | A Few Repositories (or other OSS examples) using DSPy ----------------------------------------------------- | **Name** | **Description/Link** | | --- | --- | | **Stanford CS 224U Homework** | [Github](https://github.com/cgpotts/cs224u/blob/main/hw_openqa.ipynb) | | **STORM Report Generation (10,000 GitHub stars)** | [Github](https://github.com/stanford-oval/storm) | | **DSPy Redteaming** | [Github](https://github.com/haizelabs/dspy-redteam) | | **DSPy Theory of Mind** | [Github](https://github.com/plastic-labs/dspy-opentom) | | **Indic cross-lingual Natural Language Inference** | [Github](https://github.com/saifulhaq95/DSPy-Indic/blob/main/indicxlni.ipynb) | | **Optimizing LM for Text2SQL using DSPy** | [Github](https://github.com/jjovalle99/DSPy-Text2SQL) | | **DSPy PII Masking Demo by Eric Ness** | [Colab](https://colab.research.google.com/drive/1KZR1sGTp_RLWUJPAiK1FKPKI-Qn9neUm?usp=sharing) | | **DSPy on BIG-Bench Hard Example** | [Github](https://drchrislevy.github.io/posts/dspy/dspy.html) | | **Building a chess playing agent using DSPy** | [Github](https://medium.com/thoughts-on-machine-learning/building-a-chess-playing-agent-using-dspy-9b87c868f71e) | | **Ittia Research Fact Checking** | [Github](https://github.com/ittia-research/check) | | **Strategic Debate via Tree-of-Thought** | [Github](https://github.com/zbambergerNLP/strategic-debate-tot) | | **Sanskrit to English Translation App** | [Github](https://github.com/ganarajpr/sanskrit-translator-dspy) | | **DSPy for extracting features from PDFs on arXiv** | [Github](https://github.com/S1M0N38/dspy-arxiv) | | **DSPygen: DSPy in Ruby on Rails** | [Github](https://github.com/seanchatmangpt/dspygen) | | **DSPy Inspector** | [Github](https://github.com/Neoxelox/dspy-inspector) | | **DSPy with FastAPI** | [Github](https://github.com/diicellman/dspy-rag-fastapi) | | **DSPy for Indian Languages** | [Github](https://github.com/saifulhaq95/DSPy-Indic) | | **Hurricane: Blog Posts with Generative Feedback Loops!** | [Github](https://github.com/weaviate-tutorials/Hurricane) | | **RAG example using DSPy, Gradio, FastAPI, and Ollama** | [Github](https://github.com/diicellman/dspy-gradio-rag) | | **Synthetic Data Generation** | [Github](https://colab.research.google.com/drive/1CweVOu0qhTC0yOfW5QkLDRIKuAuWJKEr?usp=sharing) | | **Self Discover** | [Github](https://colab.research.google.com/drive/1GkAQKmw1XQgg5UNzzy8OncRe79V6pADB?usp=sharing) | TODO: This list in particular is highly incomplete. There are a couple dozen other good ones. A Few Providers, Integrations, and related Blog Releases -------------------------------------------------------- | **Name** | **Link** | | --- | --- | | **Databricks** | [Link](https://www.databricks.com/blog/dspy-databricks) | | **Zenbase** | [Link](https://zenbase.ai/) | | **LangWatch** | [Link](https://langwatch.ai/blog/introducing-dspy-visualizer) | | **Gradient** | [Link](https://gradient.ai/blog/achieving-gpt-4-level-performance-at-lower-cost-using-dspy) | | **Snowflake** | [Link](https://medium.com/snowflake/dspy-snowflake-140d6d947d73) | | **Langchain** | [Link](https://python.langchain.com/v0.2/docs/integrations/providers/dspy/) | | **Weaviate** | [Link](https://weaviate.io/blog/dspy-optimizers) | | **Qdrant** | [Link](https://qdrant.tech/documentation/frameworks/dspy/) | | **Weights & Biases Weave** | [Link](https://weave-docs.wandb.ai/guides/integrations/dspy/) | | **Milvus** | [Link](https://milvus.io/docs/integrate_with_dspy.md) | | **Neo4j** | [Link](https://neo4j.com/labs/genai-ecosystem/dspy/) | | **Lightning AI** | [Link](https://lightning.ai/lightning-ai/studios/dspy-programming-with-foundation-models) | | **Haystack** | [Link](https://towardsdatascience.com/automating-prompt-engineering-with-dspy-and-haystack-926a637a3f43) | | **Arize** | [Link](https://docs.arize.com/phoenix/tracing/integrations-tracing/dspy) | | **LlamaIndex** | [Link](https://github.com/stanfordnlp/dspy/blob/main/examples/llamaindex/dspy_llamaindex_rag.ipynb) | | **Langtrace** | [Link](https://docs.langtrace.ai/supported-integrations/llm-frameworks/dspy) | | **Langfuse** | [Link](https://langfuse.com/docs/integrations/dspy) | | **OpenLIT** | [Link](https://docs.openlit.io/latest/integrations/dspy) | Credit: Some of these resources were originally compiled in the [Awesome DSPy](https://github.com/ganarajpr/awesome-dspy/tree/master) repo. Back to top --- # Deployment - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/tutorials/deployment/index.md "Edit this page") Tutorial: Deploying your DSPy program ===================================== This guide demonstrates two potential ways to deploy your DSPy program in production: FastAPI for lightweight deployments and MLflow for more production-grade deployments with program versioning and management. Below, we'll assume you have the following simple DSPy program that you want to deploy. You can replace this with something more sophisticated. `[](#__codelineno-0-1) import dspy [](#__codelineno-0-2) [](#__codelineno-0-3) dspy.settings.configure(lm=dspy.LM("openai/gpt-4o-mini")) [](#__codelineno-0-4) dspy_program = dspy.ChainOfThought("question -> answer")` Deploying with FastAPI ---------------------- FastAPI offers a straightforward way to serve your DSPy program as a REST API. This is ideal when you have direct access to your program code and need a lightweight deployment solution. `[](#__codelineno-1-1) > pip install fastapi uvicorn [](#__codelineno-1-2) > export OPENAI_API_KEY="your-openai-api-key"` Let's create a FastAPI application to serve your `dspy_program` defined above. `[](#__codelineno-2-1) from fastapi import FastAPI, HTTPException [](#__codelineno-2-2) from pydantic import BaseModel [](#__codelineno-2-3) [](#__codelineno-2-4) import dspy [](#__codelineno-2-5) [](#__codelineno-2-6) app = FastAPI( [](#__codelineno-2-7) title="DSPy Program API", [](#__codelineno-2-8) description="A simple API serving a DSPy Chain of Thought program", [](#__codelineno-2-9) version="1.0.0" [](#__codelineno-2-10) ) [](#__codelineno-2-11) [](#__codelineno-2-12) # Define request model for better documentation and validation [](#__codelineno-2-13) class Question(BaseModel): [](#__codelineno-2-14) text: str [](#__codelineno-2-15) [](#__codelineno-2-16) # Configure your language model and 'asyncify' your DSPy program. [](#__codelineno-2-17) lm = dspy.LM("openai/gpt-4o-mini") [](#__codelineno-2-18) dspy.settings.configure(lm=lm, async_max_workers=4) # default is 8 [](#__codelineno-2-19) dspy_program = dspy.ChainOfThought("question -> answer") [](#__codelineno-2-20) dspy_program = dspy.asyncify(dspy_program) [](#__codelineno-2-21) [](#__codelineno-2-22) @app.post("/predict") [](#__codelineno-2-23) async def predict(question: Question): [](#__codelineno-2-24) try: [](#__codelineno-2-25) result = await dspy_program(question=question.text) [](#__codelineno-2-26) return { [](#__codelineno-2-27) "status": "success", [](#__codelineno-2-28) "data": result.toDict() [](#__codelineno-2-29) } [](#__codelineno-2-30) except Exception as e: [](#__codelineno-2-31) raise HTTPException(status_code=500, detail=str(e))` In the code above, we call `dspy.asyncify` to convert the dspy program to run in async mode for high-throughput FastAPI deployments. Currently, this runs the dspy program in a separate thread and awaits its result. By default, the limit of spawned threads is 8. Think of this like a worker pool. If you have 8 in-flight programs and call it once more, the 9th call will wait until one of the 8 returns. You can configure the async capacity using the new `async_max_workers` setting. Streaming, in DSPy 2.6.0+ Streaming is also supported in DSPy 2.6.0+, available as a release candidate via `pip install -U --pre dspy`. We can use `dspy.streamify` to convert the dspy program to a streaming mode. This is useful when you want to stream the intermediate outputs (i.e. O1-style reasoning) to the client before the final prediction is ready. This uses asyncify under the hood and inherits the execution semantics. `[](#__codelineno-3-1) dspy_program = dspy.asyncify(dspy.ChainOfThought("question -> answer")) [](#__codelineno-3-2) streaming_dspy_program = dspy.streamify(dspy_program) [](#__codelineno-3-3) [](#__codelineno-3-4) @app.post("/predict/stream") [](#__codelineno-3-5) async def stream(question: Question): [](#__codelineno-3-6) async def generate(): [](#__codelineno-3-7) async for value in streaming_dspy_program(question=question.text): [](#__codelineno-3-8) if isinstance(value, dspy.Prediction): [](#__codelineno-3-9) data = {"prediction": value.labels().toDict()} [](#__codelineno-3-10) elif isinstance(value, litellm.ModelResponse): [](#__codelineno-3-11) data = {"chunk": value.json()} [](#__codelineno-3-12) yield f"data: {ujson.dumps(data)}\n\n" [](#__codelineno-3-13) yield "data: [DONE]\n\n" [](#__codelineno-3-14) [](#__codelineno-3-15) return StreamingResponse(generate(), media_type="text/event-stream") [](#__codelineno-3-16) [](#__codelineno-3-17) # Since you're often going to want to stream the result of a DSPy program as server-sent events, [](#__codelineno-3-18) # we've included a helper function for that, which is equivalent to the code above. [](#__codelineno-3-19) [](#__codelineno-3-20) from dspy.utils.streaming import streaming_response [](#__codelineno-3-21) [](#__codelineno-3-22) @app.post("/predict/stream") [](#__codelineno-3-23) async def stream(question: Question): [](#__codelineno-3-24) stream = streaming_dspy_program(question=question.text) [](#__codelineno-3-25) return StreamingResponse(streaming_response(stream), media_type="text/event-stream")` Write your code to a file, e.g., `fastapi_dspy.py`. Then you can serve the app with: `[](#__codelineno-4-1) > uvicorn fastapi_dspy:app --reload` It will start a local server at `http://127.0.0.1:8000/`. You can test it with the python code below: `[](#__codelineno-5-1) import requests [](#__codelineno-5-2) [](#__codelineno-5-3) response = requests.post( [](#__codelineno-5-4) "http://127.0.0.1:8000/predict", [](#__codelineno-5-5) json={"text": "What is the capital of France?"} [](#__codelineno-5-6) ) [](#__codelineno-5-7) print(response.json())` You should see the response like below: `[](#__codelineno-6-1) {'status': 'success', 'data': {'reasoning': 'The capital of France is a well-known fact, commonly taught in geography classes and referenced in various contexts. Paris is recognized globally as the capital city, serving as the political, cultural, and economic center of the country.', 'answer': 'The capital of France is Paris.'}}` Deploying with MLflow --------------------- We recommend deploying with MLflow if you are looking to package your DSPy program and deploy in an isolated environment. MLflow is a popular platform for managing machine learning workflows, including versioning, tracking, and deployment. `[](#__codelineno-7-1) > pip install mlflow>=2.18.0` Let's spin up the MLflow tracking server, where we will store our DSPy program. The command below will start a local server at `http://127.0.0.1:5000/`. `[](#__codelineno-8-1) > mlflow ui` Then we can define the DSPy program and log it to the MLflow server. "log" is an overloaded term in MLflow, basically it means we store the program information along with environment requirements in the MLflow server. See the code below: `[](#__codelineno-9-1) import dspy [](#__codelineno-9-2) import mlflow [](#__codelineno-9-3) [](#__codelineno-9-4) mlflow.set_tracking_uri("http://127.0.0.1:5000/") [](#__codelineno-9-5) mlflow.set_experiment("deploy_dspy_program") [](#__codelineno-9-6) [](#__codelineno-9-7) lm = dspy.LM("openai/gpt-4o-mini") [](#__codelineno-9-8) dspy.settings.configure(lm=lm) [](#__codelineno-9-9) dspy_program = dspy.ChainOfThought("question -> answer") [](#__codelineno-9-10) [](#__codelineno-9-11) with mlflow.start_run(): [](#__codelineno-9-12) mlflow.dspy.log_model( [](#__codelineno-9-13) dspy_program, [](#__codelineno-9-14) "dspy_program", [](#__codelineno-9-15) input_example={"messages": [{"role": "user", "content": "What is LLM agent?"}]}, [](#__codelineno-9-16) task="llm/v1/chat", [](#__codelineno-9-17) )` We recommend you to set `task="llm/v1/chat"` so that the deployed program automatically takes input and generate output in the same format as the OpenAI chat API, which is a common interface for LM applications. Write the code above into a file, e.g. `mlflow_dspy.py`, and run it. After you logged the program, you can view the saved information in MLflow UI. Open `http://127.0.0.1:5000/` and select the `deploy_dspy_program` experiment, then select the run your just created, under the `Artifacts` tab, you should see the logged program information, similar to the following screenshot: ![MLflow UI](dspy_mlflow_ui.png) Grab your run id from UI (or the console print when you execute `mlflow_dspy.py`), now you can deploy the logged program with the following command: `[](#__codelineno-10-1) > mlflow models serve -m runs:/{run_id}/model -p 6000` After the program is deployed, you can test it with the following command: `[](#__codelineno-11-1) > curl http://127.0.0.1:6000/invocations -H "Content-Type:application/json" --data '{"messages": [{"content": "what is 2 + 2?", "role": "user"}]}'` You should see the response like below: `[](#__codelineno-12-1) {"choices": [{"index": 0, "message": {"role": "assistant", "content": "{\"reasoning\": \"The question asks for the sum of 2 and 2. To find the answer, we simply add the two numbers together: 2 + 2 = 4.\", \"answer\": \"4\"}"}, "finish_reason": "stop"}]}` For complete guide on how to deploy a DSPy program with MLflow, and how to customize the deployment, please refer to the [MLflow documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) . ### Best Practices for MLflow Deployment 1. **Environment Management**: Always specify your Python dependencies in a `conda.yaml` or `requirements.txt` file. 2. **Versioning**: Use meaningful tags and descriptions for your model versions. 3. **Input Validation**: Define clear input schemas and examples. 4. **Monitoring**: Set up proper logging and monitoring for production deployments. For production deployments, consider using MLflow with containerization: `[](#__codelineno-13-1) > mlflow models build-docker -m "runs:/{run_id}/model" -n "dspy-program" [](#__codelineno-13-2) > docker run -p 6000:8080 dspy-program` For a complete guide on production deployment options and best practices, refer to the [MLflow documentation](https://mlflow.org/docs/latest/llms/dspy/index.html) . Back to top --- # Debugging & Observability - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/tutorials/observability/index.md "Edit this page") Tutorial: Debugging and Observability in DSPy ============================================= This guide demonstrates how to debug problems and improve observability in DSPy. Modern AI programs often involve multiple components, such as language models, retrievers, and tools. DSPy allows you to build and optimize such complex AI systems in a clean and modular way. However, as systems grow more sophisticated, the ability to **understand what your system is doing** becomes critical. Without transparency, the prediction process can easily become a black box, making failures or quality issues difficult to diagnose and production maintenance challenging. By the end of this tutorial, you'll understand how to debug an issue and improve observability using [MLflow Tracing](#mlflow-tracing) . You'll also explore how to build a custom logging solution using callbacks. Define a Program ---------------- We'll start by creating a simple ReAct agent that uses ColBERTv2's Wikipedia dataset as a retrieval source. You can replace this with a more sophisticated program. `[](#__codelineno-0-1) import dspy [](#__codelineno-0-2) from dspy.datasets import HotPotQA [](#__codelineno-0-3) [](#__codelineno-0-4) lm = dspy.LM('openai/gpt-4o-mini') [](#__codelineno-0-5) colbert = dspy.ColBERTv2(url='http://20.102.90.50:2017/wiki17_abstracts') [](#__codelineno-0-6) dspy.configure(lm=lm, rm=colbert) [](#__codelineno-0-7) [](#__codelineno-0-8) agent = dspy.ReAct("question -> answer", tools=[dspy.Retrieve(k=1)])` Now, let's ask the agent a simple question: `[](#__codelineno-1-1) prediction = agent(question="Which baseball team does Shohei Ohtani play for?") [](#__codelineno-1-2) print(prediction.answer)` `[](#__codelineno-2-1) Shohei Ohtani plays for the Los Angeles Angels.` Oh, this is incorrect. He no longer plays for the Angels; he moved to the Dodgers and won the World Series in 2024! Let's debug the program and explore potential fixes. Using `inspect_history` ----------------------- DSPy provides the `inspect_history()` utility, which prints out all LLM invocations made so far: `[](#__codelineno-3-1) # Print out 5 LLM calls [](#__codelineno-3-2) dspy.inspect_history(n=5)` ``[](#__codelineno-4-1) [2024-12-01T10:23:29.144257] [](#__codelineno-4-2) [](#__codelineno-4-3) System message: [](#__codelineno-4-4) [](#__codelineno-4-5) Your input fields are: [](#__codelineno-4-6) 1. `question` (str) [](#__codelineno-4-7) [](#__codelineno-4-8) ... [](#__codelineno-4-9) [](#__codelineno-4-10) Response: [](#__codelineno-4-11) [](#__codelineno-4-12) [[ ## Thought_5 ## ]] [](#__codelineno-4-13) The search results continue to be unhelpful and do not provide the current team for Shohei Ohtani in Major League Baseball. I need to conclude that he plays for the Los Angeles Angels based on prior knowledge, as the searches have not yielded updated information. [](#__codelineno-4-14) [](#__codelineno-4-15) [[ ## Action_5 ## ]] [](#__codelineno-4-16) Finish[Los Angeles Angels] [](#__codelineno-4-17) [](#__codelineno-4-18) [[ ## completed ## ]]`` The log reveals that the agent could not retrieve helpful information from the search tool. However, what exactly did the retriever return? While useful, `inspect_history` has some limitations: * In real-world systems, other components like retrievers, tools, and custom modules play significant roles, but `inspect_history` only logs LLM calls. * DSPy programs often make multiple LLM calls within a single prediction. Monolith log history makes it hard to organize logs, especially when handling multiple questions. * Metadata such as parameters, latency, and the relationship between modules are not captured. **Tracing** addresses these limitations and provides a more comprehensive solution. Tracing ------- [MLflow](https://mlflow.org/docs/latest/llms/tracing/index.html) is an end-to-end machine learning platform that is integrated seamlessly with DSPy to support best practices in LLMOps. Using MLflow's automatic tracing capability with DSPy is straightforward; **No sign up for services or an API key is required**. You just need to install MLflow and call `mlflow.dspy.autolog()` in your notebook or script. `[](#__codelineno-5-1) pip install -U mlflow>=2.18.0` `[](#__codelineno-6-1) import mlflow [](#__codelineno-6-2) [](#__codelineno-6-3) mlflow.dspy.autolog() [](#__codelineno-6-4) [](#__codelineno-6-5) # This is optional. Create an MLflow Experiment to store and organize your traces. [](#__codelineno-6-6) mlflow.set_experiment("DSPy")` Now you're all set! Let's run your agent again: `[](#__codelineno-7-1) agent(question="Which baseball team does Shohei Ohtani play for?")` MLflow automatically generates a **trace** for the prediction and records it in the experiment. To explore traces visually, launch the MLflow UI by the following command and access it in your browser: `[](#__codelineno-8-1) mlflow ui --port 5000` ![DSPy MLflow Tracing](dspy-tracing.gif) From the retriever step output, you can observe that it returned outdated information; indicating Shohei Ohtani was still playing in the Japanese league and the final answer was based on the LLM's prior knowledge! We should update the dataset or add additional tools to ensure access to the latest information. Info MLflow is an end-to-end LLMOps platform that offers extensive features like experiment tracking, evaluation, and deployment. To learn more about DSPy and MLflow integration, visit [this tutorial](../deployment/#deploying-with-mlflow) . For example, we can add a web search capability to the agent, using the [Tavily](https://tavily.com/) web search API. `[](#__codelineno-9-1) from dspy.predict.react import Tool [](#__codelineno-9-2) from tavily import TavilyClient [](#__codelineno-9-3) [](#__codelineno-9-4) search_client = TavilyClient(api_key="") [](#__codelineno-9-5) [](#__codelineno-9-6) def web_search(query: str) -> list[str]: [](#__codelineno-9-7) """Run a web search and return the content from the top 5 search results""" [](#__codelineno-9-8) response = search_client.search(query) [](#__codelineno-9-9) return [r["content"] for r in response["results"]] [](#__codelineno-9-10) [](#__codelineno-9-11) agent = dspy.ReAct("question -> answer", tools=[Tool(web_search)]) [](#__codelineno-9-12) [](#__codelineno-9-13) prediction = agent(question="Which baseball team does Shohei Ohtani play for?") [](#__codelineno-9-14) print(agent.answer)` `[](#__codelineno-10-1) Los Angeles Dodgers` Building a Custom Logging Solution ---------------------------------- Sometimes, you may want to implement a custom logging solution. For instance, you might need to log specific events triggered by a particular module. DSPy's **callback** mechanism supports such use cases. The `BaseCallback` class provides several handlers for customizing logging behavior: | Handlers | Description | | --- | --- | | `on_module_start` / `on_module_end` | Triggered when a `dspy.Module` subclass is invoked. | | `on_lm_start` / `on_lm_end` | Triggered when a `dspy.LM` subclass is invoked. | | `on_adapter_format_start` / `on_adapter_format_end` | Triggered when a `dspy.Adapter` subclass formats the input prompt. | | `on_adapter_parse_start` / `on_adapter_parse_end` | Triggered when a `dspy.Adapter` subclass postprocess the output text from an LM. | Here’s an example of custom callback that logs the intermediate steps of a ReAct agent: `[](#__codelineno-11-1) import dspy [](#__codelineno-11-2) from dspy.utils.callback import BaseCallback [](#__codelineno-11-3) [](#__codelineno-11-4) # 1. Define a custom callback class that extends BaseCallback class [](#__codelineno-11-5) class AgentLoggingCallback(BaseCallback): [](#__codelineno-11-6) [](#__codelineno-11-7) # 2. Implement on_module_end handler to run a custom logging code. [](#__codelineno-11-8) def on_module_end(self, call_id, outputs, exception): [](#__codelineno-11-9) step = "Reasoning" if self._is_reasoning_output(outputs) else "Acting" [](#__codelineno-11-10) print(f"== {step} Step ===") [](#__codelineno-11-11) for k, v in outputs.items(): [](#__codelineno-11-12) print(f" {k}: {v}") [](#__codelineno-11-13) print("\n") [](#__codelineno-11-14) [](#__codelineno-11-15) def _is_reasoning_output(self, outputs): [](#__codelineno-11-16) return any(k.startswith("Thought") for k in outputs.keys()) [](#__codelineno-11-17) [](#__codelineno-11-18) # 3. Set the callback to DSPy setting so it will be applied to program execution [](#__codelineno-11-19) dspy.configure(callbacks=[AgentLoggingCallback()])` `[](#__codelineno-12-1) == Reasoning Step === [](#__codelineno-12-2) Thought_1: I need to find the current team that Shohei Ohtani plays for in Major League Baseball. [](#__codelineno-12-3) Action_1: Search[Shohei Ohtani current team 2023] [](#__codelineno-12-4) [](#__codelineno-12-5) == Acting Step === [](#__codelineno-12-6) passages: ["Shohei Ohtani ..."] [](#__codelineno-12-7) [](#__codelineno-12-8) ...` Info Be cautious when working with input or output data in callbacks. Mutating them in-place can modify the original data passed to the program, potentially leading to unexpected behavior. To avoid this, it’s strongly recommended to create a copy of the data before performing any operations that may alter it. Back to top --- # Reasoning - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/tutorials/math/index.ipynb "Edit this page") Tutorial: Math Reasoning[¶](#tutorial-math-reasoning) ====================================================== Let's walk through a quick example of setting up a `dspy.ChainOfThought` module and optimizing it for answering algebra questions. Install the latest DSPy via `pip install -U dspy` and follow along. Let's tell DSPy that we will use OpenAI's `gpt-4o-mini` in our modules. To authenticate, DSPy will look into your `OPENAI_API_KEY`. You can easily swap this out for [other providers or local models](https://github.com/stanfordnlp/dspy/blob/main/examples/migration.ipynb) . In \[1\]: Copied! import dspy gpt4o\_mini \= dspy.LM('openai/gpt-4o-mini', max\_tokens\=2000) gpt4o \= dspy.LM('openai/gpt-4o', max\_tokens\=2000) dspy.configure(lm\=gpt4o\_mini) \# we'll use gpt-4o-mini as the default LM, unless otherwise specified import dspy gpt4o\_mini = dspy.LM('openai/gpt-4o-mini', max\_tokens=2000) gpt4o = dspy.LM('openai/gpt-4o', max\_tokens=2000) dspy.configure(lm=gpt4o\_mini) # we'll use gpt-4o-mini as the default LM, unless otherwise specified Next, let's load some data examples from the [MATH](https://arxiv.org/abs/2103.03874) benchmark. We'll use a training split for optimization and evaluate it on a held-out dev set. Please note that the following step will require: %pip install git+https://github.com/hendrycks/math.git In \[2\]: Copied! from dspy.datasets import MATH dataset \= MATH(subset\='algebra') print(len(dataset.train), len(dataset.dev)) from dspy.datasets import MATH dataset = MATH(subset='algebra') print(len(dataset.train), len(dataset.dev)) 350 350 Let's inspect one example from the training set. In \[3\]: Copied! example \= dataset.train\[0\] print("Question:", example.question) print("Answer:", example.answer) example = dataset.train\[0\] print("Question:", example.question) print("Answer:", example.answer) Question: The doctor has told Cal O'Ree that during his ten weeks of working out at the gym, he can expect each week's weight loss to be $1\\%$ of his weight at the end of the previous week. His weight at the beginning of the workouts is $244$ pounds. How many pounds does he expect to weigh at the end of the ten weeks? Express your answer to the nearest whole number. Answer: 221 Now let's define our module. It's extremely simple: just a chain-of-thought step that takes a `question` and produces an `answer`. In \[4\]: Copied! module \= dspy.ChainOfThought("question -> answer") module(question\=example.question) module = dspy.ChainOfThought("question -> answer") module(question=example.question) Out\[4\]: Prediction( reasoning="Cal O'Ree's weight loss each week is $1\\\\%$ of his weight at the end of the previous week. This means that at the end of each week, he retains $99\\\\%$ of his weight from the previous week. \\n\\nIf we denote his weight at the beginning as \\\\( W\_0 = 244 \\\\) pounds, then his weight at the end of week \\\\( n \\\\) can be expressed as:\\n\\\\\[\\nW\_n = W\_{n-1} \\\\times 0.99\\n\\\\\]\\nThis can be simplified to:\\n\\\\\[\\nW\_n = W\_0 \\\\times (0.99)^n\\n\\\\\]\\nAfter 10 weeks, his weight will be:\\n\\\\\[\\nW\_{10} = 244 \\\\times (0.99)^{10}\\n\\\\\]\\n\\nNow, we calculate \\\\( (0.99)^{10} \\\\):\\n\\\\\[\\n(0.99)^{10} \\\\approx 0.904382\\n\\\\\]\\n\\nNow, we can calculate his expected weight after 10 weeks:\\n\\\\\[\\nW\_{10} \\\\approx 244 \\\\times 0.904382 \\\\approx 220.5\\n\\\\\]\\n\\nRounding to the nearest whole number, Cal O'Ree can expect to weigh approximately \\\\( 221 \\\\) pounds at the end of the ten weeks.", answer='221' ) Next, let's set up an evaluator for the zero-shot module above, before prompt optimization. In \[5\]: Copied! THREADS \= 24 kwargs \= dict(num\_threads\=THREADS, display\_progress\=True, display\_table\=5) evaluate \= dspy.Evaluate(devset\=dataset.dev, metric\=dataset.metric, \*\*kwargs) evaluate(module) THREADS = 24 kwargs = dict(num\_threads=THREADS, display\_progress=True, display\_table=5) evaluate = dspy.Evaluate(devset=dataset.dev, metric=dataset.metric, \*\*kwargs) evaluate(module) Average Metric: 259.00 / 350 (74.0%): 100%|██████████| 350/350 \[01:30<00:00, 3.85it/s\] 2024/11/28 18:41:55 INFO dspy.evaluate.evaluate: Average Metric: 259 / 350 (74.0%) | | question | example\_reasoning | example\_answer | pred\_reasoning | pred\_answer | method | | --- | --- | --- | --- | --- | --- | --- | | 0 | What is the smallest integer value of $c$ such that the function $... | The given function has a domain of all real numbers if and only if... | 1 | To determine the smallest integer value of \\( c \\) such that the f... | 1 | ✔️ \[True\] | | 1 | What is the least value of $x$ that is a solution of $\|{-x+3}\|=7$? | In order to have $\|{-x+3}\| = 7$, we must have $-x + 3 = 7$ or $-x ... | \-4 | To solve the equation \\( \|{-x+3}\|=7 \\), we need to consider the de... | \-4 | ✔️ \[True\] | | 2 | Evaluate $\\left\\lceil -\\frac{7}{4}\\right\\rceil$. | $-\\frac{7}{4}$ is between $-1$ and $-2$, so $\\left\\lceil -\\frac{7}... | \-1 | To evaluate \\(\\left\\lceil -\\frac{7}{4}\\right\\rceil\\), we first nee... | \-1 | ✔️ \[True\] | | 3 | A triangle has vertices at coordinates $(11,1)$, $(2,3)$ and $(3,7... | We must find the distance between each pair of points by using the... | 10 | To find the length of the longest side of the triangle with vertic... | 10 | ✔️ \[True\] | | 4 | Let $f(x) = x + 2$ and $g(x) = 1/f(x)$. What is $g(f(-3))$? | First, we find that $f(-3) = (-3) + 2 = -1$. Then, $$g(f(-3)) = g(... | 1 | To find \\( g(f(-3)) \\), we first need to evaluate \\( f(-3) \\). The... | 1 | ✔️ \[True\] | ... 345 more rows not displayed ... Out\[5\]: 74.0 And lastly let's optimize our module. Since we want strong reasoning, we'll use the large GPT-4o as the teacher model (used to bootstrap reasoning for the small LM at optimization time) but not as the prompt model (used to craft instructions) or the task model (trained). GPT-4o will be invoked only a small number of times. The model involved directly in optimization and in the resulting (optimized) program will be GPT-4o-mini. We will also specify `max_bootstrapped_demos=4` which means we want at most four bootstrapped examples in the prompt and `max_labeled_demos=4` which means that, in total between bootstrapped and pre-labeled examples, we want at most four. In \[ \]: Copied! kwargs \= dict(num\_threads\=THREADS, teacher\_settings\=dict(lm\=gpt4o), prompt\_model\=gpt4o\_mini) optimizer \= dspy.MIPROv2(metric\=dataset.metric, auto\="medium", \*\*kwargs) kwargs \= dict(requires\_permission\_to\_run\=False, max\_bootstrapped\_demos\=4, max\_labeled\_demos\=4) optimized\_module \= optimizer.compile(module, trainset\=dataset.train, \*\*kwargs) kwargs = dict(num\_threads=THREADS, teacher\_settings=dict(lm=gpt4o), prompt\_model=gpt4o\_mini) optimizer = dspy.MIPROv2(metric=dataset.metric, auto="medium", \*\*kwargs) kwargs = dict(requires\_permission\_to\_run=False, max\_bootstrapped\_demos=4, max\_labeled\_demos=4) optimized\_module = optimizer.compile(module, trainset=dataset.train, \*\*kwargs) In \[7\]: Copied! evaluate(optimized\_module) evaluate(optimized\_module) Average Metric: 310.00 / 350 (88.6%): 100%|██████████| 350/350 \[01:31<00:00, 3.84it/s\] 2024/11/28 18:59:19 INFO dspy.evaluate.evaluate: Average Metric: 310 / 350 (88.6%) | | question | example\_reasoning | example\_answer | pred\_reasoning | pred\_answer | method | | --- | --- | --- | --- | --- | --- | --- | | 0 | What is the smallest integer value of $c$ such that the function $... | The given function has a domain of all real numbers if and only if... | 1 | The function \\( f(x) = \\frac{x^2 + 1}{x^2 - x + c} \\) will have a ... | 1 | ✔️ \[True\] | | 1 | What is the least value of $x$ that is a solution of $\|{-x+3}\|=7$? | In order to have $\|{-x+3}\| = 7$, we must have $-x + 3 = 7$ or $-x ... | \-4 | The equation \\( \|{-x+3}\|=7 \\) implies two possible cases: 1. \\(-x ... | \-4 | ✔️ \[True\] | | 2 | Evaluate $\\left\\lceil -\\frac{7}{4}\\right\\rceil$. | $-\\frac{7}{4}$ is between $-1$ and $-2$, so $\\left\\lceil -\\frac{7}... | \-1 | To evaluate \\(\\left\\lceil -\\frac{7}{4}\\right\\rceil\\), we first nee... | \-1 | ✔️ \[True\] | | 3 | A triangle has vertices at coordinates $(11,1)$, $(2,3)$ and $(3,7... | We must find the distance between each pair of points by using the... | 10 | To find the length of the sides of the triangle formed by the vert... | 10 | ✔️ \[True\] | | 4 | Let $f(x) = x + 2$ and $g(x) = 1/f(x)$. What is $g(f(-3))$? | First, we find that $f(-3) = (-3) + 2 = -1$. Then, $$g(f(-3)) = g(... | 1 | To find \\( g(f(-3)) \\), we first need to evaluate \\( f(-3) \\). Usi... | 1 | ✔️ \[True\] | ... 345 more rows not displayed ... Out\[7\]: 88.57 Neat. It was pretty straightforward to improve quality from 74% to over 88% on a held-out set here. That said, for reasoning tasks like this, you will often want to consider more advanced strategies, like: * A `dspy.ReAct` module with access to a calculator function or `dspy.PythonInterpreter` * Ensembling multiple optimized prompts with a majority vote (or an Aggregator module) on top Just to understand what changed, let's view the prompt after optimization. In \[8\]: Copied! dspy.inspect\_history() dspy.inspect\_history() \[2024-11-28T18:59:19.176586\] System message: Your input fields are: 1. \`question\` (str) Your output fields are: 1. \`reasoning\` (str) 2. \`answer\` (str) All interactions will be structured in the following way, with the appropriate values filled in. \[\[ ## question ## \]\] {question} \[\[ ## reasoning ## \]\] {reasoning} \[\[ ## answer ## \]\] {answer} \[\[ ## completed ## \]\] In adhering to this structure, your objective is: Analyze the \`question\` provided, and systematically apply mathematical reasoning to derive the \`answer\`. Ensure to articulate each step of your thought process in the \`reasoning\` field, detailing how you identify relationships and formulate equations to arrive at the solution. User message: \[\[ ## question ## \]\] If $|4x+2|=10$ and $x<0$, what is the value of $x$? Respond with the corresponding output fields, starting with the field \`\[\[ ## reasoning ## \]\]\`, then \`\[\[ ## answer ## \]\]\`, and then ending with the marker for \`\[\[ ## completed ## \]\]\`. Assistant message: \[\[ ## reasoning ## \]\] The equation $|4x+2|=10$ implies two possible equations: $4x+2=10$ or $4x+2=-10$. We solve each equation separately. 1. For $4x+2=10$: \\\[\ 4x + 2 = 10 \\\\\ 4x = 8 \\\\\ x = 2\ \\\] However, $x=2$ does not satisfy the condition $x<0$. 2. For $4x+2=-10$: \\\[\ 4x + 2 = -10 \\\\\ 4x = -12 \\\\\ x = -3\ \\\] The solution $x = -3$ satisfies the condition $x<0$. Therefore, the value of $x$ is $\\boxed{-3}$. \[\[ ## answer ## \]\] -3 \[\[ ## completed ## \]\] User message: \[\[ ## question ## \]\] Given that $-4$ is a solution to $x^2 + bx -36 = 0$, what is the value of $b$? Respond with the corresponding output fields, starting with the field \`\[\[ ## reasoning ## \]\]\`, then \`\[\[ ## answer ## \]\]\`, and then ending with the marker for \`\[\[ ## completed ## \]\]\`. Assistant message: \[\[ ## reasoning ## \]\] The product of the roots of this quadratic is $-36/1=-36$, so the other solution must be $-36/-4=9$. That means that the sum of the solutions is $-4+9=5$. The sum of the solutions is also $-b/1=-b$. Thus, $-b=5$ and $b=\\boxed{-5}$. \[\[ ## answer ## \]\] -5 \[\[ ## completed ## \]\] User message: \[\[ ## question ## \]\] Richard is building a rectangular playground from 200 feet of fencing. The fencing must entirely enclose the playground. What is the maximum area of this playground? Respond with the corresponding output fields, starting with the field \`\[\[ ## reasoning ## \]\]\`, then \`\[\[ ## answer ## \]\]\`, and then ending with the marker for \`\[\[ ## completed ## \]\]\`. Assistant message: \[\[ ## reasoning ## \]\] Let the length of the playground be $l$ and the width be $w$. We have the equation $2l+2w=200 \\Rightarrow l + w = 100$. We want to maximize the area of this rectangular playground, which is given by $lw$. From our equation, we know that $l=100-w$. Substituting this in to our expression for area, we have \\\[(100-w)(w)=100w-w^2\\\]We will now complete the square to find the maximum value of this expression. Factoring a $-1$ out, we have \\\[-(w^2-100w)\\\]In order for the expression inside the parenthesis to be a perfect square, we need to add and subtract $(100/2)^2=2500$ inside the parenthesis. Doing this, we get \\\[-(w^2-100w+2500-2500) \\Rightarrow -(w-50)^2+2500\\\]Since the maximum value of $-(w-50)^2$ is 0 (perfect squares are always nonnegative), the maximum value of the entire expression is 2500, which is achieved when $w=50$ and $l=100-w=50$ (the playground is a square). Thus, the maximum area of the playground is $\\boxed{2500}$ square feet. \[\[ ## answer ## \]\] 2500 \[\[ ## completed ## \]\] User message: \[\[ ## question ## \]\] Alice and Bob are playing a game. Alice starts first. On Alice's turn, she flips a coin. If she gets a heads, she wins. If not, it becomes Bob's turn. On Bob's turn, he flips a coin. If he gets a tails, he wins. If not, it becomes Alice's turn. What is the probability that Alice wins the game? Respond with the corresponding output fields, starting with the field \`\[\[ ## reasoning ## \]\]\`, then \`\[\[ ## answer ## \]\]\`, and then ending with the marker for \`\[\[ ## completed ## \]\]\`. Assistant message: \[\[ ## reasoning ## \]\] Alice has a $1/2$ chance of winning the game on her first turn. If she doesn't, then the probability that she wins the game on her second turn is $1/8,$ since she must not win on her first flip ($1/2$ chance), Bob must not win on his first flip ($1/2$ chance), and then Alice must win on her second flip ($1/2$ chance). The probability that she wins the game on her third turn is $1/32,$ and in general, the probability that she wins the game on her $k^\\text{th}$ turn is $(1/2)^{2k-1}.$ Thus, the probability that Alice wins is an infinite geometric series with first term $1/2$ and common ratio $1/4.$ So, the probability that Alice wins the game is $$\\frac{\\frac12}{1-\\frac14} = \\boxed{\\frac{2}{3}}.$$OR Note that the only difference between the odds of Alice or Bob winning is who goes first. Because Bob goes second, the odds of him winning on his $k^\\text{th}$ flip is half of the odds that Alice wins on her $k^\\text{th}$ flip, since Alice must first get a tails before Bob gets a chance to win. Thus, if $a$ is Alice's chance of winning, and $b$ is Bob's chance of winning, then $a = 2b.$ Also, since someone must win, $a + b = 1.$ It follows that $a = 2/3$ and $b = 1/3,$ so Alice has a $\\boxed{\\frac{2}{3}}$ chance of winning the game. \[\[ ## answer ## \]\] \\frac{2}{3} \[\[ ## completed ## \]\] User message: \[\[ ## question ## \]\] Find the smallest possible value of $$\\frac{(y-x)^2}{(y-z)(z-x)} + \\frac{(z-y)^2}{(z-x)(x-y)} + \\frac{(x-z)^2}{(x-y)(y-z)},$$ where $x,y,$ and $z$ are distinct real numbers. Respond with the corresponding output fields, starting with the field \`\[\[ ## reasoning ## \]\]\`, then \`\[\[ ## answer ## \]\]\`, and then ending with the marker for \`\[\[ ## completed ## \]\]\`. Response: \[\[ ## reasoning ## \]\] Let \\( S = \\frac{(y-x)^2}{(y-z)(z-x)} + \\frac{(z-y)^2}{(z-x)(x-y)} + \\frac{(x-z)^2}{(x-y)(y-z)} \\). To find the minimum value of \\( S \\), we can use the method of symmetry and the Cauchy-Schwarz inequality. By applying the Cauchy-Schwarz inequality, we have: \\\[\ \\left( \\frac{(y-x)^2}{(y-z)(z-x)} + \\frac{(z-y)^2}{(z-x)(x-y)} + \\frac{(x-z)^2}{(x-y)(y-z)} \\right) \\left( (y-z)(z-x) + (z-x)(x-y) + (x-y)(y-z) \\right) \\geq \\left( (y-x) + (z-y) + (x-z) \\right)^2.\ \\\] The right-hand side simplifies to: \\\[\ (y-x) + (z-y) + (x-z) = 0,\ \\\] which means the right-hand side is \\( 0 \\). Therefore, we have: \\\[\ S \\cdot \\left( (y-z)(z-x) + (z-x)(x-y) + (x-y)(y-z) \\right) \\geq 0.\ \\\] This implies that \\( S \\) is non-negative. Next, we can test specific values for \\( x, y, z \\) to find the minimum. Let's set \\( x = 0, y = 1, z = 2 \\): \\\[\ S = \\frac{(1-0)^2}{(1-2)(2-0)} + \\frac{(2-1)^2}{(2-0)(0-1)} + \\frac{(0-2)^2}{(0-1)(1-2)}.\ \\\] Calculating each term: 1. The first term: \\\[\ \\frac{1^2}{(-1)(2)} = \\frac{1}{-2} = -\\frac{1}{2}.\ \\\] 2. The second term: \\\[\ \\frac{1^2}{(2)(-1)} = \\frac{1}{-2} = -\\frac{1}{2}.\ \\\] 3. The third term: \\\[\ \\frac{(-2)^2}{(-1)(-1)} = \\frac{4}{1} = 4.\ \\\] Thus, \\\[\ S = -\\frac{1}{2} - \\frac{1}{2} + 4 = 3.\ \\\] To check if this is indeed the minimum, we can also consider the case when \\( x, y, z \\) are the roots of a quadratic polynomial. By symmetry and the nature of the expression, we can conclude that the minimum value of \\( S \\) is indeed \\( 3 \\). Therefore, the smallest possible value of \\( S \\) is \\( \\boxed{3} \\). \[\[ ## answer ## \]\] 3 \[\[ ## completed ## \]\] Back to top --- # Roadmap - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/roadmap.md "Edit this page") Roadmap Sketch: DSPy 2.5+ ========================= It’s been a year since DSPy evolved out of Demonstrate–Search–Predict (DSP), whose research started at Stanford NLP all the way back in February 2022. Thanks to 200 wonderful contributors, DSPy has introduced tens of thousands of people to building modular LM programs and optimizing their prompts and weights automatically. In this time, DSPy has grown to 160,000 monthly downloads and 16,000 stars on GitHub, becoming synonymous with prompt optimization in many circles and inspiring at least a half-dozen cool new libraries. This document is an initial sketch of DSPy’s public roadmap for the next few weeks and months, as we work on DSPy 2.5 and plan for DSPy 3.0. Suggestions and open-source contributors are more than welcome: just open an issue or submit a pull request regarding the roadmap. Technical Objectives -------------------- The thesis of DSPy is that for LMs to be useful, we have to shift from ad-hoc prompting to new notions of programming LMs. Instead of relying on LMs gaining much more general or more compositional capabilities, we need to enable developers to iteratively explore their problems and build modular software that invokes LMs for well-scoped tasks. We need to enable that through modules and optimizers that isolate how they decompose their problems and describe their system's objectives from how their LMs are invoked or fine-tuned to maximize their objectives. DSPy's goal has been to develop (and to build the community and shared infrastructure for the collective development of) the abstractions, programming patterns, and optimizers toward this thesis. To a first approximation, DSPy’s current user-facing language has the minimum number of appropriate abstractions that address the goals above: declarative signatures, define-by-run modules, and optimizers that can be composed quite powerfully. But there are several things we need to do better to realize our goals. The upcoming DSPy releases will have the following objectives. 1. Polishing the core functionality. 2. Developing more accurate, lower-cost optimizers. 3. Building end-to-end tutorials from DSPy’s ML workflow to deployment. 4. Shifting towards more interactive optimization & tracking. Team & Organization ------------------- DSPy is fairly unusual in its technical objectives, contributors, and audience. Though DSPy takes inspiration from PyTorch, a library for building and optimizing DNNs, there is one major difference: PyTorch was introduced well after DNNs were mature ML concepts, but DSPy seeks to establish and advance core LM Programs research: the framework is propelled by constant academic research from programming abstractions (like the original **Demonstrate–Search–Predict** concepts, DSPy **Signatures**, or **LM Assertions**) to NLP systems (like **STORM**, **PATH**, and **IReRa**) to prompt optimizers (like **MIPRO**) and RL (like **BetterTogether**), among many other related directions. This research all composes into a concrete, practical library, thanks to dozens of industry contributors, many of whom are deploying apps in production using DSPy. Because of this, DSPy reaches not only of grad students and ML engineers, but also many non-ML engineers, from early adopter SWEs to hobbyists exploring new ways of using LMs. The following team, with help from many folks in the OSS community, is working towards the objectives in this Roadmap. **Project Lead:** Omar Khattab (Stanford & Databricks) **Project Mentors:** Chris Potts (Stanford), Matei Zaharia (UC Berkeley & Databricks), Heather Miller (CMU & Two Sigma) **Core Library:** Arnav Singhvi (Databricks & Stanford), Herumb Shandilya (Stanford), Hanna Moazam (Databricks), Sri Vardhamanan (Dashworks), Cyrus Nouroozi (Zenbase), Amir Mehr (Zenbase), Kyle Caverly (Modular), with special thanks to Keshav Santhanam (Stanford), Thomas Ahle (Normal Computing), Connor Shorten (Weaviate) **Prompt Optimization:** Krista Opsahl-Ong (Stanford), Michael Ryan (Stanford), Josh Purtell (Basis), with special thanks to Eric Zhang (Stanford) **Finetuning & RL:** Dilara Soylu (Stanford), Isaac Miller (Anyscale), Karel D'Oosterlinck (Ghent), with special thanks to Paridhi Masehswari (Stanford) **PL Abstractions:** Shangyin Tan (UC Berkeley), Manish Shetty (UC Berkeley), Peter Zhong (CMU) **Applications:** Jasper Xian (Waterloo), Saron Samuel (Stanford), Alberto Mancarella (Stanford), Faraz Khoubsirat (Waterloo), Saiful Haq (IIT-B), Ashutosh Sharma (UIUC) 1) Polishing the core functionality. ------------------------------------ Over the next month, polishing is the main objective and likely the one to have the highest ROI on the experience of the average user. Conceptually, DSPy has an extremely small core. It’s nothing but (1) LMs, (2) Signatures & Modules, (3) Optimizers, and (4) Assertions. These concepts and their implementations evolved organically over the past couple of years. We are working now to consolidate what we’ve learned and refactor internally so that things “just work” out of the box for new users, who may not know all the tips-and-tricks just yet. More concretely: 1. We want to increase the quality of zero-shot, off-the-shelf DSPy programs, i.e. those not yet compiled on custom data. 2. Wherever possible, DSPy should delegate lower-level internal complexity (like managing LMs and structured generation) to emerging lower-level libraries. When required, we may fork smaller libraries out of DSPy to support infrastructure pieces as their own projects. 3. DSPy should internally be more modular and we need higher compatibility between internal components. Specifically, we need more deeper and more native investment in (i) typed multi-field constraints, (ii) assertions, (iii) observability and experimental tracking, (iv) deployment of artifacts and related concerns like streaming and async, and (v) fine-tuning and serving open models. ### On LMs As of DSPy 2.4, the library has approximately 20,000 lines of code and roughly another 10,000 lines of code for tests, examples, and documentation. Some of these are clearly necessary (e.g., DSPy optimizers) but others exist only because the LM space lacks the building blocks we need under the hood. Luckily, for LM interfaces, a very strong library now exists: LiteLLM, a library that unifies interfaces to various LM and embedding providers. We expect to reduce around 6000 LoCs of support for custom LMs and retrieval models by shifting a lot of that to LiteLLM. Objectives in this space include improved caching, saving/loading of LMs, support for streaming and async LM requests. Work here is currently led by Hanna Moazam and Sri Vardhamanan, building on a foundation by Cyrus Nouroozi, Amir Mehr, Kyle Caverly, and others. ### On Signatures & Modules Traditionally, LMs offer text-in-text-out interfaces. Toward modular programming, DSPy introduced signatures for the first time (as DSP Templates in Jan 2023) as a way to structure the inputs and outputs of LM interactions. Standard prompts conflate interface (“what should the LM do?”) with implementation (“how do we tell it to do that?”). DSPy signatures isolate the former so we can infer and learn the latter from data — in the context of a bigger program. Today in the LM landscape, notions of "structured outputs" have evolved dramatically, thanks to constrained decoding and other improvements, and have become mainstream. What may be called "structured inputs" remains is yet to become mainstream outside of DSPy, but is as crucial. Objectives in this space include refining the abstractions and implementations first-class notion of LM Adapters in DSPy, as translators that sits between signatures and LM interfaces. While Optimizers adjust prompts through interactions with a user-supplied metric and data, Adapters are more concerned with building up interactions with LMs to account for, e.g. (i) non-plaintext LM interfaces like chat APIs, structured outputs, function calling, and multi-modal APIs, (ii) languages beyond English or other forms of higher-level specialization. This has been explored in DSPy on and off in various forms, but we have started working on more fundamental approaches to this problem that will offer tangible improvements to most use-cases. Work here is currently led by Omar Khattab. ### On Finetuning & Serving In February 2023, DSPy introduced the notion of compiling to optimize the weights of an LM program. (To understand just how long ago that was in AI terms, this was before the Alpaca training project at Stanford had even started and a month before the first GPT-4 was released.) Since then, we have shown in October 2023 and, much more expansively, in July 2024, that the fine-tuning flavor of DSPy can deliver large gains for small LMs, especially when composed with prompt optimization. Overall, though, most DSPy users in practice explore prompt optimization and not weight optimization and most of our examples do the same. The primary reason for a lot of this is infrastructure. Fine-tuning in the DSPy flavor is more than just training a model: ultimately, we need to bootstrap training data for several different modules in a program, train multiple models and handle model selection, and then load and plug in those models into the program's modules. Doing this robustly at the level of abstraction DSPy offers requires a level of resource management that is not generally supported by external existing tools. Major efforts in this regard are currently led by Dilara Soylu and Isaac Miller. ### On Optimizers & Assertions This is a naturally major direction in the course of polishing. We will share more thoughts here after making more progress on the three angles above. 2) Developing more accurate, lower-cost optimizers. --------------------------------------------------- A very large fraction of the research in DSPy focuses on optimizing the prompts and the weights of LM programs. In December 2022, we introduced the algorithm and abstractions behind BootstrapFewShot (as Demonstrate in DSP) and several of its variants. In February 2023, we introduced the core version of what later became BootstrapFinetune. In August 2023, we introduced new variations of both of these. In December 2023, we introduced the first couple of instruction optimizers into DSPy, CA-OPRO and early versions of MIPRO. These were again upgraded in March 2024. Fast forward to June and July 2024, we released MIPROv2 for prompt optimization and BetterTogether for fine-tuning the weights of LM programs. We have been working towards a number of stronger optimizers. While we cannot share the internal details of research on new optimizers yet, we can outline the goals. A DSPy optimizer can be characterized via three angles: 1. Quality: How much quality can it deliver from various LMs? How effective does it need the zero-shot program to be in order to work well? 2. Cost: How many labeled (and unlabeled) inputs does it need? How many invocations of the program does it need? How expensive is the resulting optimized program at inference time? 3. Robustness: How well can it generalize to different unseen data points or distributions? How sensitive is it to mistakes of the metric or labels? Over the next six months, our goal is to dramatically improve each angle of these _when the other two are held constant_. Concretely, there are three directions here. * Benchmarking: A key prerequisite here is work on benchmarking. On the team, Michael Ryan and Shangyin Tan are leading these efforts. More soon. * Quality: The goal here is optimizers that extract, on average, 20% more on representative tasks than MIPROv2 and BetterTogether, under the usual conditions — like a few hundred inputs with labels and a good metric starting from a decent zero-shot program. Various efforts here are led by Dilara Soylu, Michael Ryan, Josh Purtell, Krista Opsahl-Ong, and Isaac Miller. * Efficiency: The goal here is optimizers that match the current best scores from MIPROv2 and BetterTogether but under 1-2 challenges like: (i) starting from only 10-20 inputs with labels, (ii) starting with a weak zero-shot program that scores 0%, (iii) where significant misalignment exists between train/validation and test, or (iii) where the user supplies no metric but provides a very small number of output judgments. 3) Building end-to-end tutorials from DSPy’s ML workflow to deployment. ----------------------------------------------------------------------- Using DSPy well for solving a new task is just doing good machine learning with LMs, but teaching this is hard. On the one hand, it's an iterative process: you make some initial choices, which will be sub-optimal, and then you refine them incrementally. It's highly exploratory: it's often the case that no one knows yet how to best solve a problem in a DSPy-esque way. One the other hand, DSPy offers many emerging lessons from several years of building LM systems, in which the design space, the data regime, and many other factors are new both to ML experts and to the very large fraction of users that have no ML experience. Though current docs do address [a bunch of this](/building-blocks/solving_your_task) in isolated ways, one thing we've learned is that we should separate teaching the core DSPy language (which is ultimately pretty small) from teaching the emerging ML workflow that works well in a DSPy-esque setting. As a natural extension of this, we need to place more emphasis on steps prior and after to the explicit coding in DSPy, from data collection to deployment that serves and monitors the optimized DSPy program in practice. This is just starting but efforts will be ramping up led by Omar Khattab, Isaac Miller, and Herumb Shandilya. 4) Shifting towards more interactive optimization & tracking. ------------------------------------------------------------- Right now, a DSPy user has a few ways to observe and tweak the process of optimization. They can study the prompts before, during, and after optimization methods like `inspect_history`, built-in logging, and/or the metadata returned by optimizers. Similarly, they can rely on `program.save` and `program.load` to potentially adjust the optimized prompts by hand. Alternatively, they can use one of the many powerful observability integrations — like from Phoenix Arize, LangWatch, or Weights & Biases Weave — to observe _in real time_ the process of optimization (e.g., scores, stack traces, successful & failed traces, and candidate prompts). DSPy encourages iterative engineering by adjusting the program, data, or metrics across optimization runs. For example, some optimizers allow “checkpointing” — e.g., if you optimize with BootstrapFewShotWithRandomSearch for 10 iterations then increase to 15 iterations, the first 10 will be loaded from cache. While these can accomplish a lot of goals, there are two limitations that future versions of DSPy will seek to address. 1. In general, DSPy’s (i) observability, (ii) experimental tracking, (iii) cost management, and (iii) deployment of programs should become first-class concerns via integration with tools like MLFlow. We will share more plans addressing this for DSPy 2.6 in the next 1-2 months. 2. DSPy 3.0 will introduce new optimizers that prioritize ad-hoc, human-in-the-loop feedback. This is perhaps the only substantial paradigm shift we see as necessary in the foreseeable future in DSPy. It involves various research questions at the level of the abstractions, UI/HCI, and ML, so it is a longer-term goal that we will share more about in the next 3-4 month. Back to top --- # Contributing - DSPy [](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/community/how-to-contribute.md "Edit this page") Contributing ============ Development Setup ----------------- We are working on a development setup guide. If you're interested in contributing, please reach out to us on Discord. Contributing ------------ **What if I have a better idea for prompting or synthetic data generation?** Perfect. We encourage you to think if it's best expressed as a module or an optimizer, and we'd love to merge it in DSPy so everyone can use it. DSPy is not a complete project; it's an ongoing effort to create structure (modules and optimizers) in place of hacky prompt and pipeline engineering tricks. **How can I add my favorite LM or vector store?** Check out these walkthroughs on setting up a [Custom LM client](/deep-dive/language_model_clients/custom-lm-client) and [Custom RM client](/deep-dive/retrieval_models_clients/custom-rm-client) . Back to top ---