# Table of Contents - [Tutorials | Bespoke Labs](#tutorials-bespoke-labs) - [Getting Started | Bespoke Labs](#getting-started-bespoke-labs) - [How-to Guides | Bespoke Labs](#how-to-guides-bespoke-labs) - [Examples | Bespoke Labs](#examples-bespoke-labs) - [Using LiteLLM for diverse providers | Bespoke Labs](#using-litellm-for-diverse-providers-bespoke-labs) - [Welcome | Bespoke Labs](#welcome-bespoke-labs) - [Integrations | Bespoke Labs](#integrations-bespoke-labs) - [Using OpenAI for batch inference | Bespoke Labs](#using-openai-for-batch-inference-bespoke-labs) - [Conceptual Guides | Bespoke Labs](#conceptual-guides-bespoke-labs) - [Using Anthropic for batch inference | Bespoke Labs](#using-anthropic-for-batch-inference-bespoke-labs) - [Quick Tour | Bespoke Labs](#quick-tour-bespoke-labs) - [Save $$$ with batch mode | Bespoke Labs](#save-with-batch-mode-bespoke-labs) - [Using vLLM with Curator | Bespoke Labs](#using-vllm-with-curator-bespoke-labs) - [Using Gemini for batch inference | Bespoke Labs](#using-gemini-for-batch-inference-bespoke-labs) - [API Service | Bespoke Labs](#api-service-bespoke-labs) - [Automatic recovery and caching | Bespoke Labs](#automatic-recovery-and-caching-bespoke-labs) - [Visualize your dataset with the Bespoke Curator Viewer | Bespoke Labs](#visualize-your-dataset-with-the-bespoke-curator-viewer-bespoke-labs) - [LLM API documentation | Bespoke Labs](#llm-api-documentation-bespoke-labs) - [Using kluster.ai for batch inference | Bespoke Labs](#using-kluster-ai-for-batch-inference-bespoke-labs) - [Self-Hosting | Bespoke Labs](#self-hosting-bespoke-labs) - [API reference | Bespoke Labs](#api-reference-bespoke-labs) - [Execute LLM-generated code | Bespoke Labs](#execute-llm-generated-code-bespoke-labs) - [Using Ollama with Curator | Bespoke Labs](#using-ollama-with-curator-bespoke-labs) - [Finetuning a model to identify features of a product | Bespoke Labs](#finetuning-a-model-to-identify-features-of-a-product-bespoke-labs) --- # Tutorials | Bespoke Labs [PreviousQuick Tour](/bespoke-curator/getting-started/quick-tour) [NextVisualize your dataset with the Bespoke Curator Viewer](/bespoke-curator/tutorials/visualize-your-dataset-with-the-bespoke-curator-viewer) Last updated 1 month ago [Visualize your dataset with the Bespoke Curator Viewer](/bespoke-curator/tutorials/visualize-your-dataset-with-the-bespoke-curator-viewer) [Automatic recovery and caching](/bespoke-curator/tutorials/automatic-recovery-and-caching) [Save $$$ with batch mode](/bespoke-curator/tutorials/save-usdusdusd-with-batch-mode) --- # Getting Started | Bespoke Labs [PreviousWelcome](/) [NextQuick Tour](/bespoke-curator/getting-started/quick-tour) Last updated 1 month ago Bespoke Curator makes it easy to create synthetic data pipelines. Whether you are training a model or extracting structure, Curator will prepare high-quality data quickly and robustly. * Rich Python based library for generating and curating synthetic data. * Interactive viewer to monitor data while it is being generated * First class support for structured outputs * Built-in performance optimizations for asynchronous operations, caching, and fault recovery at every scale * Support for a wide range of inference options via LiteLLM, vLLM, and popular batch APIs In addition, we are actively working on improving the library. Expect more changes to come in the future: 1. Verifiers: filter outputs to improve your data quality with models like [Bespoke-MiniCheck](/bespoke-minicheck/api) , or with code executors. 2. MCTS: explore reasoning trajectories using Monte Carlo Tree Search. 3. Data versioning: version your data along with the code that generates it. 4. Diversity and data quality indicators: understand the quality of your data. 5. Curator viewer: visualize and explore your generated data. Next, let's take a [Quick Tour](/bespoke-curator/getting-started/quick-tour) of the Curator library! ![](https://docs.bespokelabs.ai/~gitbook/image?url=https%3A%2F%2F1831689742-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FINVkBnpIXpC4135NE6Ex%252Fuploads%252FvkoTavHBKVG0VukoKAmY%252Fcurator-cli.gif%3Falt%3Dmedia%26token%3Dbf72486b-be67-4f01-a703-897186e9a480&width=768&dpr=4&quality=100&sign=b7900034&sv=2) --- # How-to Guides | Bespoke Labs [Using Ollama with Curator](/bespoke-curator/how-to-guides/using-ollama-with-curator) [Using LiteLLM for diverse providers](/bespoke-curator/how-to-guides/using-litellm-for-diverse-providers) [Using vLLM with Curator](/bespoke-curator/how-to-guides/using-vllm-with-curator) [Using OpenAI for batch inference](/bespoke-curator/how-to-guides/using-openai-for-batch-inference) [Using Anthropic for batch inference](/bespoke-curator/how-to-guides/using-anthropic-for-batch-inference) [Using Gemini for batch inference](/bespoke-curator/how-to-guides/using-gemini-for-batch-inference) [Using kluster.ai for batch inference](/bespoke-curator/how-to-guides/using-kluster.ai-for-batch-inference) [Execute LLM-generated code](/bespoke-curator/how-to-guides/execute-llm-generated-code) [PreviousSave $$$ with batch mode](/bespoke-curator/tutorials/save-usdusdusd-with-batch-mode) [NextUsing Ollama with Curator](/bespoke-curator/how-to-guides/using-ollama-with-curator) --- # Examples | Bespoke Labs You can find some end-to-end finetuning examples in this section (we're working on adding more examples!): * [Finetuning a model to identify features of a product](/bespoke-curator/examples/finetuning-a-model-to-identify-features-of-a-product) Our GitHub repository also has some shorter [examples](https://github.com/bespokelabsai/curator/tree/main/examples) . [PreviousExecute LLM-generated code](/bespoke-curator/how-to-guides/execute-llm-generated-code) [NextFinetuning a model to identify features of a product](/bespoke-curator/examples/finetuning-a-model-to-identify-features-of-a-product) Last updated 1 month ago --- # Using LiteLLM for diverse providers | Bespoke Labs This guide demonstrates how to use LiteLLM as a backend for curator to generate synthetic data using various LLM providers. We'll walk through an example of generating synthetic recipes, but this approach can be adapted for any synthetic data generation task. [](#prerequisites) Prerequisites ------------------------------------- * Python 3.10+ * Curator (`pip install bespokelabs-curator`) * Access to an LLM provider (e.g., Gemini API key) [](#steps) Steps --------------------- ### [](#id-1.-create-a-curator.llm-subclass) 1\. Create a curator.LLM Subclass First, create a class that inherits from `curator.LLM`. You'll need to implement two key methods: * `prompt()`: Generates the prompt for the LLM * `parse()`: Processes the LLM's response into your desired format Copy """Generate synthetic recipes for different cuisines using curator.""" from datasets import Dataset from bespokelabs import curator class RecipeGenerator(curator.LLM): """A recipe generator that generates recipes for different cuisines.""" def prompt(self, input: dict) -> str: """Generate a prompt using the template and cuisine.""" return f"Generate a random {input['cuisine']} recipe. Be creative but keep it realistic." def parse(self, input: dict, response: str) -> dict: """Parse the model response along with the input to the model into the desired output format..""" return { "recipe": response, "cuisine": input["cuisine"], } ### [](#id-2.-set-up-your-seed-dataset) 2\. Set Up Your Seed Dataset Create a dataset of inputs using the HuggingFace `Dataset` class: Copy # List of cuisines to generate recipes for cuisines = [\ {"cuisine": cuisine}\ for cuisine in [\ "Chinese",\ "Italian",\ "Mexican",\ "French",\ "Japanese",\ "Indian",\ "Thai",\ "Korean",\ "Vietnamese",\ "Brazilian",\ ]\ ] cuisines = Dataset.from_list(cuisines) ### [](#id-3.-configure-litellm-backend) 3\. Configure LiteLLM Backend Initialise your generator with LiteLLM configuration: Copy recipe_generator = RecipeGenerator( model_name="gemini/gemini-1.5-flash", # LiteLLM model identifier backend="litellm", # Specify LiteLLM backend backend_params={ "max_requests_per_minute": 2_000, # Rate limit for requests "max_tokens_per_minute": 4_000_000 # Token usage limit }, ) ### [](#id-4.-generate-data) 4\. Generate Data Generate your synthetic data: Copy recipes = recipe_generator(cuisines) print(recipes.to_pandas()) [](#litellm-configuration) LiteLLM Configuration ----------------------------------------------------- ### [](#api-keys-and-environment-variables) API Keys and Environment Variables For Gemini: Copy export GEMINI_API_KEY='your-api-key-here' # Get from https://aistudio.google.com/app/apikey [](#curator-configuration) Curator Configuration ----------------------------------------------------- ### [](#rate-limits) Rate Limits Configure rate limit with backend parameters: Copy # Custom RPM/TPM configuration # By default, this is set to: # - max_requests_per_minute: 10 # - max_tokens_per_minute: 100_000 backend_params={ "max_requests_per_minute": 2_000, # 2K requests/minute "max_tokens_per_minute": 4_000_000 # 4M tokens/minute } [PreviousUsing Ollama with Curator](/bespoke-curator/how-to-guides/using-ollama-with-curator) [NextUsing vLLM with Curator](/bespoke-curator/how-to-guides/using-vllm-with-curator) Last updated 1 month ago --- # Welcome | Bespoke Labs [NextGetting Started](/bespoke-curator/getting-started) Last updated 1 month ago ### [](#new-to-bespoke-curator-or-bespoke-minicheck-start-here) New to Bespoke Curator or Bespoke-MiniCheck? Start here! ### [](#bespoke-curator) Bespoke Curator Bespoke Curator makes it very easy to create high-quality synthetic data at scale, which you can use to finetune models or use for structured data extraction at scale. Bespoke Curator is an open-source project: 1. That comes with a rich Python based library for generating and curating synthetic data. 2. A Curator Viewer which makes it easy to view the datasets, thus aiding in the dataset creation. 3. We will also be releasing high-quality datasets that should move the needle on post-training. Start [here](/bespoke-curator/getting-started) . ### [](#using-bespoke-minicheck) Using Bespoke-MiniCheck Bespoke-MiniCheck is powered by Bespoke-MiniCheck-7B model, a best-in-class lightweight model that can be used to detect hallucinations. It tops the [LLM-AggreFact leaderboard](https://llm-aggrefact.github.io/) You can use Bespoke-MiniCheck either via: 1. [API service](/bespoke-minicheck/api) (easiest), or 2. [Host it yourself](/bespoke-minicheck/hosting) . From LLM-AggreFact Leaderboard ![](https://docs.bespokelabs.ai/~gitbook/image?url=https%3A%2F%2F1831689742-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FINVkBnpIXpC4135NE6Ex%252Fuploads%252FWNhlrC7oQq3qq9HDvp1l%252Fimage.png%3Falt%3Dmedia%26token%3D1f3fe04e-9afb-4f27-a5bc-2d1f71101fbd&width=768&dpr=4&quality=100&sign=9faa6b6a&sv=2) --- # Integrations | Bespoke Labs ### [](#guardrails) Guardrails Bespoke MiniCheck is available as a Guardrails validator here: [https://hub.guardrailsai.com/validator/bespokelabs/bespoke\_minicheck](https://hub.guardrailsai.com/validator/bespokelabs/bespoke_minicheck) Example usage: Copy # Import Guard and Validator from guardrails.hub import BespokeMiniCheck from guardrails import Guard # Setup Guard guard = Guard().use( BespokeMiniCheck, split_sentences=True, threshold=0.5, on_fail="fix" ) # Validator passes guard.validate("Alex likes cats.", metadata={"context": "Alex likes cats and dogs"}) # Validator fails guard.validate("Alex likes cats.", metadata={"context": "Alex likes dogs, but not cats."}) ### [](#ollama) Ollama Bespoke-MiniCheck-7B is available from Ollama [here](https://ollama.com/library/bespoke-minicheck) . More information can be found from their [blog post](https://ollama.com/blog/reduce-hallucinations-with-bespoke-minicheck) . Once you have Ollama, it is pretty straightforward to use the model. Note that Ollama doesn't yet support getting logits from the model, therefore we just output "yes" or "no". As part of Ollama, there are two examples available: 1. [Fact checking](https://github.com/ollama/ollama/tree/main/examples/python-grounded-factuality-simple-check) 2. [RAG use case](https://github.com/ollama/ollama/tree/main/examples/python-grounded-factuality-rag-check) [PreviousSelf-Hosting](/bespoke-minicheck/hosting) Last updated 4 months ago --- # Using OpenAI for batch inference | Bespoke Labs You can use **OpenAI** for batch inference in **Curator** to generate synthetic data. In this example, we will generate reannotation of wildchat dataset, but the approach can be adapted for any data generation task. [](#prerequisites) **Prerequisites** ----------------------------------------- * **Python 3.10+** * **Curator**: Install via `pip install bespokelabs-curator` * **OpenAI:** OpenAI API key [](#steps) **Steps** ------------------------- #### [](#id-1.-setup-environment-vars) **1\. Setup environment vars** Copy export OPENAI_API_KEY= **2\. Create a curator.LLM subclass** Create a class that inherits from `curator.LLM`. Implement two key methods: * `prompt()`: Generates the prompt for the LLM. * `parse()`: Processes the LLM's response into your desired format. Here’s the implementation: Copy """Example of reannotating the WildChat dataset using curator.""" import logging from bespokelabs import curator # To see more detail about how batches are being processed logger = logging.getLogger("bespokelabs.curator") logger.setLevel(logging.INFO) class WildChatReannotator(curator.LLM): """A reannotator for the WildChat dataset.""" def prompt(self, input: dict) -> str: """Extract the first message from a conversation to use as the prompt.""" return input["conversation"][0]["content"] def parse(self, input: dict, response: str) -> dict: """Parse the model response along with the input to the model into the desired output format..""" instruction = input["conversation"][0]["content"] return {"instruction": instruction, "new_response": response} #### [](#id-3.-configure-the-openai-model) **3\. Configure the OpenAI model** Copy distiller = WildChatReannotator(model_name="gpt-4o-mini", batch=True ) #### [](#id-4.-generate-data) **4\. Generate Data** Generate the structured data and output the results as a pandas DataFrame: Copy from datasets import load_dataset dataset = load_dataset("allenai/WildChat", split="train") dataset = dataset.select(range(100)) distilled_dataset = distiller(dataset) print(distilled_dataset) print(distilled_dataset[0]) ### [](#example-output) **Example Output** Using the above example, the output might look like this: instruction new\_response Write a very long, elaborate, descriptive and ... Scene: Omelette Apocalypse\\n\\n\*\*INT. DINER... what are you? I am a large language model, trained by OpenAI [](#batch-configuration) **Batch Configuration** ----------------------------------------------------- * Check out complete [batch configuration](https://docs.bespokelabs.ai/bespoke-curator/api-reference/llm-api-documentation#batch-processing-parameters) [PreviousUsing vLLM with Curator](/bespoke-curator/how-to-guides/using-vllm-with-curator) [NextUsing Anthropic for batch inference](/bespoke-curator/how-to-guides/using-anthropic-for-batch-inference) Last updated 1 month ago --- # Conceptual Guides | Bespoke Labs [](#key-components-of-curator.llm) Key Components of curator.LLM --------------------------------------------------------------------- Conceptually, `curator.LLM` has two important methods, `prompt` and `parse`. Copy class Poem(BaseModel): poem: str = Field(description="A poem.") class Poems(BaseModel): poems_list: List[Poem] = Field(description="A list of poems.") class Poet(curator.LLM): response_format = Poems ​ def prompt(self, input: Dict) -> str: return f"Write two poems about {input['topic']}." ​ def parse(self, input: Dict, response: Poems) -> Dict: return [{"topic": input["topic"], "poem": p.poem} for p in response.poems] ### [](#prompt) prompt This calls an LLM on each row of the input dataset in parallel. 1. Takes a dataset row as input 2. Returns the prompt for the LLM. ### [](#parse) parse Converts LLM output into structured data by adding it back to the dataset. 1. Takes two arguments: * Input row (this was given to the LLM). * LLM's response (in response\_format --- string or Pydantic) 2. Returns new rows (in list of dictionaries) [](#data-flow-example) Data Flow Example --------------------------------------------- Input Dataset: Copy Row A Row B Processing by curator.LLM: Copy Row A → prompt(A) → Response R1 → parse(A, R1) → [C, D] Row B → prompt(B) → Response R2 → parse(B, R2) → [E, F] Output Dataset: Copy Row C Row D Row E Row F In this example: * The two input rows (A and B) are processed in parallel to prompt the LLM * Each generates a response (R1 and R2) * The parse function converts each response into (multiple) new rows (C, D, E, F) * The final dataset contains all generated rows You can chain `LLM` objects together to iteratively build up a dataset. [PreviousLLM API documentation](/bespoke-curator/api-reference/llm-api-documentation) [NextAPI Service](/bespoke-minicheck/api) Last updated 1 month ago --- # Using Anthropic for batch inference | Bespoke Labs You can use **Gemini** for batch inference in **Curator** to generate synthetic data. In this example, we will generate reannotation of wildchat dataset, but the approach can be adapted for any data generation task. [](#prerequisites) **Prerequisites** ----------------------------------------- * **Python 3.10+** * **Curator**: Install via `pip install bespokelabs-curator` * **Anthropic:** Anthropic API key [](#steps) **Steps** ------------------------- #### [](#id-1.-setup-environment-vars) **1\. Setup environment vars** Copy export ANTHROPIC_API_KEY= **2\. Create a curator.LLM subclass** Create a class that inherits from `curator.LLM`. Implement two key methods: * `prompt()`: Generates the prompt for the LLM. * `parse()`: Processes the LLM's response into your desired format. Here’s the implementation: Copy """Example of reannotating the WildChat dataset using curator.""" import logging from bespokelabs import curator # To see more detail about how batches are being processed logger = logging.getLogger("bespokelabs.curator") logger.setLevel(logging.INFO) class WildChatReannotator(curator.LLM): """A reannotator for the WildChat dataset.""" def prompt(self, input: dict) -> str: """Extract the first message from a conversation to use as the prompt.""" return input["conversation"][0]["content"] def parse(self, input: dict, response: str) -> dict: """Parse the model response along with the input to the model into the desired output format..""" instruction = input["conversation"][0]["content"] return {"instruction": instruction, "new_response": response} #### [](#id-3.-configure-the-anthropic-model) **3\. Configure the Anthropic model** Copy distiller = WildChatReannotator(model_name="claude-3-5-haiku-20241022", batch=True ) #### [](#id-4.-generate-data) **4\. Generate Data** Generate the structured data and output the results as a pandas DataFrame: Copy from datasets import load_dataset dataset = load_dataset("allenai/WildChat", split="train") dataset = dataset.select(range(100)) distilled_dataset = distiller(dataset) print(distilled_dataset) print(distilled_dataset[0]) ### [](#example-output) **Example Output** Using the above example, the output might look like this: instruction new\_response Write a very long, elaborate, descriptive and ... Scene: Omelette Apocalypse\\n\\n\*\*INT. DINER... what are you? I am a large language model, trained by Anthropic [](#batch-configuration) **Batch Configuration** ----------------------------------------------------- * Check out complete [batch configuration](https://docs.bespokelabs.ai/bespoke-curator/api-reference/llm-api-documentation#batch-processing-parameters) [PreviousUsing OpenAI for batch inference](/bespoke-curator/how-to-guides/using-openai-for-batch-inference) [NextUsing Gemini for batch inference](/bespoke-curator/how-to-guides/using-gemini-for-batch-inference) Last updated 1 month ago --- # Quick Tour | Bespoke Labs [](#installation) Installation ----------------------------------- Copy pip install bespokelabs-curator [](#hello-world-with-llm) Hello World with LLM --------------------------------------------------- The `LLM`class provides a flexible interface to generate data with LLMs. Below is a minimal example of using `LLM`: we simply create an `LLM` object with a `model_name`, in this case `gpt-4o-mini`, and passing in a prompt. Copy from bespokelabs import curator llm = curator.LLM(model_name="gpt-4o-mini") poem = llm("Write a poem about the importance of data in AI.") print(poem.to_pandas()) # Output: # response # 0 In the realm where silence once held sway, \n... # Or you can pass a list of prompts to generate multiple responses. poems = llm(["Write a poem about the importance of data in AI.",\ "Write a haiku about the importance of data in AI."]) print(poems.to_pandas()) # Output: # response # 0 In the realm where silence once held sway, \n... # 1 Silent streams of truth, \nData shapes the le... [](#using-different-models) Using Different Models ------------------------------------------------------- You can also use models from other providers by simply changing `model_name` (supported via [LiteLLM](/bespoke-curator/how-to-guides/using-litellm-for-diverse-providers) ): Copy from bespokelabs import curator llm = curator.LLM(model_name="claude-3-5-haiku-20241022") poem = llm("Write a poem about the importance of data in AI.") print(poem.to_pandas()) [](#using-structured-output) Using Structured Output --------------------------------------------------------- ### [](#adding-structured-output-to-your-generation) Adding structured output to your generation Let's look at some more interesting examples of data generation using structured output. Suppose you want to generate multiple poems from a single a LLM call. Structured output is your friend! Using structure output allows you to easily validate and parse LLM responses: Copy from typing import List from pydantic import BaseModel, Field from bespokelabs import curator class Poem(BaseModel): poem: str = Field(description="A poem.") class Poems(BaseModel): poems_list: List[Poem] = Field(description="A list of poems.") llm = curator.LLM(model_name="gpt-4o-mini", response_format=Poems) poems = llm(["Write two poems about the importance of data in AI.", \ "Write three haikus about the importance of data in AI."]) print(poems.to_pandas()) # Output: # poems_list # 0 [{'poem': 'In shadows deep where silence lies,...\ # 1 [{'poem': 'Data whispers truth, \ # Patterns wea...\ \ \ Note how each row in the dataset is now a `Poems` object that is easy to parse and manipulate using Python code.\ \ ### \ \ [](#defining-your-custom-prompting-and-parsing-logic)\ \ Defining your custom prompting and parsing logic\ \ Sometimes, it might not be enough to simply get back the responses. For example, you might want to preserve the mapping between each topic and its corresponding poems, and you might want each poem to occupy only a single row. In this case, you can define a `Poet` object that inherits from `LLM`, and define your custom prompting and parsing logic:\ \ Copy\ \ from typing import Dict, List\ \ from datasets import Dataset\ from pydantic import BaseModel, Field\ \ from bespokelabs import curator\ \ \ class Poem(BaseModel):\ poem: str = Field(description="A poem.")\ \ \ class Poems(BaseModel):\ poems: List[Poem] = Field(description="A list of poems.")\ \ \ class Poet(curator.LLM):\ response_format = Poems\ \ def prompt(self, input: Dict) -> str:\ return f"Write two poems about {input['topic']}."\ \ def parse(self, input: Dict, response: Poems) -> Dict:\ return [{"topic": input["topic"], "poem": p.poem} for p in response.poems]\ \ \ poet = Poet(model_name="gpt-4o-mini")\ \ topics = Dataset.from_dict({"topic": ["Urban loneliness in a bustling city", "Beauty of Bespoke Labs's Curator library"]})\ poem = poet(topics)\ print(poem.to_pandas())\ # Output:\ # topic poem\ # 0 Urban loneliness in a bustling city In the city’s heart, where the lights never di...\ # 1 Urban loneliness in a bustling city Steps echo loudly, pavement slick with rain,\n...\ # 2 Beauty of Bespoke Labs's Curator library In the heart of Curation’s realm, \nWhere art...\ # 3 Beauty of Bespoke Labs's Curator library Step within the library’s embrace, \nA sanctu...\ \ \ In the `Poet` class:\ \ * `response_format` is the structured output class we defined above.\ \ * `prompt` takes the input (`input`) and returns the prompt for the LLM.\ \ * `parse` takes the input (`input`) and the structured output (`response`) and converts it to a list of dictionaries. This is so that we can easily convert the output to a HuggingFace Dataset object.\ \ \ ### \ \ [](#chaining-llm-calls-with-structured-output)\ \ Chaining LLM calls with structured output\ \ Using structured output along with custom prompting and parsing logic allows you to chain together multiple calls to the `LLM` class to create powerful data generation pipelines.\ \ Let's return to our example of generating poems. Suppose we want to also use LLMs to generate the topics of the poems. This can be accomplished by using another `LLM` object to generate the topics, as shown in the example below.\ \ Copy\ \ from typing import Dict, List\ \ from pydantic import BaseModel, Field\ \ from bespokelabs import curator\ \ \ class Topic(BaseModel):\ topic: str = Field(description="A topic.")\ \ \ class Topics(BaseModel):\ topics: List[Topic] = Field(description="A list of topics.")\ \ \ class Muse(curator.LLM):\ response_format = Topics\ \ def prompt(self, input: Dict) -> str:\ return "Generate ten evocative poetry topics."\ \ def parse(self, input: Dict, response: Topics) -> Dict:\ return [{"topic": topic.topic} for topic in response.topics]\ \ \ class Poem(BaseModel):\ poem: str = Field(description="A poem.")\ \ \ class Poems(BaseModel):\ poems: List[Poem] = Field(description="A list of poems.")\ \ \ class Poet(curator.LLM):\ response_format = Poems\ \ def prompt(self, input: Dict) -> str:\ return f"Write two poems about {input['topic']}."\ \ def parse(self, input: Dict, response: Poems) -> Dict:\ return [{"topic": input["topic"], "poem": p.poem} for p in response.poems]\ \ \ muse = Muse(model_name="gpt-4o-mini")\ topics = muse()\ print(topics.to_pandas())\ \ poet = Poet(model_name="gpt-4o-mini")\ poem = poet(topics)\ print(poem.to_pandas())\ # Output:\ # topic poem\ # 0 The fleeting beauty of autumn leaves In a whisper of wind, they dance and they sway...\ # 1 The fleeting beauty of autumn leaves Once vibrant with life, now a radiant fade,\nC...\ # 2 The whispers of an abandoned house In shadows deep where light won’t tread, \nAn...\ # 3 The whispers of an abandoned house Abandoned now, my heart does fade, \nOnce a h...\ # 4 The warmth of a forgotten summer day In the stillness of a memory's embrace, \nA w...\ # 5 The warmth of a forgotten summer day A gentle breeze delivers the trace \nOf a day...\ # ...\ \ Chaining multiple `LLM`calls this way allows us to build powerful synthetic data pipelines that can create millions of examples.\ \ ### \ \ [](#whats-next)\ \ What's next?\ \ For an in-depth tutorial of the core features in our library, please continue to [Tutorials](/bespoke-curator/tutorials)\ .\ \ For how-to guides on specific topics and workflows, please continue to [How-to Guides](/bespoke-curator/how-to-guides)\ .\ \ For an end-to-end example, see [Finetuning a model to identify features of a product](/bespoke-curator/examples/finetuning-a-model-to-identify-features-of-a-product)\ .\ \ [PreviousGetting Started](/bespoke-curator/getting-started)\ [NextTutorials](/bespoke-curator/tutorials)\ \ Last updated 1 month ago --- # Save $$$ with batch mode | Bespoke Labs Providers like OpenAI and Anthropic offer batch mode, which allows you to upload a bunch of prompts to be processed asynchronously, for lower costs (typically 50%). However, these APIs are often very cumbersome to manage: * You typically have to prepare your batch file, upload it, and poll for responses periodically. * Large datasets will typically not fit in a single batch due to batch size limits, and so you will need to split your dataset into mutiple smaller batches, increasing the complexity you need to manage. With Curator, you only need to toggle a single flag to save $$$, without any headache! [](#using-batch-mode) Using batch mode ------------------------------------------- Let's look at a simple example of reannotating instructions from the [WildChat](https://huggingface.co/datasets/allenai/WildChat) dataset with new responses from gpt-4o-mini. First, we need to load the WildChat dataset using HuggingFace: Copy from datasets import load_dataset dataset = load_dataset("allenai/WildChat", split="train") dataset = dataset.select(range(3_000)) # Select a subset of 3,000 samples We then create a new `LLM` class and apply to `dataset`. All you need to do to enable batching is setting `batch=True` when initializing your `LLM` object, and you're done! Copy from bespokelabs import curator class WildChatReannotator(curator.LLM): """A reannotator for the WildChat dataset.""" def prompt(self, input: dict) -> str: """Extract the first message from a conversation to use as the prompt.""" return input["conversation"][0]["content"] def parse(self, input: dict, response: str) -> dict: """Parse the model response along with the input to the model into the desired output format.""" instruction = input["conversation"][0]["content"] return {"instruction": instruction, "new_response": response} # Initialize the reannotator with batch processing reannotator = WildChatReannotator( model_name="gpt-4o-mini", batch=True, # Enable batch processing backend_params={"batch_size": 1_000}, # Specify batch size ) reannotated_dataset = reannotator(dataset) [](#supported-models) Supported Models ------------------------------------------- Check out how-to guides for using batch mode with our supported providers: * [Using OpenAI for batch inference](/bespoke-curator/how-to-guides/using-openai-for-batch-inference) * [Using Anthropic for batch inference](/bespoke-curator/how-to-guides/using-anthropic-for-batch-inference) * [Using Gemini for batch inference](/bespoke-curator/how-to-guides/using-gemini-for-batch-inference) * [Using kluster.ai for batch inference](/bespoke-curator/how-to-guides/using-kluster.ai-for-batch-inference) Feel free to tell us which providers you want us to add support for, or send a PR if you want to [contribute](https://github.com/bespokelabsai/curator/blob/main/CONTRIBUTING.md) ! [PreviousAutomatic recovery and caching](/bespoke-curator/tutorials/automatic-recovery-and-caching) [NextHow-to Guides](/bespoke-curator/how-to-guides) Last updated 1 month ago --- # Using vLLM with Curator | Bespoke Labs You can use VLLM as a backend for Curator in two modes: offline (local) and online (server). This guide demonstrates both approaches using structured recipe generation as an example. [](#prerequisites) Prerequisites ------------------------------------- * Python 3.10+ * Curator: Install via `pip install bespokelabs-curator` * VLLM: Install via `pip install vllm` [](#offline-mode-local) Offline Mode (Local) ------------------------------------------------- In offline mode, VLLM runs locally on your machine, loading the model directly into memory. ### [](#id-1-create-pydantic-models-for-structured-output) 1\. Create Pydantic Models for Structured Output First, define your data structure using Pydantic models: Copy from pydantic import BaseModel, Field from typing import List class Recipe(BaseModel): title: str = Field(description="Title of the recipe") ingredients: List[str] = Field(description="List of ingredients needed") instructions: List[str] = Field(description="Step by step cooking instructions") prep_time: int = Field(description="Preparation time in minutes") cook_time: int = Field(description="Cooking time in minutes") servings: int = Field(description="Number of servings") ### [](#id-2-create-a-curator-llm-subclass) 2\. Create a Curator LLM Subclass Create a class that inherits from `LLM` and implement two key methods: Copy from bespokelabs import curator class RecipeGenerator(curator.LLM): response_format = Recipe def prompt(self, input: dict) -> str: return f"Generate a random {input['cuisine']} recipe. Be creative but keep it realistic." def parse(self, input: dict, response: Recipe) -> dict: return { "title": response.title, "ingredients": response.ingredients, "instructions": response.instructions, "prep_time": response.prep_time, "cook_time": response.cook_time, "servings": response.servings, } #### [](#id-3-initialize-and-use-the-generator) 3\. Initialize and Use the Generator Copy # Initialize with a local model generator = RecipeGenerator( model_name="Qwen/Qwen2.5-3B-Instruct", backend="vllm", backend_params={ "tensor_parallel_size": 1, # Adjust based on GPU count "gpu_memory_utilization": 0.7 } ) # Create input dataset cuisines = [{"cuisine": c} for c in ["Italian", "Chinese", "Mexican"]] recipes = generator(cuisines) print(recipes.to_pandas()) [](#online-mode-server) Online Mode (Server) ------------------------------------------------- In online mode, VLLM runs as a server that can handle multiple requests. ### [](#id-1-start-the-vllm-server) 1\. Start the VLLM Server Start the VLLM server with your chosen model: Copy vllm serve Qwen/Qwen2.5-3B-Instruct \ --host localhost \ --port 8787 \ --api-key token-abc123 ### [](#id-2-configure-the-generator) 2\. Configure the Generator Use the same Pydantic models and LLM subclass as in offline mode, but initialize with server configuration: Copy # Set API key if required os.environ["HOSTED_VLLM_API_KEY"] = "token-abc123" # Initialize with server connection generator = RecipeGenerator( model_name="hosted_vllm/Qwen/Qwen2.5-3B-Instruct", backend="litellm", backend_params={ "base_url": "http://localhost:8787/v1", "request_timeout": 30 } ) # Generate recipes recipes = generator(cuisines) print(recipes.to_pandas()) [](#example-output) Example Output --------------------------------------- The generated recipes will be returned as structured data like: Copy { "title": "Spicy Szechuan Noodles", "ingredients": [ \ "400g wheat noodles", \ "2 tbsp Szechuan peppercorns", \ "3 cloves garlic, minced", \ "2 tbsp soy sauce" \ ], "instructions": [ \ "Boil noodles according to package instructions", \ "Heat oil in a wok over medium-high heat", \ "Add peppercorns and garlic, stir-fry until fragrant", \ "Add noodles and soy sauce, toss to combine" \ ], "prep_time": 15, "cook_time": 20, "servings": 4 } [](#configuration-options) VLLM Offline Configuration ---------------------------------------------------------- #### [](#backend-parameters) Backend Parameters (for Offline Mode) * `tensor_parallel_size`: Number of GPUs for tensor parallelism (default: 1) * `gpu_memory_utilization`: GPU memory usage fraction between 0 and 1 (default: 0.95) * `max_model_length`: Maximum sequence length (default: 4096) * `max_tokens`: Maximum number of tokens to generate (default: 4096) * `min_tokens`: Minimum number of tokens to generate (default: 1) * `enforce_eager`: Whether to enforce eager execution (default: False) * `batch_size`: Size of batches for processing (default: 256) [PreviousUsing LiteLLM for diverse providers](/bespoke-curator/how-to-guides/using-litellm-for-diverse-providers) [NextUsing OpenAI for batch inference](/bespoke-curator/how-to-guides/using-openai-for-batch-inference) Last updated 1 month ago --- # Using Gemini for batch inference | Bespoke Labs You can use **Gemini** for batch inference in **Curator** to generate synthetic data. In this example, we will generate reannotation of wildchat dataset, but the approach can be adapted for any data generation task. [](#prerequisites) **Prerequisites** ----------------------------------------- * **Python 3.10+** * **Curator**: Install via `pip install bespokelabs-curator` * **Gemini (Vertex AI):** GCP account with Vertex AI enabled. * **Google Cloud Bucket**: Access to cloud storage. [](#steps) **Steps** ------------------------- #### [](#id-1.-setup-environment-vars) **1\. Setup environment vars** Copy export GOOGLE_CLOUD_REGION=us-central1 export GOOGLE_CLOUD_PROJECT= export GEMINI_BUCKET_NAME= export GEMINI_API_KEY= #### [](#id-2.-adc-authentication) **2\. ADC authentication** Copy gcloud auth application-default login #### [](#id-3.-create-a-curator.llm-subclass) **3\. Create a curator.LLM subclass** Create a class that inherits from `curator.LLM`. Implement two key methods: * `prompt()`: Generates the prompt for the LLM. * `parse()`: Processes the LLM's response into your desired format. Here’s the implementation: Copy """Example of reannotating the WildChat dataset using curator.""" import logging from bespokelabs import curator # To see more detail about how batches are being processed logger = logging.getLogger("bespokelabs.curator") logger.setLevel(logging.INFO) class WildChatReannotator(curator.LLM): """A reannotator for the WildChat dataset.""" def prompt(self, input: dict) -> str: """Extract the first message from a conversation to use as the prompt.""" return input["conversation"][0]["content"] def parse(self, input: dict, response: str) -> dict: """Parse the model response along with the input to the model into the desired output format..""" instruction = input["conversation"][0]["content"] return {"instruction": instruction, "new_response": response} #### [](#id-3.-configure-the-gemini-backend) **3\. Configure the Gemini Backend** Copy distiller = WildChatReannotator(model_name="gemini-1.5-flash-002", backend="gemini", batch=True ) #### [](#id-4.-generate-data) **4\. Generate Data** Generate the structured data and output the results as a pandas DataFrame: Copy from datasets import load_dataset dataset = load_dataset("allenai/WildChat", split="train") dataset = dataset.select(range(100)) distilled_dataset = distiller(dataset) print(distilled_dataset) print(distilled_dataset[0]) ### [](#example-output) **Example Output** Using the above example, the output might look like this: instruction new\_response Write a very long, elaborate, descriptive and ... Scene: Omelette Apocalypse\\n\\n\*\*INT. DINER... what are you? I am a large language model, trained by Google [](#gemini-batch-configuration) **Gemini Batch Configuration** ------------------------------------------------------------------- * Check out complete [batch configuration](https://docs.bespokelabs.ai/bespoke-curator/api-reference/llm-api-documentation#batch-processing-parameters) * Check out Gemini [generation parameters](https://cloud.google.com/vertex-ai/generative-ai/docs/reference/python/latest/vertexai.generative_models.GenerationConfig) [PreviousUsing Anthropic for batch inference](/bespoke-curator/how-to-guides/using-anthropic-for-batch-inference) [NextUsing kluster.ai for batch inference](/bespoke-curator/how-to-guides/using-kluster.ai-for-batch-inference) Last updated 1 month ago --- # API Service | Bespoke Labs Using the api service is quite easy. #### [](#step-1-api-key-setup) Step 1: API Key Setup First, get your API key at the [Bespoke Console](https://console.bespokelabs.ai) . Copy export BESPOKE_API_KEY=besoke-... #### [](#step-2-install-dependencies) Step 2: Install Dependencies Install the package: Copy pip install bespokelabs #### [](#step-3-run) Step 3: Run Copy import os from bespokelabs import BespokeLabs bl = BespokeLabs( # This is the default and can be omitted auth_token=os.environ.get("BESPOKE_API_KEY"), ) response = bl.minicheck.factcheck.create( claim="claim", context="context", ) print(response.support_prob) Lot more information about the library is available at the [bespokelabs pypi page](https://pypi.org/project/bespokelabs/) . [PreviousConceptual Guides](/bespoke-curator/conceptual-guides) [NextSelf-Hosting](/bespoke-minicheck/hosting) Last updated 5 months ago --- # Automatic recovery and caching | Bespoke Labs Curator automatically caches the output generated by the `LLM` class. This is very useful for: * Recovering from failures and interruption: During large data generation runs, you can run into unexpected failures or interruption. Caching partially completed responses from a data generation run allows you to recover from the latest completed output instead of starting from scratch when you restart your run. * Caching previous completed runs: When working with multi-stage pipelines, you might want to reuse earlier stages in the pipeline while iterating on the later stages. Caching previously completed runs from the earlier stages allows you to iterate quickly while saving time & money. To see caching in action, try running the Hello World example below twice. The second run should reuse the cached responses from the first run instead of making an LLM call. Copy from bespokelabs import curator llm = curator.LLM(model_name="gpt-4o-mini") poem = llm("Write a poem about the importance of data in AI.") print(poem.to_pandas()) [](#disable-caching) Disable caching ----------------------------------------- To disable caching, you can simply set `CURATOR_DISABLE_CACHE=1` before generating your data. [](#custom-cache-directory) Custom cache directory ------------------------------------------------------- By default, all cached datasets are saved to `~/.cache/curator`, but you can change it in two ways: 1. Setting the CURATOR\_CACHE\_DIR environmental variable to the desired directory 2. Passing the desired directory to the `working_dir` parameter when applying the `LLM` object on a dataset, e.g. `llm("Write a poem about the importance of data in AI.", working_dir="/path/to/my/poems")`. [](#cache-internals-subject-to-future-changes) Cache Internals (subject to future changes) ----------------------------------------------------------------------------------------------- The cache directory contains the following: 1. metadata.db**:** a SQLite database containing metadata about data generation runs 2. cache directories of individual data generation runs: each directory is named after the fingerprint of the data generation run. Copy >> ls ~/.cache/curator 032bc5ead2892f8f 6d1f31229726231d 137851647e75f9a7 91f4ad23d5821c9f 24b1d8917f7ef6f1 a2a3c8e5a58e3fc3 metadata.db The fingerprint of a data generation run is based on the following: 1. The input dataset on which the `LLM` object is being applied. 2. The `prompt` function of the `LLM` object. 3. Whether or not the data generation is using batch mode 4. The response format of the `LLM` object. 5. The model name defined in the `LLM` object. 6. Generation parameters, e.g. temperature, top\_k, etc. [](#troubleshooting) Troubleshooting ----------------------------------------- ### [](#corrupt-or-full-working-directory) Corrupt or full working directory The cache directory can get too large or become corrupt due to unexpected errors. You can recover from these types of failures by deleting the cache directory: `rm -rf ~/.cache/curator`. Note that this will delete \*all\* cached responses. [PreviousVisualize your dataset with the Bespoke Curator Viewer](/bespoke-curator/tutorials/visualize-your-dataset-with-the-bespoke-curator-viewer) [NextSave $$$ with batch mode](/bespoke-curator/tutorials/save-usdusdusd-with-batch-mode) Last updated 26 days ago --- # Visualize your dataset with the Bespoke Curator Viewer | Bespoke Labs [PreviousTutorials](/bespoke-curator/tutorials) [NextAutomatic recovery and caching](/bespoke-curator/tutorials/automatic-recovery-and-caching) Last updated 1 month ago The Bespoke Curator Viewer provides real-time updates on the progress of your data generation runs and allows you to closely inspect the data being generated. You can start the viewer by simply running Copy curator-viewer This will pop up a browser window with the viewer running at [http://localhost:3000](http://localhost:3000) by default if you haven't specified a different host and port. [](#curator-runs-history) Curator Runs History --------------------------------------------------- The first view that you will see after navigating to the Bespoke Curator Viewer is the Runs History view, which shows all the different runs you created with Curator. [](#dataset-details) Dataset Details ----------------------------------------- Clicking on a run will open up the Dataset Details view, which shows more details about the dataset being generated by the run. The graph on the top provides some aggregate statistics about the data generation run, such as the number of requests and responses over time, or the number and distribution of prompt and completion tokens being generated. You can change the statistics being visualized by clicking on the button on the top-right corner ("Requests & Responses" in the screenshot above). The table at the bottom provides details about the content of the requests being processed. Clicking on a row lets you see the full user and assistant messages. [](#running-bespoke-curator-viewer-on-a-different-port-or-host) Running Bespoke Curator Viewer on a different port or host ------------------------------------------------------------------------------------------------------------------------------- You can run the Bespoke Curator Viewer on a different host and port by setting the `--host` and `--port` optional parameters. Copy >>> curator-viewer -h usage: curator-viewer [-h] [--host HOST] [--port PORT] [--verbose] Curator Viewer options: -h, --help show this help message and exit --host HOST Host to run the server on (default: localhost) --port PORT Port to run the server on (default: 3000) --verbose, -v Enables debug logging for more verbose output [](#troubleshooting) Troubleshooting ----------------------------------------- ### [](#installing-node) Installing Node The `curator-viewer` requires [Node.js](https://nodejs.org/) to be installed and will fail if it can't find Node.js on your system. You can install them by following the instructions [here](https://nodejs.org/en/download/package-manager) . ![](https://docs.bespokelabs.ai/~gitbook/image?url=https%3A%2F%2F1831689742-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FINVkBnpIXpC4135NE6Ex%252Fuploads%252Fsq3vUDGpWUWRIQhzPoO3%252Fcurator-viewer.gif%3Falt%3Dmedia%26token%3D9c51b78e-4d7a-47c0-b7b5-29ce66d08f73&width=768&dpr=4&quality=100&sign=4d890c65&sv=2) ![](https://docs.bespokelabs.ai/~gitbook/image?url=https%3A%2F%2F1831689742-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FINVkBnpIXpC4135NE6Ex%252Fuploads%252FSAtBvKc7gsPPWF8UFfkg%252FScreenshot%25202025-01-14%2520at%25207.50.14%25E2%2580%25AFPM.png%3Falt%3Dmedia%26token%3D9ee9ca9e-adec-4918-b51e-9a52404e112f&width=768&dpr=4&quality=100&sign=f9ecca06&sv=2) ![](https://docs.bespokelabs.ai/~gitbook/image?url=https%3A%2F%2F1831689742-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FINVkBnpIXpC4135NE6Ex%252Fuploads%252Fo8Qy3hqoFFtp1O4ELsXq%252FScreenshot%25202025-01-14%2520at%25207.50.29%25E2%2580%25AFPM.png%3Falt%3Dmedia%26token%3D4980a428-a7fc-49ba-8966-5ef3d255cca2&width=768&dpr=4&quality=100&sign=bd6a6b90&sv=2) ![](https://docs.bespokelabs.ai/~gitbook/image?url=https%3A%2F%2F1831689742-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FINVkBnpIXpC4135NE6Ex%252Fuploads%252FnWvdU2x4P0QZRi1787s4%252FScreenshot%25202025-01-14%2520at%25207.50.50%25E2%2580%25AFPM.png%3Falt%3Dmedia%26token%3Dd6d8af8d-a29f-45db-a1ae-cff087824db0&width=768&dpr=4&quality=100&sign=435ae915&sv=2) --- # LLM API documentation | Bespoke Labs [](#curator.llm) curator.LLM --------------------------------- The `LLM` class serves as the primary interface for prompting Large Language Models in Curator. It provides a flexible and extensible way to generate synthetic data using various LLM providers. ### [](#class-definition) Class Definition Copy class LLM: def __init__( self, model_name: str, response_format: Type[BaseModel] | None = None, batch: bool = False, backend: Optional[str] = None, generation_params: dict | None = None, backend_params: BackendParamsType | None = None, ) ### [](#constructor-parameters) Constructor Parameters Parameter Type Default Description `model_name` `str` Required Name of the LLM to use `response_format` `Type[BaseModel] | None` `None` Pydantic model specifying the expected response format `batch` `bool` `False` Enable batch processing mode `backend` `Optional[str]` `None` LLM backend to use ("openai", "litellm", or "vllm"). Auto-determined if None `generation_params` `dict | None` `None` Additional parameters for the generation API `backend_params` `BackendParamsType | None` `None` Configuration parameters for request processor ### [](#backend-parameters-configuration) Backend Parameters Configuration The `backend_params` dictionary supports various configuration options based on the execution mode. Here's a comprehensive breakdown: #### [](#common-parameters) Common Parameters These parameters are available across all backends: Parameter Type Default Description `max_retries` `int` `3` Maximum number of retry attempts for failed requests `require_all_responses` `bool` `False` Whether to require successful responses for all prompts `base_url` `Optional[str]` `None` Optional base URL for API endpoint `request_timeout` `int` `600` Timeout in seconds for each request Copy # Example: Common parameters configuration backend_params = { "max_retries": 3, "require_all_responses": True, "base_url": "https://custom-endpoint.com/v1", "request_timeout": 300 } #### [](#online-mode-parameters) Online Mode Parameters Parameters for online processor mode: Parameter Type Description `max_requests_per_minute` `int` Maximum number of API requests per minute `max_tokens_per_minute` `int` Maximum number of tokens per minute `seconds_to_pause_on_rate_limit` `float` Duration to pause when rate limited Copy # Example: Online mode configuration backend_params = { "max_requests_per_minute": 2000, "max_tokens_per_minute": 4_000_000, "seconds_to_pause_on_rate_limit": 15.0 } #### [](#batch-processing-parameters) Batch Processing Parameters Parameters available when `batch=True`: Parameter Type Description `batch_size` `int` Number of prompts to process in each batch `batch_check_interval` `float` Time interval between batch completion checks `delete_successful_batch_files` `bool` Whether to delete successful batch files `delete_failed_batch_files` `bool` Whether to delete failed batch files Copy # Example: Batch processing configuration backend_params = { "batch_size": 100, "batch_check_interval": 1.0, "delete_successful_batch_files": True, "delete_failed_batch_files": False, } #### [](#offline-mode-parameters-vllm) Offline Mode Parameters (VLLM) Parameters for local model deployment with VLLM: Parameter Type Description `tensor_parallel_size` `int` Number of GPUs for tensor parallelism `enforce_eager` `bool` Whether to enforce eager execution `max_model_length` `int` Maximum sequence length for the model `max_tokens` `int` Maximum tokens for generation `min_tokens` `int` Minimum tokens for generation `gpu_memory_utilization` `float` Target GPU memory utilization (0.0 to 1.0) `batch_size` `int` Batch size for VLLM processing Copy # Example: VLLM configuration backend_params = { "tensor_parallel_size": 2, "max_model_length": 4096, "max_tokens": 2048, "min_tokens": 1, "gpu_memory_utilization": 0.85, "batch_size": 32 } ### [](#methods) Methods ### [](#prompt) prompt() Copy def prompt(self, input: _DictOrBaseModel) -> _DictOrBaseModel Generates a prompt for the LLM based on the input data. **Parameters** * `input`: Input row used to construct the prompt **Returns** A prompt that can be either: 1. A string for a single user prompt 2. A list of dictionaries for multiple messages **Example** Copy def prompt(self, input: dict) -> str: return f"Generate a {input['type']} about {input['topic']}" ### [](#parse) parse() Copy def parse(self, input: _DictOrBaseModel, response: _DictOrBaseModel) -> _DictOrBaseModel Processes the LLM's response and optionally can be used to combine it with the input data. **Parameters** * `input`: Original input row used for the prompt * `response`: Raw response from the LLM **Returns** A parsed output combining the input and response data **Example** Copy def parse(self, input: dict, response: str) -> dict: return { "prompt_topic": input["topic"], "generated_text": response, "timestamp": datetime.now().isoformat() } ### [](#response-format-optional) Response Format (Optional) The `response_format` class attribute can be set to a Pydantic model to enforce structured output: Copy from pydantic import BaseModel class RecipeResponse(BaseModel): title: str ingredients: List[str] instructions: List[str] class RecipeGenerator(LLM): response_format = RecipeResponse ### [](#usage-examples) Usage Examples #### [](#basic-usage) Basic Usage Copy class Cuisines(BaseModel): """A list of cuisines.""" cuisines_list: List[str] = Field(description="A list of cuisines.") class CuisineGenerator(curator.LLM): """A cuisine generator that generates diverse cuisines.""" response_format = Cuisines def prompt(self, input: dict) -> str: """Generate a prompt for the cuisine generator.""" return "Generate 10 diverse cuisines." def parse(self, input: dict, response: Cuisines) -> dict: """Parse the model response along with the input to the model into the desired output format..""" return [{"cuisine": t} for t in response.cuisines_list] [PreviousAPI reference](/bespoke-curator/api-reference) [NextConceptual Guides](/bespoke-curator/conceptual-guides) Last updated 1 month ago --- # Using kluster.ai for batch inference | Bespoke Labs You can use **kluster.ai** for batch inference in **Curator** to generate synthetic data. In this example, we will generate answers for GSM8K dataset, but the approach can be adapted for any data generation task. [](#prerequisites) **Prerequisites** ----------------------------------------- * **Python 3.10+** * **Curator**: Install via `pip install bespokelabs-curator` * **kluster.ai API key:** Get your key from [https://www.kluster.ai/](https://www.kluster.ai/) [](#steps) **Steps** ------------------------- #### [](#id-1.-setup-environment-vars) **1\. Setup environment vars** Copy export KLUSTERAI_API_KEY= **2\. Create a curator.LLM subclass** Create a class that inherits from `curator.LLM`. Implement two key methods: * `prompt()`: Generates the prompt for the LLM. * `parse()`: Processes the LLM's response into your desired format. Here’s the implementation: Copy """Example of reannotating the WildChat dataset using curator.""" import logging from bespokelabs import curator # To see more detail about how batches are being processed logger = logging.getLogger("bespokelabs.curator") logger.setLevel(logging.INFO) class Reasoner(curator.LLM): """Curator class for processing GSM8K dataset.""" def prompt(self, input): """Create a prompt for the LLM to reason about the problem.""" return f"Answer the following question: {input['question']}" def parse(self, input, response): """Parse the LLM response to extract reasoning and solution. The response format is expected to be 'reasoninganswer' """ full_response = response # Extract reasoning and answer using regex import re reasoning_pattern = r"(.*?)" reasoning_match = re.search(reasoning_pattern, full_response, re.DOTALL) reasoning = reasoning_match.group(1).strip() if reasoning_match else "" # Answer is everything after answer = re.sub(reasoning_pattern, "", full_response, flags=re.DOTALL).strip() return [\ {\ "question": input["question"],\ "reasoning": reasoning,\ "deepseek_solution": answer,\ "gold_answer": input["answer"],\ }\ ] #### [](#id-3.-configure-reasoner-to-use-deepseek-r1-through-kluster.ai) **3\. Configure Reasoner to use DeepSeek-R1 through kluster.ai** Copy reasoner = Reasoner(model_name="deepseek-ai/DeepSeek-R1", backend="klusterai", batch=True, backend_params={"max_retries": 1, "completion_window": "1h"}) #### [](#id-4-generate-data) **4 Generate Data** Generate the structured data and output the results as a pandas DataFrame: Copy from datasets import load_dataset dataset = load_dataset("openai/gsm8k", name="main") dataset_to_use = dataset["train"].take(3) output = reasoner(dataset) ### [](#example-output) **Example Output** Using the above example, the output might look like this: Copy from IPython.display import HTML, display, Markdown which = 0 question = output[which]['question'] gold_answer = output[which]['gold_answer'] model_answer = output[which]['deepseek_solution'] thought = output[which]['reasoning'] to_display_input = question.replace("\n", "
") to_display_output = model_answer.replace("\n", "
") display(Markdown( "

Question

" f"

{question}

" )) display(Markdown( "

Model answer

" f"

{model_answer}

" )) display(Markdown( "

Gold answer

" f"

{gold_answer}

" )) display(Markdown( "

Model Thought

" f"

{thought}

" )) [](#batch-configuration) **Batch Configuration** ----------------------------------------------------- * Check out complete [batch configuration](https://docs.bespokelabs.ai/bespoke-curator/api-reference/llm-api-documentation#batch-processing-parameters) [PreviousUsing Gemini for batch inference](/bespoke-curator/how-to-guides/using-gemini-for-batch-inference) [NextExecute LLM-generated code](/bespoke-curator/how-to-guides/execute-llm-generated-code) Last updated 1 month ago ![](https://docs.bespokelabs.ai/~gitbook/image?url=https%3A%2F%2F1831689742-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FINVkBnpIXpC4135NE6Ex%252Fuploads%252FwcW8XO6RtdOyid7IiwlF%252FScreenshot%25202025-01-30%2520at%252012.42.39%25E2%2580%25AFPM.png%3Falt%3Dmedia%26token%3Debb826db-015f-446e-8081-8ed12a62e997&width=768&dpr=4&quality=100&sign=ffaabe88&sv=2) --- # Self-Hosting | Bespoke Labs You can access the model here: [https://huggingface.co/bespokelabs/Bespoke-Minicheck-7B](https://huggingface.co/bespokelabs/Bespoke-Minicheck-7B) Feel free to use this [colab](https://colab.research.google.com/drive/1s-5TYnGV3kGFMLp798r5N-FXPD8lt2dm?usp=sharing) which uses the MiniCheck library that supports automated chunking of long documents. Or, you can host the model directly on vLLM with docker as follows: Copy ```shellscript sudo docker run \ --runtime=nvidia \ --gpus=all \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HUGGING_FACE_HUB_TOKEN=hf_xyz" \ --ipc=host \ -p 8000:8000 \ vllm/vllm-openai:latest \ --model bespokelabs/Bespoke-MiniCheck-7B --trust_remote_code --api-key your_api_key --disable-log-requests \ --dtype bfloat16 \ --max-model-len 32768 \ --tensor-parallel-size 1 & ``` Please contact us for commercial licensing. [PreviousAPI Service](/bespoke-minicheck/api) [NextIntegrations](/bespoke-minicheck/integrations) Last updated 5 months ago --- # API reference | Bespoke Labs [LLM API documentation](/bespoke-curator/api-reference/llm-api-documentation) [PreviousFinetuning a model to identify features of a product](/bespoke-curator/examples/finetuning-a-model-to-identify-features-of-a-product) [NextLLM API documentation](/bespoke-curator/api-reference/llm-api-documentation) --- # Execute LLM-generated code | Bespoke Labs We have built a code-executor that can be used to execute LLM-generated code. This is useful for many situations: 1. You want to include error-free code in your training code. This method is used in [Open Thoughts](https://open-thoughts.ai) . 2. LLM generates some code to generate visualization etc. 3. Agents and tool-use. Here is a simple example of code execution in action: Copy from bespokelabs import curator from datasets import Dataset class HelloExecutor(curator.CodeExecutor): def code(self, row): return """location = input();print(f"Hello {location}")""" def code_input(self, row): return row['location'] def code_output(self, row, execution_output): row['output'] = execution_output.stdout return row locations = Dataset.from_list([{'location': 'New York'},{'location': 'Tokyo'}]) hello_executor = HelloExecutor() print(hello_executor(locations).to_pandas()) The inherited class contains three methods: 1. `code`: This is the method that returns the piece of code to be run. This is usually part of the row (you can use `curator.LLM` to generate this code). 2. `code_input`: This is optional, but can return a json that represents values to be passed to `input()` in the code. 3. `code_output`: This is where you parse the output of the execution. We offer three methods of running the code: 1. Multiprocessing: This is the default backend (and can be activated as `CodeExecutor(backend="multiprocessing")` This runs code locally and is therefore the least safe option. 2. Docker: Use `CodeExecutor(backend="docker")`to use Docker to run the code. Safer option than multiprocessing. 3. Ray: If you have a ray cluster, you can use it by setting `CodeExecutor(backend="ray")`. This is useful when your code can take a long time to run. 4. E2B: Code can also be run using [e2b.dev](https://e2b.dev/) . Use `CodeExecutor(backend="e2b")` . [PreviousUsing kluster.ai for batch inference](/bespoke-curator/how-to-guides/using-kluster.ai-for-batch-inference) [NextExamples](/bespoke-curator/examples) Last updated 5 days ago --- # Using Ollama with Curator | Bespoke Labs You can use **Ollama** as a backend for **Curator** to generate structured synthetic data. In this example, we will generate a list of countries and their capitals, but the approach can be adapted for any data generation task. [](#prerequisites) **Prerequisites** ----------------------------------------- * **Python 3.10+** * **Curator**: Install via `pip install bespokelabs-curator` * **Ollama:** Download via [https://ollama.com/download](https://ollama.com/download) [](#steps) **Steps** ------------------------- #### [](#id-1.-create-a-curator.llm-subclass) **1\. Create a curator.LLM subclass** Create a class that inherits from `curator.LLM`. Implement two key methods: * `prompt()`: Generates the prompt for the LLM. * `parse()`: Processes the LLM's response into your desired format. Here’s the implementation: Copy from bespokelabs import curator from pydantic import BaseModel, Field class Location(BaseModel): country: str = Field(description="The name of the country") capital: str = Field(description="The name of the capital city") class LocationList(BaseModel): locations: list[Location] = Field(description="A list of locations") class SimpleOllamaGenerator(curator.LLM): response_format = LocationList def prompt(self, input: dict) -> str: return "Return five countries and their capitals." def parse(self, input: dict, response: str) -> dict: return [{"country": output.country, "capital": output.capital} for output in response.locations] ### [](#id-2.-configure-the-ollama-backend) **2\. Configure the Ollama Backend** 1. Start Ollama server with `llama3.1:8b`model. Copy ollama pull llama3.1:8b ollama serve 1. Initialize your generator with Ollama configuration: Copy llm = SimpleOllamaGenerator( model_name="ollama/llama3.1:8b", # Ollama model identifier backend_params={"base_url": "http://localhost:11434"}, # Ollama instance ) ### [](#id-3.-generate-data) **3\. Generate Data** Generate the structured data and output the results as a pandas DataFrame: Copy locations = llm() print(locations.to_pandas()) ### [](#example-output) **Example Output** Using the above example, the output might look like this: Country Capital France Paris Japan Tokyo Germany Berlin India New Delhi Brazil Brasília [](#ollama-configuration) **Ollama Configuration** ------------------------------------------------------- Use `base_url` in the `backend_params` to specify the connection URL. Example: Copy backend_params={"base_url": "http://localhost:11434"} [PreviousHow-to Guides](/bespoke-curator/how-to-guides) [NextUsing LiteLLM for diverse providers](/bespoke-curator/how-to-guides/using-litellm-for-diverse-providers) Last updated 1 month ago --- # Finetuning a model to identify features of a product | Bespoke Labs [PreviousExamples](/bespoke-curator/examples) [NextAPI reference](/bespoke-curator/api-reference) Last updated 1 month ago **Note:** This example requires a GPU for finetuning. If you don't have a machine with GPUs handy, you can use the Colab version below with free T4 GPUs. [](https://colab.research.google.com/drive/1YoA23-cBcWpaSErULzBI2bo2LPGo37GQ) We will go through a small example here to create data with Ollama using Curator, finetune with Unsloth, and then evaluate it again using Curator. Imagine you are a product wizard at a fictional product company called Azanom Inc., and want to highlight product features in the description of each product. Code for displaying product[](#code-for-displaying-product) Copy from IPython.display import HTML, display import re def display_product( product_name, description, features, image_url, ): """Displays a product give its product_name, features, and description""" # Product description and features def highlight_features(text, features): # Sort features by length in descending order to handle overlapping matches sorted_features = sorted(features, key=len, reverse=True) # Create a copy of the text for highlighting highlighted_text = text # Replace each feature with its highlighted version for feature in sorted_features: pattern = re.compile(re.escape(feature), re.IGNORECASE) highlighted_text = pattern.sub( f'{feature}', highlighted_text ) return highlighted_text # Create HTML content with CSS styling html_content = f"""

{product_name}

""" if image_url: html_content += f'{product_name}' html_content += f"""

{highlight_features(description, features)}

""" display(HTML(html_content)) display_product( product_name="Apple Airpods Pro", description="The Apple AirPods Pro are a pair of wireless earbuds that are designed for comfort and convenience. They are lightweight in-ear earbuds and contoured for a comfortable fit, and they sit at an angle for easy access to the controls. The AirPods Pro also have a stem that is 33% shorter than the second generation AirPods, which makes them more compact and easier to store. The AirPods Pro also have a force sensor to easily control music and calls, and they have Spatial Audio with dynamic head tracking, which provides an immersive, three-dimensional listening experience.", features=[\ "lightweight in-ear earbuds",\ "contoured design",\ "sits at an angle for comfort",\ "better direct audio to your ear",\ "stem is 33% shorter than the second generation AirPods",\ "force sensor to easily control music and calls",\ "Spatial Audio with dynamic head tracking",\ "immersive, three-dimensional listening experience"], image_url="https://store.storeimages.cdn-apple.com/4982/as-images.apple.com/is/airpods-pro-2-hero-select-202409_FMT_WHH?wid=750&hei=556&fmt=jpeg&qlt=90&.v=1724041668836") Given a product and its description, your first instinct is to use GPT-4o, to get the features given a product description. But you quickly realize that you don't need a jackhammer to nail this one and want to find a much cheaper and scalable alternative. So let's try to train a 1B model by generating data from a 8B model. This data generation should cost $0. We can always use bigger models to generate higher-quality data. Note that we have simplified this example for demonstration purposes. [](#installation) Installation ----------------------------------- Copy # Install Python packages !pip install bespokelabs-curator==0.1.15.post1 !pip install fuzzywuzzy datasets pydantic !pip install unsloth !pip install --force-reinstall --no-cache-dir --no-deps git+https://github.com/unslothai/unsloth.git !pip install bitsandbytes triton unsloth_zoo # Install ollama !curl https://www.ollama.com/install.sh | OLLAMA_VERSION="0.5.4" sh # We need llama3.1:8b for data generation and llama3.2:1b model for finetuning !ollama pull llama3.1:8b !ollama pull llama3.2:1b Import the required library Copy # Make sure the following imports fine before proceeding, since this is needed for finetuning. from unsloth import FastLanguageModel from bespokelabs import curator import os import re import json import torch import random import numpy as np from typing import List from fuzzywuzzy import fuzz from pydantic import BaseModel, Field from datasets import Dataset, load_dataset [](#generate-training-data-using-llama-3.1-8b) Generate training data using Llama-3.1-8B --------------------------------------------------------------------------------------------- Our goal is to extract features from the product descriptions. We will use `Curator` to easily generate a dataset of products and their features. We seed the dataset with personas from PersonaHub and create products for each persona for diverse products. To make the data generation process easy for LLMs, we include the and tag in the output description. This way, we get high quality descriptions for given features. Copy # load the personas dataset personas = load_dataset("proj-persona/PersonaHub", 'persona') personas = personas['train'].take(100) personas[0] We can then create a `ProductCurator` object and curate products using personas Copy class ProductCurator(curator.LLM): # input prompt to the curator def prompt(self, row): return f"""Generate a product for the following persona: {row['persona']} The product should be a product that is relevant to the persona. Give a name, description and features for the product. Give upto 10 features for the product. Features should be relevant, extremely detailed and useful for the persona. Then, generate a description for the product. Note that each feature should exactly be mentioned in the description. Do not add any other features or miss any features. An example output is: {{ "name": "Apple AirPods Pro", "features": [\ "lightweight in-ear earbuds",\ "contoured for a comfortable fit",\ "sits at an angle for comfort",\ "better direct audio to your ear",\ "stem is 33% shorter than the second generation AirPods",\ "force sensor to easily control music and calls",\ "Spatial Audio with dynamic head tracking",\ "immersive, three-dimensional listening experience"\ ] "description": "The Apple AirPods Pro are a pair of wireless earbuds that are designed for comfort and convenience. They are lightweight in-ear earbuds and contoured for a comfortable fit, and each airpod sits at an angle for comfort. The AirPods Pro also have a stem that is 33% shorter than the second generation AirPods, which makes them more compact and easier to store. The AirPods Pro also have a force sensor to easily control music and calls, and they have Spatial Audio with dynamic head tracking, which provides an immersive, three-dimensional listening experience.", }} Ensure each feature in the paragraph matches exactly as written in the description, including the and tags. Make sure your output is a JSON and is in the following format. DO NOT OUTPUT ANYTHING ELSE. INCLUDE THE ```json tag in your response. ```json {{ "name": "name of the product", "features": [\ "feature 1",\ "feature 2",\ "feature 3",\ ...\ ], "description": "description of the product" }}``` """ def parse(self, row, response): """Parse the LLM response to extract the product name, features and description.""" default_response = { "name": "Apple AirPods Pro", "features": [\ "lightweight in-ear earbuds",\ "contoured for a comfortable fit",\ "sits at an angle for comfort",\ "better direct audio to your ear",\ "stem is 33% shorter than the second generation AirPods",\ "force sensor to easily control music and calls",\ "Spatial Audio with dynamic head tracking",\ "immersive, three-dimensional listening experience"\ ], "description": "The Apple AirPods Pro are a pair of wireless earbuds that are designed for comfort and convenience. They are lightweight in-ear earbuds and contoured for a comfortable fit, and each airpod sits at an angle for comfort. The AirPods Pro also have a stem that is 33% shorter than the second generation AirPods, which makes them more compact and easier to store. The AirPods Pro also have a force sensor to easily control music and calls, and they have Spatial Audio with dynamic head tracking, which provides an immersive, three-dimensional listening experience.", } if type(response) == type(''): pattern = r"```json(.*?)```" match_found = re.findall(pattern, response, re.DOTALL) if match_found: json_string = match_found[-1].strip() try: response = json.loads(json_string) except: response = default_response else: response = default_response else: response = response.dict() try: row['product'] = response['name'] # note that because the LLM isn't perfect, the features in the response may not be fully accurate # row['original_features'] = response.features # that's why, we parse the features from the output pattern = r"(.*?)" matches = re.findall(pattern, response['description']) if matches: row['features'] = matches else: # backup row['features'] = response['features'] row['description'] = response['description'].replace('','').replace('','') except: return [] return row product_curator = ProductCurator( model_name="ollama/llama3.1:8b", # Ollama model identifier backend_params={ "base_url": "http://localhost:11434", "max_tokens_per_minute": 3000000, "max_requests_per_minute": 10, }, ) Next, let's create some products for the personas with `ProductCurator`! This can take a while. You can use Together.ai or Deepinfra through Curator and [LiteLLM](/bespoke-curator/how-to-guides/using-litellm-for-diverse-providers) to speed up this up. Copy # Generate products for the personas. This will take a while. # You can use , for example, to speed this up. products = product_curator(personas) Here's an example of a generated product: Example generated product[](#example-generated-product) **PERSONA:** A Political Analyst specialized in El Salvador's political landscape. **PRODUCT:** Salvadoria: El Salvador's Political Landscape Analyzer **DESCRIPTION:** Salvadoria is a cutting-edge tool designed specifically for Political Analysts specializing in El Salvador's political landscape. It offers Advanced natural language processing for news articles and social media posts, allowing users to quickly analyze the tone, sentiment, and key themes of online discussions. The customizable keyword alert system enables analysts to track specific topics and hashtags in real-time, ensuring they stay up-to-date on the latest developments. Salvadoria also features an interactive map of El Salvador with election results, demographic data, and key infrastructure information, providing a comprehensive view of the country's political landscape. With access to a comprehensive database of past elections, including voter turnout, candidate performance, and electoral district boundaries, analysts can gain valuable insights into historical trends and patterns. The tool also includes an in-depth analysis of government spending, revenue, and budget allocation by department and agency, allowing users to identify areas of inefficiency or potential corruption. Salvadoria's real-time tracking of public opinion polls, surveys, and focus groups on various political issues keeps analysts informed about shifting public sentiment and policy preferences. Users can customize their dashboard with a range of visualizations and metrics using the customizable dashboard, while also exporting data in CSV format for further analysis or integration with other tools via the ability to export data in CSV format. Regular updates include new data, including special reports on election forecasts, economic indicators, and policy changes, which are integrated seamlessly through the integration with popular spreadsheet software. **FEATURES:** * advanced natural language processing for news articles and social media posts * keyword alert system * interactive map of El Salvador with election results, demographic data, and key infrastructure information * comprehensive database of past elections * in-depth analysis of government spending, revenue, and budget allocation by department and agency * real-time tracking of public opinion polls, surveys, and focus groups * customizable dashboard * ability to export data in CSV format * integration with popular spreadsheet software [](#evaluate-the-baseline-performance-on-eval-data-from-gpt-4o-mini) Evaluate the baseline performance on eval data from gpt-4o-mini ----------------------------------------------------------------------------------------------------------------------------------------- ### [](#set-up-the-evaluationllm-object-using-curator) **Set up the EvaluationLLM object using curator** We can create an `EvaluationLLM` object to evaluate the performance of our models Copy FEATURE_PROMPT = """ You are given a product's name, description and features. You will generate a list of features for the product. An example input is: {{ "name": "Apple AirPods Pro", "description": "The Apple AirPods Pro are a pair of wireless earbuds that are designed for comfort and convenience. They are lightweight in-ear earbuds and contoured for a comfortable fit, and they sit at an angle for easy access to the controls. The AirPods Pro also have a stem that is 33% shorter than the second generation AirPods, which makes them more compact and easier to store. The AirPods Pro also have a force sensor to easily control music and calls, and they have Spatial Audio with dynamic head tracking, which provides an immersive, three-dimensional listening experience.", }} An example output is: {{ "features": [\ "lightweight in-ear earbuds",\ "contoured for a comfortable fit",\ "sit at an angle for easy access to the controls",\ "stem is 33% shorter than the second generation AirPods",\ "force sensor to easily control music and calls",\ "Spatial Audio with dynamic head tracking",\ "immersive, three-dimensional listening experience"\ ] }} Now, generate a list of features for the product. You should output all the features that are mentioned in the description exactly as they are written. You should not miss any features, or add any features that are not mentioned in the description. Your output should be in this format. ```json{{features: ["feature 1","feature 2","feature 3",...]}}``` Product: Name: {product_name} Description: {product_description} Output: """ class EvaluationLLM(curator.LLM): # prompt for evaluation def prompt(self, row): return FEATURE_PROMPT.format(product_name=row['product'], product_description=row['description']) # function to parse the LLM responses given by curator # this function also contains the logic for evaluation of the LLM responses def parse(self, row, response): true_set = set(row['features']) pred_set = set() # Fuzzy matching threshold SIMILARITY_THRESHOLD = 0.85 if type(response) != type(""): predicted_features = response.features else: # string pattern = r"```json(.*?)```" match_found = re.findall(pattern, response, re.DOTALL) if match_found: json_string = match_found[-1].strip() try: out_dict = json.loads(json_string) predicted_features = out_dict.get("features", []) except: print("Incorrect output format..") predicted_features = [] else: predicted_features = [] for pred_feature in predicted_features: # Check if any true feature matches this predicted feature best_match_score = 0 for true_feature in true_set: similarity = fuzz.ratio(pred_feature.lower(), true_feature.lower()) / 100.0 best_match_score = max(best_match_score, similarity) if best_match_score >= SIMILARITY_THRESHOLD: pred_set.add(pred_feature) # Calculate metrics row['true_positives'] = len(pred_set) # Features that matched above threshold row['false_positives'] = len(predicted_features) - row['true_positives'] # Predicted features that didn't match row['false_negatives'] = len(true_set) - row['true_positives'] # True features that weren't matched return row We also set up some utilities to run the evaluation, calculate precision, recall, and F1 metrics, and tabulate them in a nice format. Copy # @title Utilities to calculate precision, recall, f1, run evaluations and tabulate results def calculate_metrics(evaluation): tp = sum(evaluation['true_positives']) fp = sum(evaluation['false_positives']) fn = sum(evaluation['false_negatives']) micro_precision = tp / (tp + fp) micro_recall = tp / (tp + fn) micro_f1 = (2 * micro_precision * micro_recall) / (micro_precision + micro_recall) return {'precision': micro_precision,'recall': micro_recall, 'f1':micro_f1} # Common function to run evaluation on different models def run_evaluation(model_name, dataset): evaluator = EvaluationLLM( model_name=model_name, backend_params={ "max_requests_per_minute":10000, "max_tokens_per_minute":30000000 } ) evaluation = evaluator(dataset) metrics = calculate_metrics(evaluation) return evaluation, metrics # Tabulate eval results from tabulate import tabulate def tabulate_eval_results(model_and_metrics): metrics_names = ['Precision', 'Recall', 'F1'] # Create table data table_data = [] for model, metrics in model_and_metrics.items(): table_data.append([\ model,\ f"{metrics['precision']:.3f}",\ f"{metrics['recall']:.3f}",\ f"{metrics['f1']:.3f}"\ ]) # Print table print(tabulate(table_data, headers=['Model', 'Precision', 'Recall', 'F1'], tablefmt='grid')) ### [](#create-an-eval-set-with-gpt-4o-mini) **Create an eval set with gpt-4o-mini** In order to prevent bias from using the same model to generate train and eval data, we are not going to create a train and test split using the newly created data from Llama-3.1-8B. Instead, we will use all of it for training but generate eval data with a completely different LLM, gpt-4o-mini. Copy import getpass os.environ["OPENAI_API_KEY"] = getpass.getpass("OpenAI API Key:") eval_product_curator = ProductCurator( model_name="gpt-4o-mini" ) train_dataset = products test_dataset = eval_product_curator(personas.take(40)) ### [](#run-the-evaluation-and-get-results) **Run the evaluation and get results** Copy evaluation_results_1b, metrics_1b = run_evaluation('ollama/llama3.2:1b', test_dataset) evaluation_results_8b, metrics_8b = run_evaluation('ollama/llama3.1:8b', test_dataset) tabulate_eval_results(model_and_metrics={"llama-3.2-1b": metrics_1b, "llama-3.1-8b": metrics_8b}) Copy +--------------+-------------+----------+-------+ | Model | Precision | Recall | F1 | +==============+=============+==========+=======+ | llama-3.2-1b | 0.668 | 0.394 | 0.496 | +--------------+-------------+----------+-------+ | llama-3.1-8b | 0.726 | 0.753 | 0.739 | +--------------+-------------+----------+-------+ We can see that Llama-3.1-8B is not able to extract the features as well as 8B (as expected). So, we will finetune it on the training set. [](#finetune-llama3.2-1b-using-unsloth) Finetune Llama3.2-1B using Unsloth ------------------------------------------------------------------------------- ### [](#prepare-data-for-finetuning) Prepare data for finetuning Copy from unsloth import FastLanguageModel from unsloth.chat_templates import get_chat_template max_seq_length = 1024 load_in_4bit = True dtype = None # for auto model, tokenizer = FastLanguageModel.from_pretrained( model_name = "unsloth/Llama-3.2-1B-Instruct-bnb-4bit", max_seq_length = max_seq_length, dtype = dtype, load_in_4bit = load_in_4bit, ) peft_model = FastLanguageModel.get_peft_model( model, r = 16, # Choose any number > 0 ! Suggested 8, 16, 32, 64, 128 target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",\ "gate_proj", "up_proj", "down_proj",], lora_alpha = 16, lora_dropout = 0, # Supports any, but = 0 is optimized bias = "none", # Supports any, but = "none" is optimized # [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes! use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context random_state = 3407, use_rslora = False, # We support rank stabilized LoRA loftq_config = None, # And LoftQ ) # doing this so ollama creates a modelfile tokenizer = get_chat_template( tokenizer, chat_template = "llama-3.1", ) # Prepare a dataset for finetuning def formatting_prompts_func(row): texts = [] features = {'features': row['features']} messages = [\ {"role": "user", "content": FEATURE_PROMPT.format(product_name=row['product'], product_description=row['description'])},\ {"role": "assistant", "content": f"```json{json.dumps(features)}```"},\ ] text = tokenizer.apply_chat_template(messages, tokenize = False, add_generation_prompt = False) return {'text': text} ft_dataset = train_dataset.map(formatting_prompts_func, batched = False,) ft_dataset[0] Copy {'persona': "A Political Analyst specialized in El Salvador's political landscape.", 'product': 'SalvadorAlert', 'features': ["real-time updates on El Salvador's legislative calendar",\ 'in-depth analysis of proposed laws and bills',\ 'topics that matter most to them',\ 'data visualization of voting patterns and trends',\ 'comparative analysis of past and present legislative data',\ 'upcoming hearings and committee meetings',\ "detailed information on El Salvador's presidential and congressional elections",\ 'analysis of public opinion polls and surveys',\ 'news from local and international sources',\ 'monitors social media activity of key politicians and influencers'], 'description': "The SalvadorAlert is a cutting-edge tool for political analysts specializing in El Salvador's political landscape. It provides real-time updates on El Salvador's legislative calendar, including key dates and events. The platform offers in-depth analysis of proposed laws and bills, allowing users to stay on top of the latest developments. Users can also customize their alert system to receive priority notifications on topics that matter most to them. The SalvadorAlert features data visualization of voting patterns and trends, providing a clear picture of the current political climate. Additionally, users can access comparative analysis of past and present legislative data to inform their research. The platform also includes alerts on upcoming hearings and committee meetings, ensuring users are always informed. SalvadorAlert provides detailed information on El Salvador's presidential and congressional elections, including key statistics and analysis. Users can also access analysis of public opinion polls and surveys to gauge public sentiment. Furthermore, the SalvadorAlert aggregates news from local and international sources related to El Salvador's politics, providing a comprehensive view of the news cycle. Finally, the platform monitors social media activity of key politicians and influencers, allowing users to stay ahead of the curve.", 'text': '<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nCutting Knowledge Date: December 2023\nToday Date: 26 July 2024\n\n<|eot_id|><|start_header_id|>user<|end_header_id|>\n\n\n You are given a product\'s name, description and features. You will generate a list of features for the product.\n\n An example input is:\n {\n "name": "Apple AirPods Pro",\n "description": "The Apple AirPods Pro are a pair of wireless earbuds that are designed for comfort and convenience. They are lightweight in-ear earbuds and contoured for a comfortable fit, and they sit at an angle for easy access to the controls. The AirPods Pro also have a stem that is 33% shorter than the second generation AirPods, which makes them more compact and easier to store. The AirPods Pro also have a force sensor to easily control music and calls, and they have Spatial Audio with dynamic head tracking, which provides an immersive, three-dimensional listening experience.",\n }\n\n An example output is:\n {\n "features": [\n "lightweight in-ear earbuds",\n "contoured for a comfortable fit",\n "sit at an angle for easy access to the controls",\n "stem is 33% shorter than the second generation AirPods",\n "force sensor to easily control music and calls",\n "Spatial Audio with dynamic head tracking",\n "immersive, three-dimensional listening experience"\n ]\n\n }\n\n Now, generate a list of features for the product. You should output all the features that are mentioned in the description exactly as they are written. You should not miss any features, or add any features that are not mentioned in the description.\n\n Your output should be in this format.\n ```json{features: ["feature 1","feature 2","feature 3",...]}```\n\n Product:\n Name: SalvadorAlert\n Description: The SalvadorAlert is a cutting-edge tool for political analysts specializing in El Salvador\'s political landscape. It provides real-time updates on El Salvador\'s legislative calendar, including key dates and events. The platform offers in-depth analysis of proposed laws and bills, allowing users to stay on top of the latest developments. Users can also customize their alert system to receive priority notifications on topics that matter most to them. The SalvadorAlert features data visualization of voting patterns and trends, providing a clear picture of the current political climate. Additionally, users can access comparative analysis of past and present legislative data to inform their research. The platform also includes alerts on upcoming hearings and committee meetings, ensuring users are always informed. SalvadorAlert provides detailed information on El Salvador\'s presidential and congressional elections, including key statistics and analysis. Users can also access analysis of public opinion polls and surveys to gauge public sentiment. Furthermore, the SalvadorAlert aggregates news from local and international sources related to El Salvador\'s politics, providing a comprehensive view of the news cycle. Finally, the platform monitors social media activity of key politicians and influencers, allowing users to stay ahead of the curve.\n Output:\n<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n```json{"features": ["real-time updates on El Salvador\'s legislative calendar", "in-depth analysis of proposed laws and bills", "topics that matter most to them", "data visualization of voting patterns and trends", "comparative analysis of past and present legislative data", "upcoming hearings and committee meetings", "detailed information on El Salvador\'s presidential and congressional elections", "analysis of public opinion polls and surveys", "news from local and international sources", "monitors social media activity of key politicians and influencers"]}```<|eot_id|>'} ### [](#run-sft-finetuning-with-unsloth) Run SFT finetuning with Unsloth Copy from trl import SFTTrainer from transformers import TrainingArguments, DataCollatorForSeq2Seq from unsloth import is_bfloat16_supported from unsloth.chat_templates import train_on_responses_only trainer = SFTTrainer( model = peft_model, tokenizer = tokenizer, train_dataset = ft_dataset, dataset_text_field = "text", max_seq_length = max_seq_length, data_collator = DataCollatorForSeq2Seq(tokenizer = tokenizer), dataset_num_proc = 2, packing = False, # Can make training 5x faster for short sequences. args = TrainingArguments( per_device_train_batch_size = 2, gradient_accumulation_steps = 4, warmup_steps = 5, num_train_epochs = 1, # Set this for 1 full training run. learning_rate = 2e-4, fp16 = not is_bfloat16_supported(), bf16 = is_bfloat16_supported(), logging_steps = 1, optim = "adamw_8bit", weight_decay = 0.01, lr_scheduler_type = "linear", seed = 3407, output_dir = "outputs", report_to = "none", # Use this for WandB etc ), ) trainer = train_on_responses_only( trainer, instruction_part = "<|start_header_id|>user<|end_header_id|>\n\n", response_part = "<|start_header_id|>assistant<|end_header_id|>\n\n", ) trainer_stats = trainer.train() ### [](#save-the-finetuned-model-and-serving-it-using-ollama) Save the finetuned model and serving it using Ollama Copy # save quantized model peft_model.save_pretrained_gguf("llama_finetune", tokenizer) Copy # Unsloth automatically creates a Modelfile! cat /content/llama_finetune/Modelfile # Create an ollama model using the saved Modelfile ollama create llama_finetune -f ./llama_finetune/Modelfile # Verify that the finetuned model exists ollama ls [](#final-results) **Final results** ----------------------------------------- Running the evaluation on the new finetuned model, we found that F1 for Llama-3.2-1B model jumped from 0.496 to 0.688, a significant improvement! Copy # evaluate the finetuned model finetuned_llama_evaluation_results, finetuned_llama_metrics = run_evaluation('ollama/llama_finetune', test_dataset) print(finetuned_llama_evaluation_results[0]) print(finetuned_llama_metrics) tabulate_eval_results(model_and_metrics={"llama-3.2-1b": metrics_1b, "llama-3.1-8b": metrics_8b, "finetuned-llama-3.2-1b": finetuned_llama_metrics}) Copy +------------------------+-------------+----------+-------+ | Model | Precision | Recall | F1 | +========================+=============+==========+=======+ | llama-3.2-1b | 0.668 | 0.394 | 0.496 | +------------------------+-------------+----------+-------+ | llama-3.1-8b | 0.726 | 0.753 | 0.739 | +------------------------+-------------+----------+-------+ | finetuned-llama-3.2-1b | 0.796 | 0.606 | 0.688 | +------------------------+-------------+----------+-------+ Just for fun, let try running our new finetuned model on a new example: Copy product_name = "Ryobi Circular Saw" product_description = """ Expand your RYOBI 18V ONE+ System with the RYOBI 18V ONE+ Cordless Circular Saw. Make over 215 fast, clean cuts per charge on the ONE+ Cordless 5 1/2 in. Circular Saw with 4,700 RPM and the included 18T Carbide Tipped Blade. This saw is ideal for cross cuts in 2-by material with 1-11/16 in. maximum depth of cut. Bevel up to 50 degrees to complete a wide variety of cuts and with 1-3/16 in. depth of cut at 45 Degrees of bevel. Purchase the accessory vacuum dust adaptor (sold separately) to connect this saw to your wet/dry vac for quick and easy clean up. Best of all, it is part of the RYOBI ONE+ System of over 300 Cordless Products that all work on the same battery platform. This 18V ONE+ Cordless 5-1/2 in. Circular Saw is backed by the RYOBI 3-Year Manufacturer's Warranty. Battery and charger sold separately.""" class Extractor(curator.LLM): def prompt(self, input): return FEATURE_PROMPT.format(product_name=input['product'], product_description=input['description']) extractor = Extractor(model_name='gpt-4o') result = extractor(Dataset.from_list([{'product': product_name, 'description': product_description}]))['response'][0] print(result) def get_parsed_response(response): if type(response) == type(''): pattern = r"```json(.*?)```" match_found = re.findall(pattern, response, re.DOTALL) if match_found: json_string = match_found[-1].strip() try: response = json.loads(json_string) except: raise ValueError("Failed to parse") else: raise ValueError("Failed to parse") else: response = response.dict() return response response = get_parsed_response(result) display_product( product_name, product_description, response['features'], image_url='' ) This is not bad for a quick start. In some cases, you will see that the LLM doesn't output exact text (which happens for even GPT-4o)! Great next steps: 1. Increase the number of training examples. 2. Systematically evaluate the error types. 3. Run this in local machine and run `curator-viewer` to visualize your data. 4. Create complex strategies for data curation (involving multiple curator.LLM stages). 5. Star https://github.com/bespokelabsai/curator/! Copy ![](https://colab.research.google.com/assets/colab-badge.svg) ![](https://docs.bespokelabs.ai/~gitbook/image?url=https%3A%2F%2F1831689742-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FINVkBnpIXpC4135NE6Ex%252Fuploads%252FNMnEpDYOxTtKlQRQrPY3%252FScreenshot%25202025-01-16%2520at%252011.29.24%25E2%2580%25AFAM.png%3Falt%3Dmedia%26token%3D652d1e45-0843-4d2f-b869-c9bc343cc34c&width=768&dpr=4&quality=100&sign=5b5618ea&sv=2) ![](https://docs.bespokelabs.ai/~gitbook/image?url=https%3A%2F%2F1831689742-files.gitbook.io%2F%7E%2Ffiles%2Fv0%2Fb%2Fgitbook-x-prod.appspot.com%2Fo%2Fspaces%252FINVkBnpIXpC4135NE6Ex%252Fuploads%252FCZZSwFdxT0zW4wH8j4OR%252FScreenshot%25202025-01-16%2520at%252012.39.40%25E2%2580%25AFPM.png%3Falt%3Dmedia%26token%3D82535d95-c6b8-4b1d-8eb0-916fd56135b6&width=768&dpr=4&quality=100&sign=295cabe1&sv=2) ---