# Table of Contents - [Python SDK - Refuel.ai](#python-sdk-refuel-ai) - [Overview - Refuel.ai](#overview-refuel-ai) - [Overview - Refuel.ai](#overview-refuel-ai) - [JavaScript SDK - Refuel.ai](#javascript-sdk-refuel-ai) - [Overview - Refuel.ai](#overview-refuel-ai) - [Overview - Refuel.ai](#overview-refuel-ai) - [Amazon S3 - Refuel.ai](#amazon-s3-refuel-ai) - [Overview - Refuel.ai](#overview-refuel-ai) - [Google Cloud Storage - Refuel.ai](#google-cloud-storage-refuel-ai) - [Overview - Refuel.ai](#overview-refuel-ai) - [Snowflake - Refuel.ai](#snowflake-refuel-ai) - [Introduction - Refuel.ai](#introduction-refuel-ai) - [Getting started - Refuel.ai](#getting-started-refuel-ai) - [List of Transformations - Refuel.ai](#list-of-transformations-refuel-ai) - [Adding datasets - Refuel.ai](#adding-datasets-refuel-ai) - [Documents and Images - Refuel.ai](#documents-and-images-refuel-ai) - [Usage - Refuel.ai](#usage-refuel-ai) - [Overview - Refuel.ai](#overview-refuel-ai) - [Creating applications - Refuel.ai](#creating-applications-refuel-ai) - [Import from Cloud Storage - Refuel.ai](#import-from-cloud-storage-refuel-ai) - [Import from Data Warehouse - Refuel.ai](#import-from-data-warehouse-refuel-ai) - [Evaluating LLM Outputs - Refuel.ai](#evaluating-llm-outputs-refuel-ai) - [Task Chaining - Refuel.ai](#task-chaining-refuel-ai) - [Structured Outputs - Refuel.ai](#structured-outputs-refuel-ai) - [Using finetuned models - Refuel.ai](#using-finetuned-models-refuel-ai) - [Taxonomy Management - Refuel.ai](#taxonomy-management-refuel-ai) - [Using applications - Refuel.ai](#using-applications-refuel-ai) - [Improving accuracy with feedback - Refuel.ai](#improving-accuracy-with-feedback-refuel-ai) - [Team and User Settings - Refuel.ai](#team-and-user-settings-refuel-ai) - [Monitoring - Refuel.ai](#monitoring-refuel-ai) - [Batch Processing - Refuel.ai](#batch-processing-refuel-ai) - [Scheduled Task Run - Refuel.ai](#scheduled-task-run-refuel-ai) - [Overview - Refuel.ai](#overview-refuel-ai) - [Finetuning models - Refuel.ai](#finetuning-models-refuel-ai) - [Tracking finetuning runs - Refuel.ai](#tracking-finetuning-runs-refuel-ai) --- # Python SDK - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation SDK Reference Python SDK Refuel provides a Python library for programmatically interacting with the platform. This guide describes how to install and use the Refuel SDK. `refuel` is available as a library on [PyPI](https://pypi.org/project/refuel/) . The code is open source and available on [GitHub](https://github.com/refuel-ai/refuel-python) . [​](#getting-started) Getting Started ---------------------------------------- ### [​](#installation) Installation Install the SDK using a package installer such as pip: pip install refuel Installing and using the SDK requires Python 3.6+. ### [​](#initializing-a-client-session) Initializing a client session Set the environment variable REFUEL\_API\_KEY. The SDK will read it during initialization, and use this value when sending requests: import refuel # Assuming you've set `REFUEL_API_KEY` in your env, # init() will pick it up automatically refuel_client = refuel.init() Alternatively, you can supply the API key as a parameter during initializtion as shared below. In the cloud application, there is a top-level dropdown to select the project you’re working on currently. And this selection powers all the pages downstream (datasets, labeling tasks etc). The SDK allows you to do this by setting the project during initialization as well: import refuel options = { "api_key": "", "project": "", } refuel_client = refuel.init(**options) Here’s the complete list of initialization options currently supported: | Option | Is Required | Default Value | Comments | | --- | --- | --- | --- | | `api_key` | Yes | None | Used to authenticate all requests to the API server | | `project` | Yes | None | The name of the project you plan to use for the current session | | `timeout` | No | 60 | Timeout in seconds | | `max_retries` | No | 3 | Max number of retries for failed requests | | `max_workers` | No | Num CPUs (os.cpu\_count()) | Max number of concurrent requests to the API server | [​](#projects) Projects -------------------------- These functions let you create a new project in your team’s Refuel account, or get information about projects already defined. ### [​](#get-projects) Get Projects The get\_projects API will return a list of all projects that belong to your team: projects = refuel_client.get_projects() You can also retrieve information about a specific project, either by name or by UUID project = refuel_client.get_project(project='') ### [​](#create-project) Create Project refuel_client.create_project( project='', description='', ) [​](#datasets) Datasets -------------------------- These functions let you upload/export a full dataset, or fetch rows and LLM labels within a dataset. ### [​](#get-all-current-datasets) Get all current datasets The get\_datasets function will return a list of all datasets in the project: import refuel options = { "api_key": "", "project": "", } refuel_client = refuel.init(**options) datasets = refuel_client.get_datasets() ### [​](#upload-dataset) Upload Dataset This function lets you upload a local CSV file as a new dataset to Refuel. import refuel options = { "api_key": "", "project": "", } refuel_client = refuel.init(**options) dataset = refuel_client.upload_dataset( file_path='', dataset_name='', source='file|uri', wait_for_completion='True|False' ) Some details about the function parameters: | Option | Is Required | Default Value | Comments | | --- | --- | --- | --- | | `file_path` | Yes | \- | Path to the data you wish to upload | | `dataset_name` | Yes | \- | Unique name of the dataset being uploaded | | `source` | Yes | file | Place where the file resides | | `wait_for_completion` | No | False | Whether to poll for the completion of the dataset ingestion | Note: When uploading dataset from `uri` source, the `file_path` should be publicly accessible (eg. S3 presigned url) for Refuel to process it. ### [​](#download-dataset) Download Dataset This function create a snapshot of your dataset and generate presigned URL for secure download. response = refuel_client.download_dataset( dataset='', email_address='', ) This is an asynchronous workflow, and when the dataset is available for download, Refuel will send an email to the email address. Make sure the email address belongs to a valid user from your team. Depending on the size of the dataset, this export step can take a few minutes. Once the download link is created and emailed, it will be valid for 24 hours. ### [​](#adding-items-rows-to-an-uploaded-dataset) Adding items (rows) to an uploaded dataset This function lets you append new rows to an existing dataset. Keep in mind that the dataset schema is decided during initial upload and is not updated here. import refuel options = { "api_key": "", "project": "", } refuel_client = refuel.init(**options) new_items_to_add = [\ {"column1": "value1", "column2": "value2", ...},\ {"column1": "value3", "column2": "value4", ...},\ ...\ ] refuel_client.add_items(dataset='', items=new_items_to_add) ### [​](#querying-items-rows-in-a-dataset) Querying items (rows) in a dataset In addition to downloading the entire dataset, you can also fetch a list of items (rows) from the dataset as follows: import refuel options = { "api_key": "", "project": "", } refuel_client = refuel.init(**options) items = refuel_client.get_items( dataset='', max_items=20, offset=0 ) This function will return a pandas dataframe. Some details about the function parameters: | Option | Is Required | Default Value | Comments | | --- | --- | --- | --- | | `dataset` | Yes | \- | Name of the dataset you want to query and retrieve items (rows) from | | `max_items` | No | 20 | Max number of rows you want to fetch | | `offset` | No | 0 | If this is set to a positive number, say N, then the first N rows will be skipped and the API will return “max\_items” number of rows after skipping the first N rows. | #### [​](#querying-items-along-with-labels-from-a-labeling-task) Querying items, along with labels from a labeling task get\_items() also allows you to provide an optional parameter - a labeling task. When provided, the function will also include the task results (LLM labels, confidence and manually confirmed labels, if any) for the returned items. items = refuel_client.get_items( dataset='', task='', max_items=100 ) #### [​](#applying-sort-ordering-when-querying-items) Applying sort ordering when querying items By default, the API will use Refuel’s sort order (by decreasing order of diversity). You can use the `order_by` param to sort by any other columns in the dataset or by the label or confidence score from a labeling task. 1. Sort by dataset column items = refuel_client.get_items( dataset='', max_items=100, order_by=[{'field': '', 'direction': ''}], ) 2. Sort by label or confidence score from a labeling task. Note that this requires a task name and a subtask name to be specified. `field` can be either ‘label’ or ‘confidence’. items = refuel_client.get_items( dataset='', task='', max_items=100, order_by=[{'field': 'confidence', 'direction': '', 'subtask': ''}], ) You may have multiple dicts in the `order_by` list if you would like to sort by multiple columns (used in the case of ties). Some details about the keys for each dict in the `order_by` list: | Key | Is Required | Default Value | Description | Comments | | --- | --- | --- | --- | --- | | `field` | Yes | | The name of the column in the dataset to sort by | In addition to the columns in the dataset, the field can also be ‘label’ or ‘confidence’, if the task and subtask names are specified. | | `direction` | No | `ASC` | The direction that you would like to sort the specified column by | Should be `ASC` or `DESC` | | `subtask` | No | null | The name of the subtask for which you would like to sort by label or confidence | This should only be provided if the field is ‘label’ or ‘confidence’ and requires a task name to be specified in the function params. | #### [​](#applying-filters-when-querying-items) Applying filters when querying items In addition to sorting options, you can also define filters to only fetch items in the dataset that match a specific criteria. The SDK supports three types of filters: 1. Metadata Filters: Filter based on the value of a specific column, for e.g. a filter such as “column = value” is defined as a Python dictionary with three keys: * field: This is the column on which you want to apply the filter * operator: This is the comparison operator * value: This is the value to compare to Here’s an example: items_filter = { 'field': 'transaction_category', 'operator': '=', 'value': 'Food' } items = refuel_client.get_dataset_items( dataset='', max_items=100, filters = [items_filter] ) 2. LLM output value/confidence filter: Filter items based on the LLM output value or confidence score from a specific task configured in Refuel. Here’s a concrete example: Let’s say you configured a classification task called `Sentiment Analysis` in your Refuel account, which has two subtasks (output fields): (i) `predicted_sentiment` - the predicted sentiment (ii) `explanation` - a one sentence explanation of why the LLM output the predicted sentiment as Positive or Negative for the item. Here are a few filters we can define for this task: * “predicted sentiment is Positive”: { 'field': 'llm_label', 'subtask': 'predicted_sentiment' 'operator': '=', 'value': 'Positive' } * “predicted sentiment confidence >= 80%”: { 'field': 'confidence', 'subtask': 'predicted_sentiment' 'operator': '>=', 'value': '0.8' } * “predicted sentiment does not agree with the ground truth label (available in a column called `ground_truth_sentiment`)”: { 'field': 'llm_label', 'subtask': 'predicted_sentiment' 'operator': '<>', 'value': 'ground_truth_sentiment' } Here’s the complete list of filter operators that are currently supported: | Operator | Description | | :------------ | :---------------------------------------------------------------------- | --- | | `>` | Greater than | | `>=` | Greater than or equal to | | `=` | Equals | | `<>` | Not equal to | | `<` | Less than | | `<=` | Less than or equal to | | `IS NULL` | True if field is undefined | | `IS NOT NULL` | True if field is defined | | `LIKE` | String matching: True if value is in field | | `ILIKE` | String matching (case insensitive): True if value is in field | | `NOT LIKE` | String does not match: True if value is not in field | | `NOT ILIKE` | String does not match (case insensitive): True if value is not in field | | [​](#tasks) Tasks -------------------- These functions let you retrieve information about labeling tasks defined within a project, and start and cancel a task run. ### [​](#define-a-new-task) Define a new Task You can create a new task programmatically within a given project using the `create_task` function: import refuel options = { "api_key": "", "project": "", } refuel_client = refuel.init(**options) refuel_client.create_task( task='', dataset='', context = 'You are an expert at analyzing sentiment of an online review about a business...', fields = [\ {\ 'name': '...',\ 'type': '...',\ 'guidelines': '...',\ 'labels': [...],\ 'input_columns': [...],\ 'fallback_value': '...'\ },\ ...\ ], model = 'GPT-4 Turbo' ) Some details about the various parameters you see in the function signature above: | Parameter | Is Required | Default Value | Comments | | --- | --- | --- | --- | | `task` | Yes | None | Name of the new task you’re creating | | `dataset` | Yes | None | Dataset (in Refuel) for which you are defining this task | | `context` | Yes | None | Context is a high level description of the problem domain and the dataset that the LLM will be working with. It typically starts with something like ‘You are and expert at …’ | | `fields` | Yes | None | This is a list of dictionaries. Each entry in this list defines an output field generated in the task. See below for details about the schema of each field | | `model` | No | team default | LLM that will be used for this task. If not specified, we will use the default LLM set for your team, e.g. GPT-4 Turbo | Next, let’s take a look at the schema of each entry in the `fields` list above: | Parameter | Is Required | Default Value | Comments | | --- | --- | --- | --- | | `name` | Yes | None | Name of the output field, e.g. `llm_predicted_sentiment` | | `type` | Yes | None | Type of output field. This is one of: \[`classification`, `multilabel_classification`, `attribute_extraction`, `webpage_transform`, `web_search`\] | | `guidelines` | Yes | None | Output guidelines for the LLM for this field. Note that if the field type is a `web_search` type, the guidelines will be simply the query template | | `labels` | Yes (for classification field types) | None | list of valid labels, this field is only required for classification type tasks | | `input_columns` | Yes | None | Columns from the dataset to use as input when passing a “row” in the dataset to the LLM. | | `ground_truth_column` | No | None | A column in the dataset that contains ground truth value for this field, if one exists. Note this is an optional parameter. | | `fallback_value` | No | None | A fallback/default value that the LLM should generate for this field if a row cannot be processed successfully | Finally, here is the list of LLMs currently supported (use the model name as the parameter value): | Provider | Name | | --- | --- | | OpenAI | GPT-4 Turbo | | OpenAI | GPT-4o | | OpenAI | GPT-4o mini | | OpenAI | GPT-4 | | OpenAI | GPT-3.5 Turbo | | Anthropic | Claude 3.5 (Sonnet) | | Anthropic | Claude 3 (Opus) | | Anthropic | Claude 3 (Haiku) | | Google | Gemini 1.5 (Pro) | | Mistral | Mistral Small | | Mistral | Mistral Large | | Refuel | Refuel LLM-2 | | Refuel | Refuel LLM-2-small | ### [​](#get-tasks) Get Tasks You can retrieve a list of all tasks within a given project as follows tasks = refuel_client.get_tasks() ### [​](#start-a-labeling-task-run) Start a Labeling Task Run You can begin running a labeling task on a dataset with the following: response = refuel_client.start_task_run( task='', dataset='', num_items=100 ) This will kick off a bulk labeling run for the specified task and dataset, and label 100 items in the dataset. If `num_items` parameter is not specified, it will label the entire dataset. ### [​](#cancel-an-ongoing-labeling-task-run) Cancel an ongoing labeling task run You can also cancel an ongoing labelling task with the same function as follows. response = refuel_client.cancel_task_run( task='', dataset='' ) ### [​](#get-task-run-status-progress) Get Task run status/progress To check on the status of an ongoing labeling task run, you can use the following function task_run = refuel_client.get_task_run( task='', dataset='' ) The `task_run` object has the following schema: { 'id': '...', 'task_id': '...', 'task_name': '...', 'dataset_id': '...', 'dataset_name': '...', 'model_name': '...', # This is the LLM used for the run 'status': 'active', # Status of the task run 'metrics': [ # Metrics related to the task run\ {'name': 'num_labeled', 'value': ... },\ {'name': 'num_remaining', 'value': ...},\ {'name': 'time_elapsed_seconds', 'value': ...},\ {'name': 'time_remaining_seconds', 'value': ...},\ ] } `status` enum shows the current task run status. It can be one of the following values: * `not_started`: this is the starting state before the batch run has been kicked off * `active`: a batch task run is ongoing * `paused`: the batch run was paused before the full dataset was labeled. * `failed`: the batch run failed due to a platform error. This should ideally never happen * `completed`: the batch run was completed successfully `metrics` is a list containing all metrics for the current task run. Currently the platform supports the following metrics: * `num_labeled`: number of rows from the dataset that have been labeled * `num_remaining` number of rows from the dataset that are remaining * `time_elapsed_seconds`: time (in seconds) since the task run started. This is only populated when the task run is active (since this metric is not valid when there is no active run). * `time_remaining_seconds` estimated time (in seconds) remaining for the task run to complete. This is only populated when the task run is active (since this metric is not valid when there is no active run). [​](#applications) Applications ---------------------------------- Refuel allows you to deploy a labeling task as an application. Applications allow you to label data synchronously on demand, primarily for online workloads. ### [​](#deploy-labeling-application) Deploy labeling application To deploy an existing task as a labeling application, you can use the following function import refuel options = { "api_key": "", "project": "", } refuel_client = refuel.init(**options) response = refuel_client.deploy_task(task='') ### [​](#get-all-labeling-application) Get all labeling application To get all labeling applications that are currently deployed, use the following function applications = refuel_client.get_applications() ### [​](#label-using-a-deployed-application) Label using a deployed application You can use the deployed application for online predictions as follows: inputs = [\ {'col_1': 'value_1', 'col_2': 'value_2' ...},\ {'col_1': 'value_1', 'col_2': 'value_2' ...},\ ] response = refuel_client.label(application='', inputs=inputs, explain=False) Each element in `inputs` is a dictionary, with keys as names of the input columns defined in the task. For example, let’s consider an application for sentiment classification called `my_sentiment_classifier`, with two input fields - `source` and `text`. You can use it as follows: inputs = [\ {'source': 'yelp.com', 'text': 'I really liked the pizza at this restaurant.'}\ ] response = refuel_client.label(application='my_sentiment_classifier', inputs=inputs) `response` has the following schema: * `refuel_output[i]` contains the output for `inputs[i]` * `refuel_fields` is a list whose length is equal to the number of fields defined in the application. For example, let’s say `my_sentiment_classifier` has just one field, `sentiment`. In this case the output will be: { 'application_id': '...', 'application_name': 'my_sentiment_classifier', 'refuel_output': [\ {'refuel_uuid': '...',\ 'refuel_api_timestamp': '...',\ 'refuel_fields':\ {\ 'sentiment': {\ 'label': ['positive'],\ 'confidence': 0.9758\ }\ }\ }] } You can also set the optional `explain` parameter to `True` to get an explanation for why the provided label was returned. The explanation will be returned in the `explanation` field in the response, along with the `label` and `confidence`: 'sentiment': { 'label': ['positive'], 'confidence': 0.9758, 'explanation': 'The model predicted a positive sentiment because the text contains positive words like "liked" and "good".' } If you would only like to get an explanation for certain fields, you can optionally provide a list of field names in the `explain_fields` parameter for which you want an explanation returned. If `explain_fields` is provided, explanations will be returned regardless of whether `explain` is set to `True` or `False`. Here’s an example of how to get explanations for the `sentiment` field in the `my_sentiment_classifier` application: response = refuel_client.label(application='my_sentiment_classifier', inputs=inputs, explain=True, explain_fields=['sentiment']) You can also set the optional `telemetry` parameter to `True` to get additional info such as the model, provider, and number of tokens used (prompt, output, and total) in the request. The telemetry data will be returned in the `usage` field in the response. response = refuel_client.label(application='my_sentiment_classifier', inputs=inputs, telemetry=True) ### [​](#async-labeling) Async Labeling If you do not want to wait for the labeling to be completed, you can instead use the method `alabel` with the exact same parameters as with `label`. This will submit the inputs for labeling with Refuel and returns the refuel\_uuid to get the labeled item back. response = refuel_client.alabel(application='my_sentiment_classifier', inputs=inputs) The output will be: { 'application_id': '...', 'application_name': 'my_sentiment_classifier', 'refuel_output': [\ {'refuel_uuid': '...',\ 'refuel_api_timestamp': '...',\ 'uri': '...'\ }] } You can eith use the `refuel_uuid` from the output to get the labeled item back using `get_labeled_item` method. response = refuel_client.get_labeled_item(application='my_sentiment_classifier', refuel_uuid='dcbb0266-aeaa-4c0e-87b3-4341b5f3b7bc') You can also directly call the returned uri to get the labeled item back. Async Labeling is also useful when the input data is large or the LLM generates large amount of data which can lead to timeouts on Refuel’s Application. ### [​](#share-feedback-for-application-outputs) Share feedback for application outputs The SDK allows users to log feedback for online predictions. When logging predictions, it is important to identify the input request for which you’re logging feedback using `refuel_uuid` from the response above: label = {'sentiment': ['negative']} refuel_client.feedback(application='my_sentiment_classifier', refuel_uuid='...', label=label) Any row with logged feedback will appear with the verified check mark (”✓”) in the cloud application. [​](#finetuning) Finetuning ------------------------------ Refuel allows you to finetune a model based on all the human reviewed data and optionally data labeled by an LLM. Finetuned models allow you to reduce labeling cost and latency while achieving similar, and in some cases better, performance than LLMs like GPT-4. ### [​](#starting-a-finetuning-run) Starting a finetuning run To start a model finetuning run, you can use the following function: import refuel options = { "api_key": "", "project": "", } refuel_client = refuel.init(**options) hyperparameters = {"num_epochs": 1} datasets = ["dataset_0_id", "dataset_1_id"] response = refuel_client.finetune_model(task_id='', model='', hyperparameters=hyperparameters, datasets=datasets) Supported Base Models: \[‘refuel-llm-v2-large’, ‘refuel-llm-v2-small’\] Supported hyperparameters: \[‘lora\_r’, ‘lora\_alpha’, ‘lora\_dropout’, ‘weight\_decay’, ‘learning\_rate’, ‘cosine\_min\_lr\_ratio’\] ### [​](#get-all-finetuned-models) Get all finetuned models You can retrieve a list of all models within a given project as follows models = refuel_client.get_finetuned_models(task_id='') ### [​](#cancel-a-finetuning-run) Cancel a finetuning run You can also cancel an ongoing finetuning run as follows. response = refuel_client.cancel_finetuning(model_id='') [JavaScript SDK](/sdk/javascript) On this page * [Getting Started](#getting-started) * [Installation](#installation) * [Initializing a client session](#initializing-a-client-session) * [Projects](#projects) * [Get Projects](#get-projects) * [Create Project](#create-project) * [Datasets](#datasets) * [Get all current datasets](#get-all-current-datasets) * [Upload Dataset](#upload-dataset) * [Download Dataset](#download-dataset) * [Adding items (rows) to an uploaded dataset](#adding-items-rows-to-an-uploaded-dataset) * [Querying items (rows) in a dataset](#querying-items-rows-in-a-dataset) * [Querying items, along with labels from a labeling task](#querying-items-along-with-labels-from-a-labeling-task) * [Applying sort ordering when querying items](#applying-sort-ordering-when-querying-items) * [Applying filters when querying items](#applying-filters-when-querying-items) * [Tasks](#tasks) * [Define a new Task](#define-a-new-task) * [Get Tasks](#get-tasks) * [Start a Labeling Task Run](#start-a-labeling-task-run) * [Cancel an ongoing labeling task run](#cancel-an-ongoing-labeling-task-run) * [Get Task run status/progress](#get-task-run-status-progress) * [Applications](#applications) * [Deploy labeling application](#deploy-labeling-application) * [Get all labeling application](#get-all-labeling-application) * [Label using a deployed application](#label-using-a-deployed-application) * [Async Labeling](#async-labeling) * [Share feedback for application outputs](#share-feedback-for-application-outputs) * [Finetuning](#finetuning) * [Starting a finetuning run](#starting-a-finetuning-run) * [Get all finetuned models](#get-all-finetuned-models) * [Cancel a finetuning run](#cancel-a-finetuning-run) --- # Overview - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Integrations Overview This is an overview of common integrations supported in Refuel. [LLM Providers\ -------------\ \ LLM Providers such as OpenAI and Anthropic](/integrations/llm_providers) [Cloud Storage\ -------------\ \ Cloud storage locations such as AWS S3, GCS, Azure Blob Storage](/integrations/cloud_storage) [Data Warehouses and Data Lakes\ ------------------------------\ \ Data Warehouses and data lakes such as Snowflake, BigQuery, Databricks](/integrations/data_warehouses) [API Sources\ -----------\ \ Custom APIs and internal knowledge bases](/integrations/api_sources) [Overview](/integrations/llm_providers) --- # Overview - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Introduction Overview Refuel is a platform to clean, structure and transform your data at scale and superhuman quality by leveraging state-of-the-art large language models (LLMs). Transforming your data with Refuel is a 4 step process: 1. **Connect your data**: Connect your data to the platform, via integrations into your cloud storage, or by uploading it directly to Refuel. 2. **Define your usecase:** Describe how you want to transform, enrich or label your data, or pick from a library of pre-built transformations. 3. **Iterate with feedback**: Improve LLM quality and reliability with a simple thumbs-up or down feedback loop, and model fine-tuning. 4. **Deploy and Monitor:** Deploy your application to process data at scale, with realtime or batch processing workloads using the model that best fits your latency, cost and security needs. [Getting started](/quickstart) --- # JavaScript SDK - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation SDK Reference JavaScript SDK [Python SDK](/sdk/python) --- # Overview - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation LLM Providers Overview Refuel supports a wide range of LLM providers (other than Refuel). You can use any of the following providers in your Refuel workflows, either with your own API keys or with our managed API keys. [​](#llm-providers-supported) LLM Providers Supported -------------------------------------------------------- * OpenAI * Anthropic * Gemini * Mistral * Azure OpenAI * AWS Bedrock [​](#adding-your-own-api-keys) Adding your own API keys ---------------------------------------------------------- In case you want to use your own API keys for any of the supported LLM providers, you can do so by adding them to the Integrations tab in the Settings section, or following [this link](https://app.refuel.ai/settings/integrations) . When you’re on that page, you can click on “Connect” for the LLM provider you want to connect to, and then add your API key as shown below. [Overview](/integrations/introduction) [Overview](/integrations/cloud_storage) On this page * [LLM Providers Supported](#llm-providers-supported) * [Adding your own API keys](#adding-your-own-api-keys) --- # Overview - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Cloud Storage Overview Refuel supports direct integrations with Amazon S3, Google Cloud Storage and Azure Blob Storage. [​](#supported-data-warehouses) Supported Data Warehouses ------------------------------------------------------------ | Data Store | Availability | | --- | --- | | [Amazon S3](/integrations/s3) | ✅ Available now | | [Google Cloud Storage](/integrations/gcs) | ✅ Available now | | Azure Blob Storage | ✅ Available now | If you have suggestions for other cloud storage options that you would like to see supported, please let us know by dropping us a note at [support@refuel.ai](mailto:support@refuel.ai) . [Overview](/integrations/llm_providers) [Amazon S3](/integrations/s3) On this page * [Supported Data Warehouses](#supported-data-warehouses) --- # Amazon S3 - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Cloud Storage Amazon S3 [Amazon S3](https://aws.amazon.com/s3/) is the cloud storage solution offered by Amazon Web Services, and you can use set up an integration in Refuel to read from S3. This is commonly used for reading documents and images, but also for uploading and ingesting CSVs and other files formats. [​](#setup-guide) Setup guide -------------------------------- ### [​](#step-0-get-refuel-aws-account-id-and-external-id) Step 0: Get Refuel AWS Account ID and External ID Contact us at [support@refuel.ai](mailto:support@refuel.ai) to get Refuel’s AWS Account ID and external ID that you can use in the later steps. ### [​](#step-1-create-iam-policy-for-s3-access) Step 1: Create IAM Policy for S3 Access Navigate to the IAM Console in AWS, click on Policies in the left navbar, and then hit the “Create Policy” button. Click on the JSON tab, copy the policy below (substitute your bucket name, add multiple buckets, restrict to specific keys in a bucket) and then hit Next. Click Next again, give the policy a name, such as “RefuelS3ReadAccessPolicy” and then hit “Create policy”. { "Version": "2012-10-17", "Statement": [\ {\ "Effect": "Allow",\ "Action": [\ "s3:GetObject",\ "s3:ListBucket"\ ],\ "Resource": [\ "arn:aws:s3:::YourBucketName/*",\ "arn:aws:s3:::YourBucketName"\ ]\ }\ ] } ### [​](#step-2-create-iam-role-for-s3-access) Step 2: Create IAM Role for S3 Access Follow the steps below to create an IAM role that Refuel can assume to access your S3 bucket. * In the IAM Console, click on Roles in the left navbar and click on “Create role”. * In “Trusted entity type”, click on “AWS account” and then click on “Another AWS account”. * Paste in the Refuel AWS account ID from Step 0. * Click “Require external ID” and paste in the external ID from step 0, and click Next. * Search for, and add the policy you just created: “RefuelS3ReadAcccessPolicy” and hit Next. * Give this role a name, let’s say “RefuelS3ReadAccessRole” and click “Create role”. ### [​](#step-3-send-refuel-the-arn-of-the-newly-created-role) Step 3: Send Refuel the ARN of the newly created role Share the role ARN with Refuel via email or Slack. ### [​](#step-4-start-sending-refuel-your-data) Step 4: Start sending Refuel your data! That’s it! When you send Refuel any pieces of content that sit in S3, all you need to send is the path in S3, and Refuel will be able to securely access it. [Overview](/integrations/cloud_storage) [Google Cloud Storage](/integrations/gcs) On this page * [Setup guide](#setup-guide) * [Step 0: Get Refuel AWS Account ID and External ID](#step-0-get-refuel-aws-account-id-and-external-id) * [Step 1: Create IAM Policy for S3 Access](#step-1-create-iam-policy-for-s3-access) * [Step 2: Create IAM Role for S3 Access](#step-2-create-iam-role-for-s3-access) * [Step 3: Send Refuel the ARN of the newly created role](#step-3-send-refuel-the-arn-of-the-newly-created-role) * [Step 4: Start sending Refuel your data!](#step-4-start-sending-refuel-your-data) --- # Overview - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Data Warehouses and Data Lakes Overview Refuel supports direct integrations with Snowflake, and we are actively adding support for other data warehouses and data lakes (expected timelines are mentioned in the table below). [​](#supported-data-warehouses) Supported Data Warehouses ------------------------------------------------------------ | Data Store | Availability | | --- | --- | | [Snowflake](/integrations/snowflake) | ✅ Available now | | Databricks | ⏳ Early 2025 | | BigQuery | ⏳ Early 2025 | If you have suggestions for other data warehouses or data lakes that you would like to see supported, please let us know by dropping us a note at [support@refuel.ai](mailto:support@refuel.ai) . [Google Cloud Storage](/integrations/gcs) [Snowflake](/integrations/snowflake) On this page * [Supported Data Warehouses](#supported-data-warehouses) --- # Google Cloud Storage - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Cloud Storage Google Cloud Storage [Google Cloud Storage (GCS)](https://cloud.google.com/storage) is the cloud storage solution offered by Google Cloud, and you can set up an integration in Refuel to read from GCS. This is commonly used for reading documents and images, but also for uploading and ingesting CSVs and other files formats. [​](#setup-guide) Setup guide -------------------------------- ### [​](#step-0-refuel-gcp-service-account) Step 0: Refuel GCP Service Account Contact us at [support@refuel.ai](mailto:support@refuel.ai) to get access to Refuel’s GCP service account email. ### [​](#step-1-configure-iam-permissions-for-your-gcp-bucket) Step 1: Configure IAM Permissions for your GCP Bucket You can watch the following Loom to understand how this works. Steps: * Navigate to your Google Cloud Storage bucket in your GCP account. * Click on the Permissions tab and then hit the “Grant Access” button * Add Refuel’s GCP service account email as a principal and add Cloud Storage: Storage Object Viewer as the Role * Hit “Save” to submit your selection. ### [​](#step-2-start-sending-refuel-your-data) Step 2: Start sending Refuel your data! That’s it! When you send Refuel any pieces of content that sit in GCS, all you need to send is the path in GCS (gs://bucket\_name/path/to/object), and Refuel will be able to securely access it. [Amazon S3](/integrations/s3) [Overview](/integrations/data_warehouses) On this page * [Setup guide](#setup-guide) * [Step 0: Refuel GCP Service Account](#step-0-refuel-gcp-service-account) * [Step 1: Configure IAM Permissions for your GCP Bucket](#step-1-configure-iam-permissions-for-your-gcp-bucket) * [Step 2: Start sending Refuel your data!](#step-2-start-sending-refuel-your-data) --- # Overview - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation API Sources Overview Refuel already supports API-based enrichments for Google Web Search and Google Maps Search natively. We’re actively working on adding more native integrations so we can fetch data and relevant business context from your enterprise API sources. Drop us a line at [support@refuel.ai](mailto:support@refuel.ai) if you’d like to see a specific API source supported. [Snowflake](/integrations/snowflake) --- # Snowflake - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Data Warehouses and Data Lakes Snowflake [Snowflake](https://www.snowflake.com/) is a leading cloud data warehouse, and you can use Refuel to read and write data back into it. [​](#setup-guide) Setup guide -------------------------------- For Refuel to read and write data back into Snowflake, the following steps need to be followed. ### [​](#step-1-create-a-role-and-user) Step 1: Create a role and user Run the following script as a user role that has the permissions to create a user, role, and warehouse (such as the `sysadmin` or `securityadmin` roles). Create roles, users and permissions within Snowflake begin; -- create variables for user / password / role / warehouse / database (needs to be uppercase for objects) set role_name = 'REFUEL_ROLE'; set user_name = 'REFUEL_USER'; set user_password = ''; set warehouse_name = 'REFUEL_WAREHOUSE'; set database_name = 'REFUEL_DATABASE'; -- change role to securityadmin for user / role steps use role securityadmin; -- create role for refuel create role if not exists identifier($role_name); grant role identifier($role_name) to role SYSADMIN; -- create a user for refuel create user if not exists identifier($user_name) password = $user_password default_role = $role_name default_warehouse = $warehouse_name; grant role identifier($role_name) to user identifier($user_name); -- set binary_input_format to BASE64 alter user identifier($user_name) set binary_input_format = 'BASE64'; -- change role to sysadmin for warehouse / database steps use role sysadmin; -- create a warehouse for refuel create warehouse if not exists identifier($warehouse_name) warehouse_size = xsmall warehouse_type = standard auto_suspend = 60 auto_resume = true initially_suspended = true; -- create database for refuel create database if not exists identifier($database_name); -- grant refuel role access to warehouse grant usage on warehouse identifier($warehouse_name) to role identifier($role_name); -- grant refuel access to database grant create schema, monitor, usage on database identifier($database_name) to role identifier($role_name); -- grant refuel read access to all tables and views in database grant usage on all schemas in database identifier($database_name) to role identifier($role_name); grant select on all tables in database identifier($database_name) to role identifier($role_name); grant select on all views in database identifier($database_name) to role identifier($role_name); -- if you want to limit access to specific schemas, you can use the following commands -- grant USAGE on SCHEMA identifier($database_name).schema_name to role identifier($role_name); -- grant SELECT on ALL TABLES in SCHEMA identifier($database_name).schema_name to role identifier($role_name); -- grant SELECT on ALL VIEWS in SCHEMA identifier($database_name).schema_name to role identifier($role_name); -- change role to ACCOUNTADMIN for STORAGE INTEGRATION support (only needed for Snowflake on GCP) use role ACCOUNTADMIN; grant CREATE INTEGRATION on account to role identifier($role_name); use role sysadmin; commit; ### [​](#optional-step-2-enabling-key-pair-authentication) \[Optional\] Step 2: Enabling Key Pair Authentication If you want a more secure way for Refuel to communicate with Snowflake, you can enable Key Pair Authentication. For that, follow the steps [here](https://docs.snowflake.com/en/user-guide/key-pair-auth#generate-the-private-key) . The only change to the list of steps we recommend is to use the AES256 algorithm instead of DES3 (faster and more secure). The complete command to generate the private key is Generate a private key openssl genrsa 2048 | openssl pkcs8 -topk8 -v2 AES256 -inform PEM -out rsa_key.p8 ### [​](#step-3-setup-snowflake-integration-in-refuel-cloud) Step 3: Setup Snowflake integration in Refuel Cloud In order to setup the Snowflake integration, navigate to [Integrations](https://app.refuel.ai/settings/integrations) within Refuel Cloud Settings, select Snowflake, and fill in the following fields: * Snowflake Host (account\_name.snowflakecomputing.com) * Warehouse * Database * Role * Username * Password (If not using Key Pair authentication) * Private Key (If using Key Pair authentication) * Passphrase (If using Key Pair authentication) [Overview](/integrations/data_warehouses) [Overview](/integrations/api_sources) On this page * [Setup guide](#setup-guide) * [Step 1: Create a role and user](#step-1-create-a-role-and-user) * [\[Optional\] Step 2: Enabling Key Pair Authentication](#optional-step-2-enabling-key-pair-authentication) * [Step 3: Setup Snowflake integration in Refuel Cloud](#step-3-setup-snowflake-integration-in-refuel-cloud) --- # Introduction - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Data Transformations Catalog Introduction This is a collection of pre-built data transformations to parse, structure or categorize your data within minutes, without having to build a custom workflow from scratch. These are particularly useful for out-of-the-box use cases — where the end use case is standard, and you can immediately start consuming the transformed data. [​](#access) Access ---------------------- Catalog of data transformations is currently available to all Refuel Cloud users. You can get started by signing up [here](https://www.refuel.ai/get-started) . [​](#browsing-the-catalog) Browsing the Catalog -------------------------------------------------- Browse the list of data transformations currently available in the catalog by visiting [https://app.refuel.ai/catalog](https://app.refuel.ai/catalog/) . [​](#testing-a-function-from-the-catalog) Testing a function from the catalog -------------------------------------------------------------------------------- Each transformation published in the catalog has a dedicated playground, where you can test the transformation with your own data: You can either test it out right in the browser playground, or call the transformation programmatically via a REST API, or by using one of our [SDKs](https://docs.refuel.ai/sdk/) . For example, here’s how you can call the `Resume parser`: curl -X 'POST' \ 'https://cloud-api.refuel.ai/applications/resume-parsing/label' \ -H 'accept: application/json' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' \ -d '[{\ "resume_link": "https://path/to/resume.pdf"\ }]' [​](#processing-large-data-volumes) Processing large data volumes -------------------------------------------------------------------- Transformations available via the catalog currently have the following rate limits: * 300 requests/min * Upto 100 concurrent requests Requests will be throttled and the platform will return an `HTTP 429` status code if these limits are exceeded. If you’d like higher rate limits, or want to leverage one of our data warehouse integrations for batch processing, you can sign up for Refuel Cloud ([here](https://www.refuel.ai/get-started) ). [Usage](/guides/team/usage) [List of Transformations](/catalog/catalog_list) On this page * [Access](#access) * [Browsing the Catalog](#browsing-the-catalog) * [Testing a function from the catalog](#testing-a-function-from-the-catalog) * [Processing large data volumes](#processing-large-data-volumes) --- # Getting started - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Introduction Getting started In this tutorial, we’ll walk through the steps to get set up with Refuel, and building your first data transformation application. 1 Get access to Refuel Make sure you are able to log in to [https://app.refuel.ai](https://app.refuel.ai) . If you don’t have access yet, sign up [here](https://www.refuel.ai/get-started) . 2 Create a new project Once you log into Refuel Cloud, the first thing you’ll do is create a project. A project is a logical grouping for your datasets, models, and applications. 3 Upload your first dataset In order to build your first application, you’ll need to upload a dataset, a collection of rows of structured/semi-structured data that you want to transform with LLMs. The easiest way to get started is to use our [sample dataset](https://gist.github.com/nihit/74abe1d926acba2d45c68631f216b096) . It is a .csv file consisting of a collection of user reviews of businesses on Yelp.com. In the UI, click on “Add Dataset”, and upload the CSV file: For more information on how to upload datasets, see [here](/guides/datasets) . 4 Define your first task Once you’ve uploaded your dataset, you’ll need to define a task. A task is a set of LLM guidelines for the transformation you want to perform on your dataset. For example, for the Yelp dataset, you might want to create a task that categorizes the sentiment of the review as positive, negative, or neutral. Tasks can be fairly complex and consist of a chained execution of multiple steps. For more information on how to define and iterate on task definitions, see [here](/guides/tasks) . 5 Improve quality with feedback Once you’ve defined a task and saved it, Refuel will start generating LLM outputs for your dataset. You can review outputs, and provide feedback to iteratively improve quality. 6 Deploy your task as an application Once you’ve iterated on task guidelines and improved output quality with feedback, you can deploy it as an application. Think of an application as a versioned snapshot of a Task that is deployed behind an API to serve realtime traffic. Any deployed application can be used to transform new data in realtime. For example, you can use the sentiment analysis application to transform new Yelp reviews as they come in: The `Playground` tab is a great way to test out your application, and retrieve the relevant code snippets to integrate it programmatically into your stack. For more information on how to deploy and manage applications, see [here](/guides/applications) . That’s it! You’ve now built your first application with Refuel 🎉 [Overview](/overview) [Adding datasets](/guides/datasets/datasets) --- # List of Transformations - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Data Transformations Catalog List of Transformations This document lays out the input and output schema for the full list of Transformations provided in the Catalog. Note that for every single output, Refuel will also produce a confidnence score. [​](#staffing-recruiting-and-hrtech) Staffing, Recruiting and HRTech ----------------------------------------------------------------------- ### [​](#resume-parsing) Resume Parsing **Input:** * `resume_link` (str): Either a publicly readable URL, or a path to S3 or GCS that can be read by Refuel through our integration. **Output:** * `candidate_name` (str): Name of the candidate. * `contact_info` (json): A JSON object containing any physical addresses, email addresses, phone numbers or web addresses (LinkedIn, Github, personal websites, etc) for the candidate. * `education` (list(json)): A list of JSON objects, where each JSON contains the school, major, degree, start year, end year and other information about a specific educational degree for the candidate. * `work_history` (list(json)): A list of JSON objects, where each JSON contains the job title, company, start month, start year, end month, end year and description about a specific job held by the candidate. * `skills` (str): List of skills demonstrated by the candidate based on evidence in their resume. ### [​](#job-description-parsing) Job Description Parsing **Input:** * `text` (str): Raw text from a job description. **Output:** * `company` (str): The company or organization offering this job. * `title` (str): The job title for this job. * `location` (str): Location where the job is based. * `pay` (json): A JSON object containing information about the pay period (hourly, weekly, monthly, etc), minimum and maximum amounts, any bonuses, etc. * `skills` (str): List of skills required by the job description. ### [​](#skills-extraction-and-mapping) Skills Extraction and Mapping **Input:** * `link` (str): Either a publicly readable URL, or a path to S3 or GCS for a resume, job description or other document from which skills needs to be extracted. **Output:** * `skills` (str): List of skills demonstrated by the candidate based on evidence in their resume, and mapped against a taxonomy. ### [​](#job-title-normalization) Job Title Normalization **Input:** * `title` (str): Job title to be normalized. **Output:** * `normalized_title` (str): Job title as a string, with typos corrected, short forms expanded (ex. Sr to Senior), unnecessary modifiers or adjectives removed. ### [​](#job-title-seniority-classification) Job Title Seniority Classification **Input:** * `title` (str): Job title. **Output:** * `seniority` (str): The job title will be categorized against the following taxonomy: 1. Owner 2. Founder 3. C Suite 4. Partner 5. VP 6. Head 7. Director 8. Manager 9. Senior 10. Entry 11. Intern [​](#sales-data-and-salestech) Sales data and SalesTech ---------------------------------------------------------- ### [​](#headquarters-or-physical-address-for-a-business) Headquarters or Physical Address for a Business **Input:** * `business_name` (str): Name of the business. **Output:** * `address` (str): The physical address or headquarters for the business name supplied. If no physical address is found, “Not Found” is returned. ### [​](#revenue-estimate) Revenue Estimate **Input:** * `business_name` (str): Name of the business. * `website` (str): Business website (domain). * `address` (str): Complete address of the Business HQ. **Output:** * `revenue` (str): The latest estimated revenue of the business. If a revenue number cannot be extracted, “Not Found” is returned. ### [​](#lead-scoring) Lead Scoring **Input:** * `business_description` (str): Description of the business that is qualifying leads. * `icp_description` (str): Description of the ideal customer profile for the business. * `customer_title` (str): Job title of the lead at the lead’s company. * `customer_company` (str): Company of the lead. * `customer_name` (str): Name of the lead. **Output:** * `lead_score` (str): A score between 0 and 100, indicating the likelihood of the lead being a good fit for the business. * `lead_score_rationale` (str): A rationale for the lead score, explaining the reasoning behind the score. ### [​](#get-phone-numbers-for-business) Get Phone Numbers for business **Input:** * `business_name` (str): Name of the business to extract the phone number for. * `website` (str): Website of the business to extract the phone number for. * `address` (str): Address of the business to extract the phone number for. **Output:** * `phone_number` (str): Phone number of the business. ### [​](#domain-name-extraction) Domain Name Extraction **Input:** * `business_name` (str): The name of the business for which the domain name needs to be extracted. * `address` (str): The address of the business for which the domain name needs to be extracted. **Output:** * `domain_name` (str): The domain name/website of the business. ### [​](#icp-fit-classification) ICP Fit Classification **Input:** * `business_name` (str): The name of the business looking for potential customers. * `business_description` (str): The description of the business looking for potential customers. A text description of the business and its offerings. * `icp_description` (str): The description of the ideal customer profile (ICP) of the business looking for potential customers. This describes the ICP in detail which will be matched to information extracted about the potential customer. * `customer_company` (str): The name of the company that is being evaluated for potential fit with the business. * `customer_website` (str): The website of the company that is being evaluated for potential fit with the business. **Output:** * `icp_fit` (str): The ICP fit of the business. This returns how good of a fit the customer is for the business based on the ICP description. This will be one of the following values - `High`, `Medium`, `Low`. ### [​](#sic-classification) SIC Classification **Input:** * `business_name` (str): The name of the business for which SIC code needs to be found. * `website` (str): The website of of the business. * `address` (str): The address of the business. **Output:** * `sic_code` (str): The relevant SIC codes of the business. The full list of possible codes can be found [here](https://www.sec.gov/search-filings/standard-industrial-classification-sic-code-list) . ### [​](#naics-industry-classification) NAICS Industry Classification **Input:** * `business_name` (str): The name of the business. * `website` (str): The website of of the business. * `address` (str): The address of the business. **Output:** * `naics_sector` (str): The NAICS sector of the business. The sector is the first two digits of the 6-digit NAICS code. * `naics_industry` (str): One or more 6-digit NAICS codes under which the business is categorized. If the business has multiple industries, the codes will be returned as a semicolon-separated list. The full NAICS taxonomy can be found [here](https://www.census.gov/naics/?68772-00) . ### [​](#mcc-industry-classification) MCC Industry Classification **Input:** * `business_name` (str): The name of the business. * `website` (str): The website of of the business. * `address` (str): The address of the business. **Output:** * `mcc_categories` (str): One or more Merchant Category Codes (MCC) under which the business is categorized. If the business has multiple MCC codes, the codes will be returned as a semicolon-separated list. The full list of possible codes can be found [here](https://www.citibank.com/tts/solutions/commercial-cards/assets/docs/govt/Merchant-Category-Codes.pdf) . ### [​](#number-of-employees) Number of Employees **Input:** * `business name` (str): The name of the business. * `website` (str): The website of of the business. * `address` (str): The address of the business. **Output:** * `Number of employees` (str): The number of employees who work at the business. ### [​](#address-cleaning-and-normalization) Address Cleaning and Normalization **Input:** * `address` (str): The unformatted address to be cleaned and normalized. **Output:** * `clean addresses` (str): The clean address in a standard format. [Introduction](/catalog/introduction) On this page * [Staffing, Recruiting and HRTech](#staffing-recruiting-and-hrtech) * [Resume Parsing](#resume-parsing) * [Job Description Parsing](#job-description-parsing) * [Skills Extraction and Mapping](#skills-extraction-and-mapping) * [Job Title Normalization](#job-title-normalization) * [Job Title Seniority Classification](#job-title-seniority-classification) * [Sales data and SalesTech](#sales-data-and-salestech) * [Headquarters or Physical Address for a Business](#headquarters-or-physical-address-for-a-business) * [Revenue Estimate](#revenue-estimate) * [Lead Scoring](#lead-scoring) * [Get Phone Numbers for business](#get-phone-numbers-for-business) * [Domain Name Extraction](#domain-name-extraction) * [ICP Fit Classification](#icp-fit-classification) * [SIC Classification](#sic-classification) * [NAICS Industry Classification](#naics-industry-classification) * [MCC Industry Classification](#mcc-industry-classification) * [Number of Employees](#number-of-employees) * [Address Cleaning and Normalization](#address-cleaning-and-normalization) --- # Adding datasets - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Datasets Adding datasets In Refuel, a dataset is a collection of rows of structured/semi-structured data that you want to transform with LLMs. [​](#upload-a-csv-file-as-a-new-dataset) Upload a CSV file as a new dataset ------------------------------------------------------------------------------ You can upload a CSV file as a dataset to Refuel. If you have data with PII fields, you have the option to Remove the PII data before the dataset get ingested by Refuel. [​](#append-to-an-existing-dataset) Append to an existing dataset -------------------------------------------------------------------- If you want to add additonal data to an existing dataset in Refuel, you can do so by clicking on **Add to Dataset** button on the dataset page and uploading another CSV with new data. [​](#preparing-data-for-upload) Preparing data for upload ------------------------------------------------------------ Before you upload the file, make sure it is a valid CSV file with the following requirements: * The file should have a header row to identify the column names. * The file should be encoded in UTF-8. * Maximum file size is 2GB. If you have a larger file, you can ingest it into Refuel via a [cloud storage integration](/guides/datasets/cloud_storage) . [Getting started](/quickstart) [Documents and Images](/guides/datasets/documents) On this page * [Upload a CSV file as a new dataset](#upload-a-csv-file-as-a-new-dataset) * [Append to an existing dataset](#append-to-an-existing-dataset) * [Preparing data for upload](#preparing-data-for-upload) --- # Documents and Images - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Datasets Documents and Images With Refuel, you can upload a CSV containing images or PDF documents and using Text Recognition, we will be able to parse the text from the Image/PDF. The Images or PDFs can be present as Web URIs or Cloud URI (S3, GCS). If using Cloud URIs, please setup the [Cloud Integration](/integrations/cloud_storage) so we are able to access the URIs. ### [​](#dataset-with-pdf-documents) Dataset with PDF documents ### [​](#dataset-with-images) Dataset with Images Note: In order for the LLM to use the text from Images or PDF documents, please make sure to add Text Recognition Enrichment in your Labeling Task. [Adding datasets](/guides/datasets/datasets) [Import from Cloud Storage](/guides/datasets/cloud_storage) On this page * [Dataset with PDF documents](#dataset-with-pdf-documents) * [Dataset with Images](#dataset-with-images) --- # Usage - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Settings Usage You can track request volume and token usage for your organization in “Usage” tab under Settings: This is where you’ll be able to see aggregate usage, or broken down by LLM or application. [Team and User Settings](/guides/team/users) [Introduction](/catalog/introduction) --- # Overview - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Tasks Overview A task in Refuel, is a sequence of steps to define how you want to transform your data with LLMs. Tasks can range from simple to complex and may involve multiple steps executed in a chained sequence to achieve the desired transformation. ### [​](#defining-your-first-task) Defining your first task Conceptually a task is a sequence of steps to be executed in a specific order to transform your data with LLMs. Every task in Refuel has the following components: * **Task name**: A name for your task. * **Context**: This is an overview of the dataset you are working with, and the problem you are trying to solve. These will be used to guide the LLM to take on a specific role/persona. * **Field(s)**: One or more output fields that will be generated by the LLM. Conceptually, a field is a new column you’re adding to your dataset - the result of the data transformation. * **Models**: Provider and model to use for the task. * **Advanced settings**: Additional settings to configure task behavior. Here’s a quick video overview of how to define a new task: ### [​](#output-fields) Output fields Refuel supports a variety of output field types, depending on the task you are trying to solve: * **Single category classification**: A single category classification task is a task where the LLM will classify an input into one of a set of predefined categories. * **Multi category classification**: A multi category classification task is a task where the LLM will classify an input into one or more of a set of predefined categories. * **Extraction**: LLM will extract attributes from the input, and return them as a structured JSON schema. * **Generation**: LLM will generate a free-format text output based on the input. ### [​](#enrichments-fields) Enrichments fields Enrichments allow you to configure external data sources from which relevant context can be fetched and supplied to the LLM when producing an output. This is critical for tasks where it is safer to rely on an external knowledge base (vs the LLM’s internal knowledge) to ensure accuracy/freshness of outputs. Refuel currently supports enrichments from the following sources: * Web search * Maps search * Website scraping * Extracting text from images * Custom enrichments (beta) - see custom enrichments for more details. You can add enrichments to your task by clicking on “Add field” and selecting the enrichment source you want to use. When defining an enrichment field, you will typically select the input columns that will be used to fetch the enrichment data for each row, and optionally any other configurations required. ### [​](#supported-models) Supported Models Refuel supports LLMs from a variety of providers: | Provider | Name | | --- | --- | | OpenAI | GPT-4 Turbo | | OpenAI | GPT-4o | | OpenAI | GPT-4o mini | | OpenAI | GPT-4 | | OpenAI | GPT-3.5 Turbo | | Anthropic | Claude 3.5 (Sonnet) | | Anthropic | Claude 3 (Opus) | | Anthropic | Claude 3 (Haiku) | | Google | Gemini 1.5 (Pro) | | Google | Gemini 1.5 (Flash) | | Google | Gemini 2.0 (Flash) | | Mistral | Mistral Small | | Mistral | Mistral Large | | Refuel | Refuel LLM-2 | | Refuel | Refuel LLM-2-small | You can select the model you want to use for your task from the dropdown in the task editor. In addition to the base models, any LLMs that you have [finetuned](https://docs.refuel.ai/guides/models/overview) for a specific task will also be available in the same dropdown: ### [​](#advanced-settings) Advanced Settings In addition to the model and field settings, you can also configure task behavior related to how LLM responses are cached, how is provided feedback used for few shot prompting and more by navigating to the `Advanced` section in the task editor. * **Few shot prompting**: is a technique where you provide examples of the desired output to the LLM to help it understand the task better. You can update the number of examples you want to provide to the LLM in the `Few shot learning` section. Refuel will dynamically select the most relevant examples for each row to be processed. * **Caching LLM responses**: By default, Refuel will cache the LLM responses. This means that if the LLM is called with the exact same prompt (model configuration + task guidelines + input data values to be processed) we will serve the completion from the cache instead of calling the LLM. This is useful in practice to improve latenct and reduce costs. You can disable this by setting the `Cache LLM responses` toggle to off. * **Beam search** : Beam search is a technique where the LLM generates multiple completions in parallel and then selects the most likely completion. This behavior is enabled by default, but you can disable it by setting the `Beam search` toggle to off. * **Confidence**: By default, Refuel will calculate the confidence score for each row based on the LLM response. You can disable this by setting the `Compute Confidence Scores` toggle to off. [Import from Data Warehouse](/guides/datasets/data_warehouse) [Task Chaining](/guides/tasks/chaining) On this page * [Defining your first task](#defining-your-first-task) * [Output fields](#output-fields) * [Enrichments fields](#enrichments-fields) * [Supported Models](#supported-models) * [Advanced Settings](#advanced-settings) --- # Creating applications - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Applications Creating applications You can deploy any Task as an Application. Think of an application as a versioned snapshot of a Task that is deployed behind an API to serve realtime traffic. Applications are best suited to support real-time, user-facing workloads - bring the power of LLMs to your product, along with the data engine to detect issues and improve performance. [Using finetuned models](/guides/models/using_finetuned_models) [Using applications](/guides/applications/using_applications) --- # Import from Cloud Storage - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Datasets Import from Cloud Storage 🧑‍🍳 More updates soon! [Documents and Images](/guides/datasets/documents) [Import from Data Warehouse](/guides/datasets/data_warehouse) --- # Import from Data Warehouse - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Datasets Import from Data Warehouse Refuel makes it a breeze to import your data residing in Data Warehouses such as Snowflake, Databricks (more coming soon). Refuel is committed to keeping your data safe by following industry-standard practices for securing physical deployments, setting access policies, and leveraging the security features of leading Cloud providers. Generally, the import is a 2-step process: * Setting up the [Data Warehouse Integration](/integrations/data_warehouses) * Telling Refuel the location of the data [​](#snowflake) Snowflake ---------------------------- **Prerequisite:** Setup the [snowflake integration](/integrations/snowflake) 1 Choose snowflake from the list of Data Sources 2 Specify the table name Add the name of the table that you want to import in the format **schema\_name.table\_name**. The user/role specified during snowflake integration should have read access to the table 3 Setup the dataset Select the set of primary keys that can uniquely identify a row in the provided table. You also need to select a timestamp column that will act as a checkpoint so we only read new data as it arrives in your Snowflake table. If your table does not have a primary key or timestamp column, please drop us a message. If you want to automatically run a task on new data as it lands in your Data Warehouse, you can setup a [Scheduled Task Run](/guides/tasks/scheduled_run) on your dataset. [Import from Cloud Storage](/guides/datasets/cloud_storage) [Overview](/guides/tasks/tasks) On this page * [Snowflake](#snowflake) --- # Evaluating LLM Outputs - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Tasks Evaluating LLM Outputs 🧑‍🍳 More updates soon! [Improving accuracy with feedback](/guides/tasks/feedback) [Structured Outputs](/guides/tasks/structured_outputs) --- # Task Chaining - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Tasks Task Chaining 🧑‍🍳 More updates soon! [Overview](/guides/tasks/tasks) [Improving accuracy with feedback](/guides/tasks/feedback) --- # Structured Outputs - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Tasks Structured Outputs 🧑‍🍳 More updates soon! [Evaluating LLM Outputs](/guides/tasks/evals) [Taxonomy Management](/guides/tasks/taxonomy_management) --- # Using finetuned models - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Finetuning Using finetuned models Once you have finetuned a model, you can select it from the model dropdown list, and save the task to start using it. When you save the task, the results for the selected dataset will be refreshed with the newly selected finetuned model and you can examine the results like for any other model. [Tracking finetuning runs](/guides/models/models) [Creating applications](/guides/applications/deployment) --- # Taxonomy Management - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Tasks Taxonomy Management 🧑‍🍳 More updates soon! [Structured Outputs](/guides/tasks/structured_outputs) [Batch Processing](/guides/tasks/task_runs) --- # Using applications - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Applications Using applications [​](#playground) Playground ------------------------------ The playground is a tool for testing, debugging and playing around with your deployed application. The playground can be used to test out an application by sending different inputs to the application and see how the output of the application changes. Edit the input in the Browser section to see how the corresponding output in the Response section changes. This is the exact response that any client will receive when they make a request to this application. [​](#examples-of-using-applications) Examples of using applications ---------------------------------------------------------------------- The playground also provides code snippets for using this application in different clients (currently we support python, javascript and curl). Here is an example snippet for the syntax of using an application. This will be adapted to your application in the playground. [​](#synchronous-vs-asynchronous-application-calls) Synchronous vs Asynchronous application calls ---------------------------------------------------------------------------------------------------- LLM applications defined in Refuel can be used in 2 ways, depending on the usecase and latency constraints: 1. Synchronous This is meant for processing individual or small batches of inputs (10s of items) in real time. It is best suited for usecases that need an immediate response/low latency. This involves making a call to the application synchronously i.e sending a request to the application and blocking until the response is received. The default code snippets in the playground are for synchronous calls. **Limitations**: There is some possibility of timeouts if the LLM is temporarily unavailable (e.g. OpenAI outage) or traffic is throttled. 2. Asynchronous This is meant for cases where latency is less critical or we expect each request to take a long time to complete. This involves making a call to the application asynchronously i.e sending a request to the application and polling for the response separately. **Advantages compared to synchronous calls**: * No timeout errors - even if the LLM takes a long time to process, or if we have to make multiple LLM calls (e.g. in a multi-step chain) * More robust to LLM throttling/transient availability - Refuel will take care of retries **Limitations**: * Marginally higher latency per request * Separate API calls to submit a request, and then to retrieve the results * Currently not supported through the Python or Javascript SDKs **How to use**: The async workflow has two steps: 1. Submit a request The request will be exactly the same as the curl request for a synchronous call, but with an extra query parameter to tell Refuel to process this request asynchronously: `is_async=True`. The endpoint will return an HTTP status code 202, and the response schema, if the is\_async flag is set to True, is the following: { "application_id": "", "application_name": "", "refuel_output": [\ {\ "refuel_uuid": "",\ "refuel_api_timestamp": "",\ "uri": ""\ }\ ] } The `refuel_uuid` is the unique identifier for this resource, and should be used for retrieving results in Step 2. 2. Retrieve results This is supported by the GET /applications//items/ endpoint. An example curl request - Curl curl -X 'GET' \ 'https://cloud-api.refuel.ai/applications//items/' \ -H 'accept: application/json' \ -H 'Authorization: Bearer ' \ -H 'Content-Type: application/json' This endpoint will return the following status codes: * Status 202 - if the request is still being processed * Status 200 - if the request is completed processing (along with the LLM output response) * Status 404 - if no such `refuel_uuid` can be found The response schema of the asynchronous call is the same as that of the synchronous response seen in the playground. [​](#optional-parameters) Optional parameters ------------------------------------------------ ### [​](#explanations) Explanations In addition to the label output, you can also request explanations for why a model chose a particular label. ### [​](#model-override) Model override The model used for the application call can be changed from the default model specified in the application definition. This is useful to try out different models for the same application, without having to re-deploy the application. [​](#rate-limits) Rate limits -------------------------------- Currently, we impose the following rate limits on all applications: * 600 requests/min * Upto 100 concurrent requests Requests will be throttled if these limits are exceeded. We can increase these limits for specific applications if needed. Contact us if you need to increase these limits. [Creating applications](/guides/applications/deployment) [Monitoring](/guides/applications/logs) On this page * [Playground](#playground) * [Examples of using applications](#examples-of-using-applications) * [Synchronous vs Asynchronous application calls](#synchronous-vs-asynchronous-application-calls) * [Optional parameters](#optional-parameters) * [Explanations](#explanations) * [Model override](#model-override) * [Rate limits](#rate-limits) --- # Improving accuracy with feedback - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Tasks Improving accuracy with feedback 🧑‍🍳 More updates soon! [Task Chaining](/guides/tasks/chaining) [Evaluating LLM Outputs](/guides/tasks/evals) --- # Team and User Settings - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Settings Team and User Settings [​](#inviting-users) Inviting Users​ --------------------------------------- You can invite new users into your organization by [clicking here](https://app.refuel.ai/settings/users) or navigating to the Users tab within Settings. When you click on the “Invite Users” button, and provide the email address of the users you want to invite (as a comma separated list), they will receive an email with a link to join your organization. [​](#user-profile) User Profile ---------------------------------- You can update your name or profile picture by [clicking here](https://app.refuel.ai/settings/profile) or navigating to the Profile tab within Settings: [Monitoring](/guides/applications/logs) [Usage](/guides/team/usage) On this page * [Inviting Users​](#inviting-users) * [User Profile](#user-profile) --- # Monitoring - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Applications Monitoring [​](#metrics) Metrics ------------------------ This pane contains metrics to understand the health and performance of your application. There are 3 metrics that are displayed: * **Volume**: The number of requests made to your application in the specified time period. It also shows the number of 4xx and 5xx errors that occured. * **Latency**: This shows the 95th percentile latency (p95) of requests to your application. * **Tokens**: This shows the number of input and output tokens used by your application in the specified time period. You can control two things about the metrics: * **Date range**: This is the time period over which the metrics are displayed. This can range from the last 24 hours to the last 90 days. * **Period**: This is the time period over which the metrics are aggregated. For eg, if the period is selected as 1 hour, the latency would correspond to the 95th percentile latency of requests in the last hour. [​](#request-logs) Request logs ---------------------------------- This pane contains all the requests made to your application. There is usually a small delay between when a request is made and when it is reflected in the logs. You can see the input and output of the request, as well as the latency and tokens used by the request. Additionally, you can provide feedback to the model if you think the output is incorrect. This will help us improve the model over time. Click on the request and provide feedback similar to how you would provide feedback on the task page. The data from this feedback can be used for few-shot, finetuning and evaluation of your model. [Using applications](/guides/applications/using_applications) [Team and User Settings](/guides/team/users) On this page * [Metrics](#metrics) * [Request logs](#request-logs) --- # Batch Processing - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Tasks Batch Processing 🧑‍🍳 More updates soon! [Taxonomy Management](/guides/tasks/taxonomy_management) [Scheduled Task Run](/guides/tasks/scheduled_run) --- # Scheduled Task Run - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Tasks Scheduled Task Run For datasets imported from Data Warehouses, you can also setup a Scheduled Task Run that will automatically run a Labeling Task on new data as it arrives in your Data Warehouse. Refuel periodically reads new data, labels it using the guidelines and writes the data back into your Data Warehouse. This allows a super efficient way to clean and enrich data coming from different applications in your Warehouse. 1 Import Dataset You can import a dataset from any of our supported [Data Warehouses](/guides/datasets/data_warehouse) 2 Setup a Labeling Task Once the dataset is imported, you can setup a task consisting of LLM guidelines for the transformation you want to perform on your dataset. For example, for the Yelp dataset, you might want to create a task that categorizes review sentiment, extracts topics and opinions from the review. 3 Setup Scheduled Run On the datasets page, you can click on the Schedule button from the dropdown menu and select the tasks you want to run on a schedule. By default, we run the scheduled tasks once every day. If you wish to run it more frequently, drop us a message at [support@refuel.ai](mailto:support@refuel.ai) . [Batch Processing](/guides/tasks/task_runs) [Overview](/guides/models/overview) --- # Overview - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Finetuning Overview ### [​](#what-is-llm-finetuning) What is LLM Finetuning? Finetuning is the process of taking a pre-trained language model (LLM) and further training it on a specific dataset to adapt it to a particular task or domain. This involves adjusting the model’s weights based on the new data, allowing it to learn patterns and nuances specific to the desired domain. ### [​](#benefits-of-finetuning) Benefits of Finetuning Finetuning offers several advantages when compared to using the base LLM directly: * **Improved Performance**: By training the model on task-specific data, finetuning can significantly enhance the model’s performance on that task, leading to more accurate and relevant outputs. * **Customization**: Finetuning allows for the customization of the model to better align with specific requirements, such as industry jargon, user preferences, output style and format etc. * **Efficiency**: Once finetuned, the model can generate more precise results without needing extensive context in the input. This means lower token usage, making it faster and cheaper. ### [​](#when-to-finetune) When to Finetune Consider finetuning a base LLM in the following scenarios: * **Specialized Tasks**: When the task requires domain-specific knowledge or terminology that the base model may not be familiar with. * **High Accuracy Requirements**: When the application demands high accuracy and precision, such as in medical diagnosis, legal document analysis, or financial forecasting. * **Consistent Output**: When consistent and reliable output is crucial, and in-context learning may not provide the necessary stability. ### [​](#prompt-engineering-vs-few-shot-learning-vs-finetuning) Prompt engineering vs Few-shot learning vs Finetuning When leveraging LLMs for specific tasks, you have multiple strategies to improve performance: * **Prompt Engineering**: The practice of crafting precise and optimized input prompts to achieve desired model outputs without modifying the underlying model. * **Few-shot Learning**: This involves providing the LLM with a few examples (input and expectedoutput pairs) along the guidelines in the prompt. One way to think about few-shot (in-context learning) is that it is a form of “reasoning by analogy”. * **Finetuning**: This involves adjusting the LLM’s parameters by training it on a domain-specific or task-specific dataset. Each strategy has its own trade-offs and is best leveraged at different stages of development ([image credit](https://x.com/karpathy/status/1655994367033884672) ): [Scheduled Task Run](/guides/tasks/scheduled_run) [Finetuning models](/guides/models/finetuning) On this page * [What is LLM Finetuning?](#what-is-llm-finetuning) * [Benefits of Finetuning](#benefits-of-finetuning) * [When to Finetune](#when-to-finetune) * [Prompt engineering vs Few-shot learning vs Finetuning](#prompt-engineering-vs-few-shot-learning-vs-finetuning) --- # Finetuning models - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Finetuning Finetuning models Click on the “Finetune Model” button on the task page to get started! ### [​](#step-1-selecting-the-base-model) Step 1: Selecting the Base Model Navigate to the Finetune model interface. Under the Base Model section, choose the appropriate model based on your use case: Refuel-LLM V2 Most capable model with a 32k context length and a 47B parameter size. Suited for complex reasoning tasks where detailed understanding and large input/output processing are required. **Pros:** Handles long context inputs effectively (e.g., large documents or conversations). Best choice for tasks requiring high-level reasoning and accuracy. High parameter count allows for nuanced understanding and generation. **Cons:** Requires more training data to fine-tune effectively. Higher latency due to its size, making it slower for real-time applications. More resource-intensive (computational cost and time). Refuel-LLM V2 Small A highly capable model with an 8k context length and a 8B parameter size. Optimized for tasks such as data extraction and pattern matching. **Pros:** Faster latency compared to Refuel-LLM V2, making it more efficient for smaller tasks. Requires less training data and computational resources. Well-suited for structured tasks like entity recognition or data classification. **Cons:** Lower capacity for handling longer context inputs (e.g., long documents). Limited reasoning capability compared to larger models. May struggle with complex tasks requiring high precision. Refuel-LLM V2 Mini Smallest model with an 8k context length and a 1.5B parameter size. Designed for high throughput and low latency tasks. **Pros:** Extremely fast and efficient, making it ideal for real-time use cases (e.g., chatbots, low-latency inference). Requires minimal computational resources for training and deployment. Best choice when performance speed is critical, or large-scale deployment is needed. **Cons:** Limited capacity for reasoning and complex tasks due to its small parameter size. Performs less effectively on tasks involving nuanced language understanding. Handles only moderate context lengths, making it unsuitable for tasks with extensive inputs. ### [​](#step-2-selecting-training-data) Step 2: Selecting Training Data Under the Training data section, select the datasets you want to use for fine-tuning. Use the dropdown menu labeled Select datasets to choose the desired training datasets. Selected datasets will appear under the Selected datasets list. Ensure the datasets include human-verified labels for optimal results. Optionally, toggle Augment human-verified labels to allow the system to enhance training data with automated augmentation using models like GPT-4o. ### [​](#step-3-configuring-advanced-settings) Step 3: Configuring Advanced Settings Expand the Advanced settings section to adjust fine-tuning parameters: LoRA Rank LoRA (Low-Rank Adaptation) is a technique that reduces the number of trainable parameters by introducing low-rank matrices during fine-tuning. The LoRA rank determines the rank (or size) of these low-rank matrices. Higher values increase the capacity of the fine-tuning process, allowing the model to learn more complex patterns. A higher LoRA rank improves the model’s ability to adapt to your training data, which can result in better fine-tuning performance. A lower LoRA rank reduces computational cost and training time but may sacrifice some accuracy. A rank of 32 (as shown in the slider) indicates a balance between complexity and efficiency. **Recommendation:** Increase the rank for tasks requiring more complex learning (e.g., nuanced patterns or long sequences). Decrease it for simpler tasks to save resources. Epochs An epoch represents one complete pass of the training data through the model during fine-tuning. The number of epochs determines how many times the model sees and learns from the training data. A higher number of epochs allows the model to learn more from the data, improving its performance—up to a point. If the number of epochs is too high, the model may overfit the training data, reducing its performance on unseen tasks. A lower number of epochs speeds up the training but may result in underfitting. 10 epochs (as shown in the slider) is a reasonable default for balanced learning. Adjust this based on the dataset size and complexity. **Recommendation:** For smaller datasets, use more epochs to ensure the model learns enough. For large datasets, fewer epochs are usually sufficient. Learning Rate Multiplier The learning rate controls how much the model’s weights are updated with each training step. The learning rate multiplier scales the base learning rate, allowing you to fine-tune its impact. A higher learning rate speeds up training but may cause the model to converge too quickly, potentially missing optimal solutions. A lower learning rate makes training slower but ensures more gradual and stable convergence. A multiplier of 20 (as shown in the slider) indicates a moderate learning rate. This should work for most datasets but could be adjusted for more delicate training. **Recommendation:** Start with a moderate multiplier. If the model fails to converge or produces poor results, reduce the learning rate. If training is too slow or the model seems stuck, increase the multiplier. [Overview](/guides/models/overview) [Tracking finetuning runs](/guides/models/models) On this page * [Step 1: Selecting the Base Model](#step-1-selecting-the-base-model) * [Step 2: Selecting Training Data](#step-2-selecting-training-data) * [Step 3: Configuring Advanced Settings](#step-3-configuring-advanced-settings) --- # Tracking finetuning runs - Refuel.ai [Refuel.ai home page![light logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo.svg)![dark logo](https://mintlify.s3.us-west-1.amazonaws.com/refuelai/logo/logo_dark.svg)](https://refuel.ai) [Documentation](/overview) [Integrations](/integrations/introduction) [SDK Reference](/sdk/python) Search... Navigation Finetuning Tracking finetuning runs Track the history of your model finetuning runs under the **Models** tab on the left side navigation bar. ### [​](#finetuning-run-page) Finetuning Run Page Click on a finetuning run to see detailed information about the training run, including the dataets used for training, hyperparameter settings, and evaluation metrics. Some details about the various sections on this page: #### [​](#availability) **Availability**: Shows the availability status of the model. If the training is still in progress, or failed for some reason, the model will be marked as **unavailable**. If the training is successful, the model will be marked as **available**. #### [​](#workflow-steps) **Workflow Steps**: Any finetuning run will go through the following steps: 1. **Initialization**: Initial setup phase for the model fine-tuning. 2. **Dataset Preparation**: During this step, the training dataset is collected and prepared with the correct formatting and few-shot examples. 3. **Resource Allocation**: Infrastructure (GPU) resources and provisioned for the training run. 4. **Training**: During this step, the model weights are tuned using the training dataset. For most finetuning tasks, this is the step that takes the longest. 5. **Deployment**: Once the training completes, the new model weights are deployed online for use in your tasks. 6. **Evaluation**: During this step, the model’s output quality is evaluated on a heldout evaluation dataset. #### [​](#training-progress) Training Progress Visual representation of the training and evaluation loss over time, showcasing the model’s learning curve and performance improvement. Refuel will automatically save model checkpoints at regular intervals during training. At the end, we will select the best checkpoint (i.e. the one with lowest loss on the validation set) for deployment. #### [​](#evaluation-summary) Evaluation Summary A graphical representation (bar chart) of the model’s evaluation metrics: * **Accuracy**: Not displayed in the graph. * **F1 Score**: Measures a balance between precision and recall. * **Precision**: Indicates the percentage of relevant instances correctly identified by the model. * **Recall**: Measures the model’s ability to capture all relevant instances. Compares the performance of the fine-tuned model against a baseline or preview model (e.g., `gpt-4-1106-preview`). [Finetuning models](/guides/models/finetuning) [Using finetuned models](/guides/models/using_finetuned_models) On this page * [Finetuning Run Page](#finetuning-run-page) * [Availability:](#availability) * [Workflow Steps:](#workflow-steps) * [Training Progress](#training-progress) * [Evaluation Summary](#evaluation-summary) ---