# Table of Contents - [Welcome to InternVL’s tutorials! — InternVL](#welcome-to-internvl-s-tutorials-internvl) - [Unknown](#unknown) - [Installation — InternVL](#installation-internvl) - [Chat Data Format — InternVL](#chat-data-format-internvl) - [InternVL-Chat API — InternVL](#internvl-chat-api-internvl) - [Search - InternVL](#search-internvl) - [Index — InternVL](#index-internvl) - [Local Chat Demo — InternVL](#local-chat-demo-internvl) - [Introduction of InternVL3.0 Series — InternVL](#introduction-of-internvl3-0-series-internvl) - [Enhancing InternVL2 on COCO Caption Using LoRA Fine-Tuning — InternVL](#enhancing-internvl2-on-coco-caption-using-lora-fine-tuning-internvl) - [Evaluation of InternVL3 Series — InternVL](#evaluation-of-internvl3-series-internvl) - [Fine-tune on a Custom Dataset — InternVL](#fine-tune-on-a-custom-dataset-internvl) - [FAQs — InternVL](#faqs-internvl) - [Introduction of InternVL2.5 Series — InternVL](#introduction-of-internvl2-5-series-internvl) - [Mixed Preference Optimization — InternVL](#mixed-preference-optimization-internvl) - [Evaluation Data Preparation — InternVL](#evaluation-data-preparation-internvl) - [Fine-tune on a Custom Dataset — InternVL](#fine-tune-on-a-custom-dataset-internvl) - [Evaluation of InternVL2.5 Series — InternVL](#evaluation-of-internvl2-5-series-internvl) - [Mixed Preference Optimization — InternVL](#mixed-preference-optimization-internvl) - [Introduction of InternVL2 Series — InternVL](#introduction-of-internvl2-series-internvl) - [Quick Start of InternVL 3.0 Series — InternVL](#quick-start-of-internvl-3-0-series-internvl) - [Introduction of InternVL-Chat-V1-2 — InternVL](#introduction-of-internvl-chat-v1-2-internvl) - [Domain Adaptation — InternVL](#domain-adaptation-internvl) - [Fine-tune on a Custom Dataset — InternVL](#fine-tune-on-a-custom-dataset-internvl) - [Introduction of InternVL 1.5 Series — InternVL](#introduction-of-internvl-1-5-series-internvl) - [Fine-tune on a Custom Dataset — InternVL](#fine-tune-on-a-custom-dataset-internvl) - [Deploy InternVL3 Series — InternVL](#deploy-internvl3-series-internvl) - [Introduction of InternVL-Chat-V1-1 — InternVL](#introduction-of-internvl-chat-v1-1-internvl) - [Reproduce InternVL-Chat-V1-2 — InternVL](#reproduce-internvl-chat-v1-2-internvl) - [Mixed Preference Optimization — InternVL](#mixed-preference-optimization-internvl) - [Fine-tune on a Custom Dataset — InternVL](#fine-tune-on-a-custom-dataset-internvl) - [InternViT-6B for Semantic Segmentation — InternVL](#internvit-6b-for-semantic-segmentation-internvl) - [InternViT-6B for Image Classification — InternVL](#internvit-6b-for-image-classification-internvl) - [Deploy InternVL2.5 Series — InternVL](#deploy-internvl2-5-series-internvl) - [Deploy InternVL 1.5 Series — InternVL](#deploy-internvl-1-5-series-internvl) - [Quick Start of InternVL 2.5 Series — InternVL](#quick-start-of-internvl-2-5-series-internvl) - [Quick Start of InternVL-Chat-V1-2 — InternVL](#quick-start-of-internvl-chat-v1-2-internvl) - [Deploy InternVL2 Series — InternVL](#deploy-internvl2-series-internvl) - [Quick Start of InternVL2 Series — InternVL](#quick-start-of-internvl2-series-internvl) - [InternVL Stage-2 Pre-training & Retrieval Fine-tuning — InternVL](#internvl-stage-2-pre-training-retrieval-fine-tuning-internvl) - [Quick Start of InternVL-Chat-V1-1 — InternVL](#quick-start-of-internvl-chat-v1-1-internvl) - [InternVL for Multimodal Dialogue using LLaVA Codebase — InternVL](#internvl-for-multimodal-dialogue-using-llava-codebase-internvl) - [Evaluation of InternVL 1.5 Series — InternVL](#evaluation-of-internvl-1-5-series-internvl) - [Quick Start of InternVL 1.5 Series — InternVL](#quick-start-of-internvl-1-5-series-internvl) - [Evaluation of InternVL-Chat-V1-2 — InternVL](#evaluation-of-internvl-chat-v1-2-internvl) - [Evaluation of InternVL-Chat-V1-1 — InternVL](#evaluation-of-internvl-chat-v1-1-internvl) - [InternVL for Zero-Shot Image Classification & Image-Text Retrieval — InternVL](#internvl-for-zero-shot-image-classification-image-text-retrieval-internvl) - [Evaluation of InternVL2 Series — InternVL](#evaluation-of-internvl2-series-internvl) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) - [Unknown](#unknown) --- # Welcome to InternVL’s tutorials! — InternVL [Skip to main content](https://internvl.readthedocs.io/en/latest/#main-content) Back to top Ctrl+K [![InternVL - Home](https://internvl.readthedocs.io/en/latest/_static/new_logo.svg) ![InternVL - Home](https://internvl.readthedocs.io/en/latest/_static/new_logo.svg)](https://internvl.readthedocs.io/en/latest/#) Search Ctrl+K * [Repository](https://github.com/OpenGVLab/InternVL) * [Show source](https://github.com/OpenGVLab/InternVL/blob/main/docs/en/index.rst?plain=1) * [Suggest edit](https://github.com/OpenGVLab/InternVL/edit/main/docs/en/index.rst) * [Open issue](https://github.com/OpenGVLab/InternVL/issues/new?title=Issue%20on%20page%20%2Findex.html&body=Your%20issue%20content%20here.) * [.rst](https://internvl.readthedocs.io/en/latest/_sources/index.rst) * .pdf Welcome to InternVL’s tutorials! ================================ Contents -------- Welcome to InternVL’s tutorials![#](https://internvl.readthedocs.io/en/latest/#welcome-to-internvl-s-tutorials "Link to this heading") ======================================================================================================================================= ![InternVL](https://internvl.readthedocs.io/en/latest/_images/new_logo.svg) **InternVL Family: Closing the Gap to Commercial Multimodal Models with Open-Source Suites —— A Pioneering Open-Source Alternative to GPT-4o.** [![OpenGVLab%2FInternVL | Trendshift](https://trendshift.io/api/badge/repositories/9803)](https://trendshift.io/repositories/9803) ![image](https://internvl.readthedocs.io/en/latest/_static/image/cvpr_oral.png) Documentation[#](https://internvl.readthedocs.io/en/latest/#documentation "Link to this heading") -------------------------------------------------------------------------------------------------- Get Started * [Installation](https://internvl.readthedocs.io/en/latest/get_started/installation.html) * [Chat Data Format](https://internvl.readthedocs.io/en/latest/get_started/chat_data_format.html) * [Evaluation Data Preparation](https://internvl.readthedocs.io/en/latest/get_started/eval_data_preparation.html) * [Local Chat Demo](https://internvl.readthedocs.io/en/latest/get_started/local_chat_demo.html) * [InternVL-Chat API](https://internvl.readthedocs.io/en/latest/get_started/internvl_chat_api.html) Tutorials * [Enhancing InternVL2 on COCO Caption Using LoRA Fine-Tuning](https://internvl.readthedocs.io/en/latest/tutorials/coco_caption_finetune.html) * [FAQs](https://internvl.readthedocs.io/en/latest/tutorials/faqs.html) InternVL 3.0 * [Introduction](https://internvl.readthedocs.io/en/latest/internvl3.0/introduction.html) * [Quick Start](https://internvl.readthedocs.io/en/latest/internvl3.0/quick_start.html) * [Finetune](https://internvl.readthedocs.io/en/latest/internvl3.0/finetune.html) * [Evaluation](https://internvl.readthedocs.io/en/latest/internvl3.0/evaluation.html) * [Deployment](https://internvl.readthedocs.io/en/latest/internvl3.0/deployment.html) * [Preference Optimization](https://internvl.readthedocs.io/en/latest/internvl3.0/preference_optimization.html) InternVL 2.5 * [Introduction](https://internvl.readthedocs.io/en/latest/internvl2.5/introduction.html) * [Quick Start](https://internvl.readthedocs.io/en/latest/internvl2.5/quick_start.html) * [Finetune](https://internvl.readthedocs.io/en/latest/internvl2.5/finetune.html) * [Evaluation](https://internvl.readthedocs.io/en/latest/internvl2.5/evaluation.html) * [Deployment](https://internvl.readthedocs.io/en/latest/internvl2.5/deployment.html) * [Preference Optimization](https://internvl.readthedocs.io/en/latest/internvl2.5/preference_optimization.html) InternVL 2.0 * [Introduction](https://internvl.readthedocs.io/en/latest/internvl2.0/introduction.html) * [Quick Start](https://internvl.readthedocs.io/en/latest/internvl2.0/quick_start.html) * [Finetune](https://internvl.readthedocs.io/en/latest/internvl2.0/finetune.html) * [Evaluation](https://internvl.readthedocs.io/en/latest/internvl2.0/evaluation.html) * [Deployment](https://internvl.readthedocs.io/en/latest/internvl2.0/deployment.html) * [Domain Adaptation](https://internvl.readthedocs.io/en/latest/internvl2.0/domain_adaptation.html) * [Preference Optimization](https://internvl.readthedocs.io/en/latest/internvl2.0/preference_optimization.html) InternVL 1.5 * [Introduction](https://internvl.readthedocs.io/en/latest/internvl1.5/introduction.html) * [Quick Start](https://internvl.readthedocs.io/en/latest/internvl1.5/quick_start.html) * [Finetune](https://internvl.readthedocs.io/en/latest/internvl1.5/finetune.html) * [Evaluation](https://internvl.readthedocs.io/en/latest/internvl1.5/evaluation.html) * [Deployment](https://internvl.readthedocs.io/en/latest/internvl1.5/deployment.html) InternVL 1.2 * [Introduction](https://internvl.readthedocs.io/en/latest/internvl1.2/introduction.html) * [Quick Start](https://internvl.readthedocs.io/en/latest/internvl1.2/quick_start.html) * [Reproduce](https://internvl.readthedocs.io/en/latest/internvl1.2/reproduce.html) * [Finetune](https://internvl.readthedocs.io/en/latest/internvl1.2/finetune.html) * [Evaluation](https://internvl.readthedocs.io/en/latest/internvl1.2/evaluation.html) InternVL 1.1 * [Introduction](https://internvl.readthedocs.io/en/latest/internvl1.1/introduction.html) * [Quick Start](https://internvl.readthedocs.io/en/latest/internvl1.1/quick_start.html) * [Evaluation](https://internvl.readthedocs.io/en/latest/internvl1.1/evaluation.html) InternVL 1.0 * [classification](https://internvl.readthedocs.io/en/latest/internvl1.0/classification.html) * [clip\_benchmark](https://internvl.readthedocs.io/en/latest/internvl1.0/clip_benchmark.html) * [segmentation](https://internvl.readthedocs.io/en/latest/internvl1.0/segmentation.html) * [internvl\_chat\_llava](https://internvl.readthedocs.io/en/latest/internvl1.0/internvl_chat_llava.html) * [internvl\_g](https://internvl.readthedocs.io/en/latest/internvl1.0/internvl_g.html) Indices and tables[#](https://internvl.readthedocs.io/en/latest/#indices-and-tables "Link to this heading") ============================================================================================================ * [Index](https://internvl.readthedocs.io/en/latest/genindex.html) * [Search Page](https://internvl.readthedocs.io/en/latest/search.html) Contents [Develop and launch modern apps with MongoDB Atlas, a resilient data platform.](https://server.ethicalads.io/proxy/click/9026/0198cb37-3b4b-7041-bd48-85ca46d84c4f/) [Ads by EthicalAds](https://www.ethicalads.io/advertisers/?ref=ea-text) --- # Unknown Welcome to InternVL's tutorials! ==================================== .. figure:: ./\_static/image/new\_logo.svg :width: 50% :align: center :alt: InternVL :class: no-scaled-link .. raw:: html **InternVL Family: Closing the Gap to Commercial Multimodal Models with Open-Source Suites —— A Pioneering Open-Source Alternative to GPT-4o.** [![OpenGVLab%2FInternVL | Trendshift](https://trendshift.io/api/badge/repositories/9803)](https://trendshift.io/repositories/9803) ![image](https://internvl.readthedocs.io/en/latest/_sources/_static/image/cvpr_oral.png) [Star](https://github.com/OpenGVLab/InternVL) [Watch](https://github.com/OpenGVLab/InternVL/subscription) [Fork](https://github.com/OpenGVLab/InternVL/fork) Documentation ------------- .. \_get\_started: .. toctree:: :maxdepth: 1 :caption: Get Started get\_started/installation.md get\_started/chat\_data\_format.md get\_started/eval\_data\_preparation.md get\_started/local\_chat\_demo.md get\_started/internvl\_chat\_api.md .. \_tutorials: .. toctree:: :maxdepth: 1 :caption: Tutorials tutorials/coco\_caption\_finetune.md tutorials/faqs.md .. \_internvl\_3\_0: .. toctree:: :maxdepth: 1 :caption: InternVL 3.0 Introduction Quick Start Finetune Evaluation Deployment Preference Optimization .. \_internvl\_2\_5: .. toctree:: :maxdepth: 1 :caption: InternVL 2.5 Introduction Quick Start Finetune Evaluation Deployment Preference Optimization .. \_internvl\_2\_0: .. toctree:: :maxdepth: 1 :caption: InternVL 2.0 Introduction Quick Start Finetune Evaluation Deployment Domain Adaptation Preference Optimization .. \_internvl\_1\_5: .. toctree:: :maxdepth: 1 :caption: InternVL 1.5 Introduction Quick Start Finetune Evaluation Deployment .. \_internvl\_1\_2: .. toctree:: :maxdepth: 1 :caption: InternVL 1.2 Introduction Quick Start Reproduce Finetune Evaluation .. \_internvl\_1\_1: .. toctree:: :maxdepth: 1 :caption: InternVL 1.1 Introduction Quick Start Evaluation .. \_internvl\_1\_0: .. toctree:: :maxdepth: 1 :caption: InternVL 1.0 :titlesonly: classification clip\_benchmark segmentation internvl\_chat\_llava internvl\_g .. \_classic\_questions: .. toctree:: :maxdepth: 1 :caption: Classic Questions Classic Questions Indices and tables ================== \* :ref:\`genindex\` \* :ref:\`search\` --- # Installation — InternVL [Skip to main content](https://internvl.readthedocs.io/en/latest/get_started/installation.html#main-content) Back to top Ctrl+K [![InternVL - Home](https://internvl.readthedocs.io/en/latest/_static/new_logo.svg)](https://internvl.readthedocs.io/en/latest/index.html) * [Repository](https://github.com/OpenGVLab/InternVL "Source repository") * [Show source](https://github.com/OpenGVLab/InternVL/blob/main/docs/en/get_started/installation.md?plain=1 "Show source") * [Suggest edit](https://github.com/OpenGVLab/InternVL/edit/main/docs/en/get_started/installation.md "Suggest edit") * [Open issue](https://github.com/OpenGVLab/InternVL/issues/new?title=Issue%20on%20page%20%2Fget_started/installation.html&body=Your%20issue%20content%20here. "Open an issue") * [.md](https://internvl.readthedocs.io/en/latest/_sources/get_started/installation.md "Download source file") * .pdf Installation ============ Contents -------- Installation[#](https://internvl.readthedocs.io/en/latest/get_started/installation.html#installation "Link to this heading") ============================================================================================================================= * Clone this repository: git clone https://github.com/OpenGVLab/InternVL.git * Create a conda virtual environment and activate it: conda create \-n internvl python\=3.9 conda activate internvl * Install dependencies using `requirements.txt`: pip install \-r requirements.txt By default, our `requirements.txt` file includes the following dependencies: * `-r requirements/internvl_chat.txt` * `-r requirements/streamlit_demo.txt` * `-r requirements/classification.txt` * `-r requirements/segmentation.txt` The `clip_benchmark.txt` is **not** included in the default installation. If you require the `clip_benchmark` functionality (for the evaluation of zero-shot classification or retrieval), please install it manually by running the following command: pip install \-r requirements/clip\_benchmark.txt Additional Instructions[#](https://internvl.readthedocs.io/en/latest/get_started/installation.html#additional-instructions "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------- * Install `flash-attn==2.3.6` (optional, for training chat models): pip install flash-attn\==2.3.6 \--no-build-isolation Alternatively you can compile from source: git clone https://github.com/Dao-AILab/flash-attention.git cd flash-attention git checkout v2.3.6 python setup.py install * Install `mmcv-full==1.6.2` (optional, for `segmentation`): pip install \-U openmim mim install mmcv-full\==1.6.2 * Install `apex` (optional, for `segmentation`): git clone https://github.com/NVIDIA/apex.git git checkout 2386a912164b0c5cfcd8be7a2b890fbac5607c82 \# https://github.com/NVIDIA/apex/issues/1735 pip install \-v \--disable-pip-version-check \--no-cache-dir \--no-build-isolation \--config-settings "--build-option=--cpp\_ext" \--config-settings "--build-option=--cuda\_ext" ./ If you encounter `ModuleNotFoundError: No module named 'fused_layer_norm_cuda'`, it is because apex’s CUDA extensions are not being installed successfully. You can try uninstalling apex and the code will default to the PyTorch version of RMSNorm. Alternatively, if you prefer using apex, try adding a few lines to `setup.py` and then recompiling. ![](https://internvl.readthedocs.io/en/latest/_static/image/apex_code.png) Contents --- # Chat Data Format — InternVL [Skip to main content](https://internvl.readthedocs.io/en/latest/get_started/chat_data_format.html#main-content) Back to top Ctrl+K [![InternVL - Home](https://internvl.readthedocs.io/en/latest/_static/new_logo.svg)](https://internvl.readthedocs.io/en/latest/index.html) * [Repository](https://github.com/OpenGVLab/InternVL "Source repository") * [Show source](https://github.com/OpenGVLab/InternVL/blob/main/docs/en/get_started/chat_data_format.md?plain=1 "Show source") * [Suggest edit](https://github.com/OpenGVLab/InternVL/edit/main/docs/en/get_started/chat_data_format.md "Suggest edit") * [Open issue](https://github.com/OpenGVLab/InternVL/issues/new?title=Issue%20on%20page%20%2Fget_started/chat_data_format.html&body=Your%20issue%20content%20here. "Open an issue") * [.md](https://internvl.readthedocs.io/en/latest/_sources/get_started/chat_data_format.md "Download source file") * .pdf Chat Data Format ================ Contents -------- Chat Data Format[#](https://internvl.readthedocs.io/en/latest/get_started/chat_data_format.html#chat-data-format "Link to this heading") ========================================================================================================================================= Dataset Configuration[#](https://internvl.readthedocs.io/en/latest/get_started/chat_data_format.html#dataset-configuration "Link to this heading") --------------------------------------------------------------------------------------------------------------------------------------------------- In InternVL 2.0 and 2.5, the organization of the training data is controlled by several key parameters to optimize the balance and distribution of datasets during training. ![image/png](https://internvl.readthedocs.io/en/latest/_images/data_configuration.png) * **Data Augmentation:** JPEG compression is applied conditionally: enabled for image datasets to enhance robustness and disabled for video datasets to maintain consistent frame quality. * **Maximum Tile Number:** The parameter `n_max` controls the maximum tiles per dataset. For example, higher values (24–36) are used for multi-image or high-resolution data, lower values (6–12) for standard images, and 1 for videos. * **Repeat Factor:** The repeat factor `r` adjusts dataset sampling frequency. Values below 1 reduce a dataset’s weight, while values above 1 increase it. This ensures balanced training across tasks and prevents overfitting or underfitting. Meta File[#](https://internvl.readthedocs.io/en/latest/get_started/chat_data_format.html#meta-file "Link to this heading") --------------------------------------------------------------------------------------------------------------------------- In this document, we will detail the organization format of our conversation data. Currently, we use a JSON file to manage the meta information of all datasets. The format is as follows: { "your-custom-dataset-1": { "root": "path/to/the/image/", "annotation": "path/to/the/jsonl/annotation", "data\_augment": false, "max\_dynamic\_patch": 12, "repeat\_time": 1, "length": "number of samples in the dataset" }, ... } Here, `root` is the root directory of the dataset, `annotation` is the path to the annotation file, `data_augment` indicates whether data augmentation is needed, `repeat_time` is the number of times the dataset is repeated, and `length` is the number of samples in the dataset. For example, a file for the ShareGPT4V dataset looks like this: { "sharegpt4v\_instruct\_gpt4-vision\_cap100k": { "root": "playground/data/", "annotation": "playground/opensource/sharegpt4v\_instruct\_gpt4-vision\_cap100k.jsonl", "data\_augment": false, "max\_dynamic\_patch": 12, "repeat\_time": 1, "length": 102025 }, ... } You can add multiple datasets in this JSON file, similar to [this file](https://github.com/OpenGVLab/InternVL/blob/main/internvl_chat/shell/data/internvl_1_2_finetune.json) . We currently support the following four types of datasets: `pure text data`, `single-image data`, `multi-image (interleaved) data`, and `video data`. We do not require all entries in a JSONL file to be of the same type, meaning your JSONL file can contain different types of data. Pure Text Data[#](https://internvl.readthedocs.io/en/latest/get_started/chat_data_format.html#pure-text-data "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------- For pure text data, we use a JSONL file to store the data. Each entry is a dictionary organized in the following format. Note that entries for pure text data should not contain an `image` field. { "id": 0, "conversations": \[\ {"from": "human", "value": "user input"},\ {"from": "gpt", "value": "assistant output"},\ {"from": "human", "value": "user input"},\ {"from": "gpt", "value": "assistant output"}\ \] } Here, `id` is the unique identifier for the data, and `conversations` is a list containing multiple conversations. Each conversation is a dictionary with two key-value pairs: `from` and `value`. `from` indicates the speaker’s identity, and `value` indicates the speaker’s input. This data format is consistent with LLaVA 1.5, with the difference being that LLaVA 1.5 uses JSON, while we use JSONL. An example of pure text data: {"id": 0, "conversations": \[{"from": "human", "value": "Find the length of the hypotenuse given two sides\\n\\"side\_length1 = 5, side\_length2 = 6\\""}, {"from": "gpt", "value": "To find the length of the hypotenuse given the two sides of a right triangle, you can use the Pythagorean theorem. The theorem states that the square of the length of the hypotenuse is equal to the sum of the squares of the lengths of the two other sides.\\n \\nIn this case, the length of one side is 5, and the length of the other side is 6. Plugging these values into the formula, we get:\\n\\nhypotenuse2 = 5^2 + 6^2\\n = 25 + 36\\n = 61\\n\\nTo get the length of the hypotenuse, take the square root of 61:\\n\\nhypotenuse = √61\\n = 7.81 (rounded to two decimal places)\\n\\nTherefore, the length of the hypotenuse is approximately 7.81 units."}\]} Here is the more readable version: { "id": 0, "conversations": \[\ {\ "from": "human",\ "value": "Find the length of the hypotenuse given two sides\\n\\"side\_length1 = 5, side\_length2 = 6\\""\ },\ {\ "from": "gpt",\ "value": "To find the length of the hypotenuse given the two sides of a right triangle, you can use the Pythagorean theorem. The theorem states that the square of the length of the hypotenuse is equal to the sum of the squares of the lengths of the two other sides.\\n\\nIn this case, the length of one side is 5, and the length of the other side is 6. Plugging these values into the formula, we get:\\n\\nhypotenuse^2 = 5^2 + 6^2\\n = 25 + 36\\n = 61\\n\\nTo get the length of the hypotenuse, take the square root of 61:\\n\\nhypotenuse = √61\\n = 7.81 (rounded to two decimal places)\\n\\nTherefore, the length of the hypotenuse is approximately 7.81 units."\ }\ \] } Single-Image Data[#](https://internvl.readthedocs.io/en/latest/get_started/chat_data_format.html#single-image-data "Link to this heading") ------------------------------------------------------------------------------------------------------------------------------------------- For single-image data, we use a JSONL file to store the data. Each entry is a dictionary organized in the following format. Note that each entry for single-image data must contain an `image` field, which is a string. The path in the `image` field is relative to the `root` field. Concatenating the `root` field and the `image` field gives the complete path to the image. It is recommended to include `width` and `height` information for each data sample for future use. { "id": 0, "image": "path/to/image.jpg", "width": 111, "height": 222, "conversations": \[\ {"from": "human", "value": "\\nuser input"},\ {"from": "gpt", "value": "assistant output"},\ {"from": "human", "value": "user input"},\ {"from": "gpt", "value": "assistant output"}\ \] } Here, `` indicates the position where the image is inserted, and the number of `` placeholders should match the number of images. In single-image data, the `` placeholder should appear only once across all conversations. An example of single-image data: {"id": 0, "image": "images/00000000.jpg", "conversations": \[{"from": "human", "value": "\\nCan you extract any readable text from the image?"}, {"from": "gpt", "value": "Dares Wins Vol. 5 Tommy's Heroes Vol. 6: For Tomorrow Vol. 7: Closing Time miniseries. Clark Kent is being interviewed about Superman's connection to notorious killer Tommy Monaghan. Taking the conversation..."}\], "width": 897, "height": 1152} Here is the more readable version: { "id": 0, "image": "images/00000000.jpg", "width": 897, "height": 1152, "conversations": \[\ {\ "from": "human",\ "value": "\\nCan you extract any readable text from the image?"\ },\ {\ "from": "gpt",\ "value": "Dares Wins Vol. 5 Tommy's Heroes Vol. 6: For Tomorrow Vol. 7: Closing Time miniseries. Clark Kent is being interviewed about Superman's connection to notorious killer Tommy Monaghan. Taking the conversation..."\ }\ \] } ### Grounding / Detection Data[#](https://internvl.readthedocs.io/en/latest/get_started/chat_data_format.html#grounding-detection-data "Link to this heading") For grounding or detection data, we normalize each coordinate box to the range `[0, 1000]` using the width and height of the image, and round the normalized coordinates to integers: def normalize\_coordinates(box, image\_width, image\_height): x1, y1, x2, y2 \= box normalized\_box \= \[\ round((x1 / image\_width) \* 1000),\ round((y1 / image\_height) \* 1000),\ round((x2 / image\_width) \* 1000),\ round((y2 / image\_height) \* 1000)\ \] return normalized\_box We organize the model’s responses in the following format: `class name[[x1, y1, x2, y2], ...]` An example of the RefCOCO dataset: {"id": 2324, "image": "COCO\_train2014\_000000581857.jpg", "width": 427, "height": 640, "conversations": \[{"from": "human", "value": "\\nPlease provide the bounding box coordinate of the region this sentence describes: the lady with the blue shirt"}, {"from": "gpt", "value": "the lady with the blue shirt\[\[243, 469, 558, 746\]\]"}, {"from": "human", "value": "Please provide the bounding box coordinate of the region this sentence describes: woman in gray shirt facing camera on right"}, {"from": "gpt", "value": "woman in gray shirt facing camera on right\[\[507, 409, 698, 740\]\]"}, {"from": "human", "value": "Please provide the bounding box coordinate of the region this sentence describes: a woman in glasses shops in an open air fruit market"}, {"from": "gpt", "value": "a woman in glasses shops in an open air fruit market\[\[196, 422, 395, 708\]\]"}\]} Here is the more readable version: { "id": 2324, "image": "COCO\_train2014\_000000581857.jpg", "width": 427, "height": 640, "conversations": \[\ {\ "from": "human",\ "value": "\\nPlease provide the bounding box coordinate of the region this sentence describes: the lady with the blue shirt"\ },\ {\ "from": "gpt",\ "value": "the lady with the blue shirt\[\[243, 469, 558, 746\]\]"\ },\ {\ "from": "human",\ "value": "Please provide the bounding box coordinate of the region this sentence describes: woman in gray shirt facing camera on right"\ },\ {\ "from": "gpt",\ "value": "woman in gray shirt facing camera on right\[\[507, 409, 698, 740\]\]"\ },\ {\ "from": "human",\ "value": "Please provide the bounding box coordinate of the region this sentence describes: a woman in glasses shops in an open air fruit market"\ },\ {\ "from": "gpt",\ "value": "a woman in glasses shops in an open air fruit market\[\[196, 422, 395, 708\]\]"\ }\ \] } An example of the V3Det dataset: {"id": 78281, "image": "images/x00001541/000106464.jpg", "width": 800, "height": 800, "conversations": \[{"from": "human", "value": "\\nPlease detect and label all objects in the following image and mark their positions."}, {"from": "gpt", "value": "Sure, I will detect and label all objects in the image and mark their positions.\\n\\n\`\`\`\\nBulletproof Helmet\[\[1, 2, 269, 235\]\]\\nBulletproof Clothing\[\[650, 619, 990, 993\]\]\\nGun Model\[\[32, 231, 977, 662\]\]\\nscrew\[\[754, 376, 851, 429\]\]\\nhandcuff\[\[698, 228, 931, 386\]\]\\n95 Type Assault Rifle\[\[39, 229, 983, 667\]\]\\nshield\[\[30, 597, 273, 993\]\]\\ntelescope\[\[666, 38, 890, 217\]\]\\nWireless Walkie-Talkie\[\[295, 2, 370, 226\], \[374, 0, 447, 226\]\]\\nbomb\[\[473, 61, 552, 181\], \[569, 61, 648, 183\]\]\\nweapon\[\[302, 617, 342, 993\]\]\\nvessel\[\[355, 653, 644, 991\]\]\\nartifact\[\[915, 0, 981, 294\]\]\\n\`\`\`\\n"}\]} Here is the more readable version: { "id": 78281, "image": "images/x00001541/000106464.jpg", "width": 800, "height": 800, "conversations": \[\ {\ "from": "human",\ "value": "\\nPlease detect and label all objects in the following image and mark their positions."\ },\ {\ "from": "gpt",\ "value": "Sure, I will detect and label all objects in the image and mark their positions.\\n\\nBulletproof Helmet\[\[1, 2, 269, 235\]\]\\nBulletproof Clothing\[\[650, 619, 990, 993\]\]\\nGun Model\[\[32, 231, 977, 662\]\]\\nscrew\[\[754, 376, 851, 429\]\]\\nhandcuff\[\[698, 228, 931, 386\]\]\\n95 Type Assault Rifle\[\[39, 229, 983, 667\]\]\\nshield\[\[30, 597, 273, 993\]\]\\ntelescope\[\[666, 38, 890, 217\]\]\\nWireless Walkie-Talkie\[\[295, 2, 370, 226\], \[374, 0, 447, 226\]\]\\nbomb\[\[473, 61, 552, 181\], \[569, 61, 648, 183\]\]\\nweapon\[\[302, 617, 342, 993\]\]\\nvessel\[\[355, 653, 644, 991\]\]\\nartifact\[\[915, 0, 981, 294\]\]\\n"\ }\ \] } Multi-Image Data[#](https://internvl.readthedocs.io/en/latest/get_started/chat_data_format.html#multi-image-data "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------------------- For multi-image data, we use a JSONL file to store the data. Each entry is a dictionary organized in the following format. Note that each entry for multi-image data must contain an `image` field, which is a list of strings. Each element in the list is a path relative to the `root` field. Concatenating the `root` field and each element gives the complete path to the images. It is recommended to include `width_list` and `height_list` information for each data sample for future use. { "id": 0, "image": \["path/to/image1.jpg", "path/to/image2.jpg", "path/to/image3.jpg"\], "width\_list": \[111, 222, 333\], "height\_list": \[111, 222, 333\], "conversations": \[\ {"from": "human", "value": "\\nuser input \\nuser input"},\ {"from": "gpt", "value": "assistant output"},\ {"from": "human", "value": "\\nuser input"},\ {"from": "gpt", "value": "assistant output"}\ \] } Here, `` indicates the position where the images are inserted, and the number of `` placeholders should match the number of images. In this example, the `image` field list contains three elements, so the `` placeholder also needs to appear three times. An example of multi-image data: {"id": 0, "image": \["cimages/multimages/16/5pc.png", "cimages/multimages/16/5pd.png", "cimages/multimages/16/1602207874\_p5b.png", "cimages/multimages/16/5pe.png", "cimages/multimages/16/1473016381\_p5a.png"\], "height\_list": \[23, 22, 23, 41, 52\], "width\_list": \[240, 240, 240, 240, 240\], "conversations": \[{"from": "human", "value": "Let F = {2, 5, 7, 9}\\n\\nLet G = {1, 4, 6, 8}\\n\\nWhich of the following is true?\\nA. \\n\\n\\nB. /\\n\\n\\nC. /\\n\\n\\nD. /\\n\\n\\nE. /\\n\\n\\nAnswer with the option's letter from the given choices directly."}, {"from": "gpt", "value": "A"}\]} Here is the more readable version: { "id": 0, "image": \[\ "cimages/multimages/16/5pc.png",\ "cimages/multimages/16/5pd.png",\ "cimages/multimages/16/1602207874\_p5b.png",\ "cimages/multimages/16/5pe.png",\ "cimages/multimages/16/1473016381\_p5a.png"\ \], "height\_list": \[23, 22, 23, 41, 52\], "width\_list": \[240, 240, 240, 240, 240\], "conversations": \[\ {\ "from": "human",\ "value": "Let F = {2, 5, 7, 9}\\n\\nLet G = {1, 4, 6, 8}\\n\\nWhich of the following is true?\\nA. \\n\\n\\nB. /\\n\\n\\nC. /\\n\\n\\nD. /\\n\\n\\nE. /\\n\\n\\nAnswer with the option's letter from the given choices directly."\ },\ {\ "from": "gpt",\ "value": "A"\ }\ \] } Video Data[#](https://internvl.readthedocs.io/en/latest/get_started/chat_data_format.html#video-data "Link to this heading") ----------------------------------------------------------------------------------------------------------------------------- For video data, we use a JSONL file to store the data. Each entry is a dictionary organized in the following format. Note that each entry for video data must contain a `video` field, which is a string. The path in the `video` field is relative to the `root` field. Concatenating the `root` field and the `video` field gives the complete path to the video. { "id": 0, "video": "path/to/video.mp4", "conversations": \[\ {"from": "human", "value": "