JSONL Format

The Vantage JSONL format is used to bulk upload your data via the Console UI and via the direct upload links available in the REST API. Additionally, the format is also used for the Upload Documents API.

๐Ÿ‘

Vantage Ingestion Format

The schema and format of the documents for ingestion.

Required Fields

  • id: Physical type is BYTE_ARRAY, logical type is String
  • text: Physical type is BYTE_ARRAY, logical type is String
  • embeddings: Physical type is a list of DOUBLE

๐Ÿ“˜

Typically you'd have only one of text or embeddings.

For more details please check Vantage Documents page.

JSONL Documents

Examples of correctly prepared JSONL data are below:

{"id": "1", "text": "Example text", "meta_color": "green"}\n
{"id": "2", "text": "Sample text", "meta_color": "blue"}
{"id": "1", "text": "Example text", "meta_color": "green", "embeddings": [1,2,3, ...]}\n
{"id": "2", "text": "Sample text", "meta_color": "blue", "embeddings": [4,5,6, ...]}

Note: meta_color field is optional.

When you have your data you can use Vantage Python SDK to upload it easily:

from vantage_sdk import VantageClient

vantage_instance = VantageClient.using_vantage_api_key(
    vantage_api_key=VANTAGE_API_KEY,
    account_id=ACCOUNT_ID,
)

# JSONL data from above 
documents = '{
  "id": "1", 
  "text": "Example text", 
  "meta_color": "green", 
  "embeddings": [1,2,3, ...]
}\\n{
  "id": "2", 
  "text": "Sample text", 
  "meta_color": "blue",
  "embeddings": [4,5,6, ...]
}'

vantage_instance.upsert_documents_from_jsonl_string(
    collection_id="example-collection",
    documents_jsonl=documents
)
from vantage_sdk import VantageClient

vantage_instance = VantageClient.using_vantage_api_key(
    vantage_api_key=VANTAGE_API_KEY,
    account_id=ACCOUNT_ID,
)

# Proper JSONL data written in file
documents_path = "my_documents.jsonl"

vantage_instance.upsert_documents_from_jsonl_file(
    collection_id="example-collection",
    jsonl_file_path=documents_path
)