Scale API | Knowledge Bases

Knowledge Bases

knowledge_bases

Methods

Create Knowledge Base ->

CreateKnowledgeBaseResponse

post/v4/knowledge-bases

Description

Creates an EGP knowledge base.

Details

A knowledge base is a storage device for all data that needs to be accessible to EGP models. Users can upload data from a variety of data sources into a knowledge base, and then query the knowledge base for chunks that are semantically relevant to the query.

Every knowledge base must be associated with a fixed embedding model. This embedding model will be used to embed all data that is stored in the knowledge base. The embedding model cannot be changed once the knowledge base is created. Only the embedding models in the dropdown menu below are supported.

Differences from V1

V1 data ingestion consisted of knowledge bases, vector stores, and data connectors. V1 Knowledge bases interacted with natural language, V1 vector stores interacted with chunks and embeddings, and V1 data connectors set up automatic ingestion pipelines with third party data sources.
In V2, all data ingestion is done through knowledge bases. Low level configuration such as chunking strategies and data sources are now handled by this unified knowledge base v2 upload API.
The way data is stores in V2 allows for better observability on the ingestion progress and content of the knowledge base.
Reliability and scalability is also improved via distributed temporal workflows.

Backwards Compatibility

V2 and V1 Knowledge Bases are entirely separate and not backwards compatible. Users who have existing V1 knowledge bases will need to migrate their data to V2 knowledge bases.

Delete Knowledge Base ->

DeleteKnowledgeBaseResponse

delete/v4/knowledge-bases/{knowledge_base_id}

Description

Deletes a knowledge base.

Details

This API can be used to delete a knowledge base by ID. To use this API, pass in the knowledge_base_id that was returned from your Create Knowledge Base API call as a path parameter.

Backwards Compatibility

V2 and V1 Knowledge Bases are entirely separate and not backwards compatible. Users who have existing V1 knowledge bases will need to migrate their data to V2 knowledge bases.

List Knowledge Bases -> PageResponse<

KnowledgeBase

get/v4/knowledge-bases

Description

Lists all knowledge bases owned by the authorized user.

Details

This API can be used to list all knowledge bases that have been created by the user. This API will return the details of all knowledge bases including their IDs, names, the embedding models they use, any metadata associated with the knowledge bases, and the timestamps for their creation, last-updated time.

Backwards Compatibility

V2 and V1 Knowledge Bases are entirely separate and not backwards compatible. Users who have existing V1 knowledge bases will need to migrate their data to V2 knowledge bases.

Query Relevant Chunks -> { chunks, completed_at, request_id, 1 more... }

post/v4/knowledge-bases/{knowledge_base_id}/query

Description

Query a knowledge base for text chunks that are most semantically relevant to the query.

Details

Given a query expressed as an embedding, this API runs a similarity search amongst the embeddings indexed in the knowledge base to find the most relevant chunk embeddings. To use this API, specify the knowledge_base_id of the knowledge base you want to query, pass in the natural language query that you want to search for, specify the value top_k, which is the number of similar chunks that will be returned, and specify whether you want the returned chunks to include_embeddings.

Similarity search is used to efficiently find, retrieve, and rank chunks based on their similarity to a given query, which is also expressed as an embedding. Similarity scores ( using the cosine similarity metric) are calculated between each chunk embedding and the embedded query, and the chunks are ranked based on similarity score. The top-ranked chunks are returned as the query results.

We are using the Hierarchical Navigable Small World (HNSW) algorithm to perform a k nearest neighbors search in the vector space. This algorithm returns an estimate of the best k nearest neighbors and is optimized for datasets with hundreds of thousands of vectors. You can read more about the specifics of this algorithm here.

Backwards Compatibility

V2 and V1 Knowledge Bases are entirely separate and not backwards compatible. Users who have existing V1 knowledge bases will need to migrate their data to V2 knowledge bases.

Get Knowledge Base ->

KnowledgeBase

get/v4/knowledge-bases/{knowledge_base_id}

Description

Gets the details of a knowledge base.

Details

This API can be used to get information about a single knowledge base by ID. To use this API, pass in the knowledge_base_id that was returned from your Create Knowledge Base API call as a path parameter.

This API will return the details of a knowledge base including its ID, name, the embedding model it uses, any metadata associated with the knowledge base, and the timestamps for its creation, last-updated time.

Backwards Compatibility

V2 and V1 Knowledge Bases are entirely separate and not backwards compatible. Users who have existing V1 knowledge bases will need to migrate their data to V2 knowledge bases.

Patch Knowledge Base -> { knowledge_base_name, metadata }

patch/v4/knowledge-bases/{knowledge_base_id}

Patch Knowledge Base

Submit Upload Job With Local Files ->

CreateKnowledgeBaseUploadsFromFilesResponse

post/v4/knowledge-bases/{knowledge_base_id}/upload_files

Submit Upload Job with local files

Domain types

CharacterChunkingStrategyConfig = { strategy, chunk_overlap, chunk_size, 1 more... }

ChunksResponse = { chunks }

CreateKnowledgeBaseResponse = { knowledge_base_id }

CreateKnowledgeBaseUploadsFromFilesResponse = { upload_ids }

CustomChunkingStrategyConfig = { endpoint, strategy, params }

DeleteKnowledgeBaseResponse = { deleted, knowledge_base_id }

KnowledgeBase = { created_at, embedding_config, knowledge_base_id, 8 more... }

PaginatedKnowledgeBaseUploads = { current_page, items, items_per_page, 1 more... }

PaginatedKnowledgeBases = { current_page, items, items_per_page, 1 more... }

TokenChunkingStrategyConfig = { strategy, chunk_overlap, max_chunk_size, 2 more... }

Knowledge Bases

Artifacts

knowledge_bases.artifacts

Methods

Batch Delete Locally Stored Artifacts -> { artifact_ids }

post/v4/knowledge-bases/{knowledge_base_id}/artifacts/batch-delete

Batch Delete Locally Stored Artifacts

Delete Locally Stored Artifact -> { artifact_id, deleted }

delete/v4/knowledge-bases/{knowledge_base_id}/artifacts/{artifact_id}

Delete Locally Stored Artifact

List Tracked Artifacts -> PageResponse<{ artifact_id, artifact_name, artifact_uri, 8 more... }>

get/v4/knowledge-bases/{knowledge_base_id}/artifacts

Description

List all artifacts tracked by a knowledge base.

Details

This API can be used to list all artifacts that are currently tracked in a knowledge base. This API will return the details of all artifacts including their IDs, names, the source they originated from, their current upload statuses, and the timestamps for their creation and last-updated time.

This list should be consistent with the state of the data source at the time of start of the latest upload. If the state is not consistent, create a new upload to update the knowledge base to reflect the latest state of the data source.

Get Tracked Artifact Details ->

Artifact

get/v4/knowledge-bases/{knowledge_base_id}/artifacts/{artifact_id}

Description

Gets the details of an artifact tracked by a knowledge base.

Details

This API can be used to get information about a single artifact by ID. This response will contain much more detail about the artifact than show in the List Artifacts API call. To use this API, pass in the knowledge_base_id and artifact_id that were returned from your List Artifacts API call as path parameters.

Compatibility with V1

V2 and V1 Knowledge Bases are entirely separate and not backwards compatible. Users who have existing V1 knowledge bases will need to migrate their data to V2 knowledge bases.

Patch Artifact Information -> { artifact_name, artifact_uri, created_at, 12 more... }

patch/v4/knowledge-bases/{knowledge_base_id}/artifacts/{artifact_id}

Patch Artifact Information

Domain types

Artifact = { artifact_id, artifact_name, artifact_uri, 8 more... }

PaginatedArtifacts = { current_page, items, items_per_page, 1 more... }

Knowledge Bases

Async Jobs

knowledge_bases.async_jobs

Methods

List Upload Jobs -> TopLevelArray<{ id, account_id, created_at, 7 more... }>

get/v4/knowledge-bases/{knowledge_base_id}/async-jobs

List Upload Jobs

Knowledge Bases

Chunks

knowledge_bases.chunks

Methods

Get Chunks -> ChunkPagination<{ chunk_id, text, attachment_url, 4 more... }>

get/v4/knowledge-bases/{knowledge_base_id}/chunks

Description

Get chunks from a knowledge base using chunk IDs or a matching metadata field

### Details
This API can be used to get a list of chunks from a knowledge base. Given a chunk id,             a metadata field and value, or both, matching chunks are searched for in the knowledge base             given by knowledge base id.

Knowledge Bases

Data Source Connections

knowledge_bases.data_source_connections

Methods

Delete Knowledge Base Data Source Connection ->

DeleteKnowledgeBaseDataSourceConnectionResponse

post/v4/knowledge-bases/{knowledge_base_id}/data-source-connections/{knowledge_base_data_source_id}/delete

Delete Knowledge Base Data Source Connection

Domain types

DeleteKnowledgeBaseDataSourceConnectionResponse = { artifacts_to_delete, deletion_workflow_id }

Knowledge Bases

Upload Schedules

knowledge_bases.upload_schedules

Methods

Create Upload Schedule ->

UploadSchedule

post/v4/knowledge-bases/{knowledge_base_id}/upload-schedules

Description

Creates a upload schedule

Details

This API can be used to create a upload schedule. To use this API, review the request schema and pass in all fields that are required to create a upload schedule.

Delete Upload Schedule ->

GenericDeleteResponse

delete/v4/knowledge-bases/{knowledge_base_id}/upload-schedules/{upload_schedule_id}

Description

Deletes a upload schedule

Details

This API can be used to delete a upload schedule by ID. To use this API, pass in the id that was returned from your Create Upload Schedule API call as a path parameter.

List Upload Schedules -> PageResponse<

UploadScheduleWithViews

get/v4/knowledge-bases/{knowledge_base_id}/upload-schedules

Description

Lists all upload schedules accessible to the user.

Details

This API can be used to list upload schedules. If a user has access to multiple accounts, all upload schedules from all accounts the user is associated with will be returned.

Get Upload Schedule ->

UploadScheduleWithViews

get/v4/knowledge-bases/{knowledge_base_id}/upload-schedules/{upload_schedule_id}

Description

Gets the details of a upload schedule

Details

This API can be used to get information about a single upload schedule by ID. To use this API, pass in the id that was returned from your Create Upload Schedule API call as a path parameter.

Review the response schema to see the fields that will be returned.

Update Upload Schedule ->

UploadSchedule

patch/v4/knowledge-bases/{knowledge_base_id}/upload-schedules/{upload_schedule_id}

Description

Updates a upload schedule

Details

This API can be used to update the upload schedule that matches the ID that was passed in as a path parameter. To use this API, pass in the id that was returned from your Create Upload Schedule API call as a path parameter.

Review the request schema to see the fields that can be updated.

Domain types

PaginatedUploadScheduleWithViews = { current_page, items, items_per_page, 1 more... }

UploadSchedule = { id, account_id, chunking_strategy_config, 9 more... }

UploadScheduleWithViews = { id, account_id, chunking_strategy_config, 10 more... }

Knowledge Bases

Uploads

knowledge_bases.uploads

Methods

Cancel Upload Job ->

CancelKnowledgeBaseUploadResponse

post/v4/knowledge-bases/{knowledge_base_id}/uploads/{upload_id}/cancel

Cancel Upload Job

Submit Upload Job ->

CreateKnowledgeBaseUploadResponse

post/v4/knowledge-bases/{knowledge_base_id}/uploads

Submit Upload Job

List Upload Jobs -> PageResponse<{ id, data_source_config, knowledge_base_id, 7 more... }>

get/v4/knowledge-bases/{knowledge_base_id}/uploads

Description

List all uploads for a knowledge base.

Details

This API can be used to list all uploads that have been created for a knowledge base. This API will return the details of all uploads including their IDs, their statuses, the data source configs they use, the chunking strategy configs they use, and the timestamps for their creation and last-updated time.

Backwards Compatibility

V2 and V1 Knowledge Bases are entirely separate and not backwards compatible. Users who have existing V1 knowledge bases will need to migrate their data to V2 knowledge bases.

Get Upload Job ->

KnowledgeBaseUpload

get/v4/knowledge-bases/{knowledge_base_id}/uploads/{upload_id}

Get Upload Job

Domain types

CancelKnowledgeBaseUploadResponse = { canceled, upload_id }

CreateKnowledgeBaseUploadResponse = { upload_id }

KnowledgeBaseUpload = { artifacts, artifacts_status, created_at, 7 more... }