Scale API

Evaluation Datasets

evaluation_datasets

Methods

Approve Auto Generated Test Cases Batch ->
AutoGeneratedDraftTestCaseApproveBatchResponse
post/v4/evaluation-datasets/{evaluation_dataset_id}/approve-batch

Approve Auto Generated Test Cases Batch

Create Evaluation Dataset ->
EvaluationDataset
post/v4/evaluation-datasets

Description

Creates a evaluation dataset

Details

This API can be used to create a evaluation dataset. To use this API, review the request schema and pass in all fields that are required to create a evaluation dataset.

List Evaluation Datasets -> PageResponse<{ id, account_id, created_at, 11 more... }>
get/v4/evaluation-datasets

Description

Lists all evaluation datasets accessible to the user.

Details

This API can be used to list evaluation datasets. If a user has access to multiple accounts, all evaluation datasets from all accounts the user is associated with will be returned.

Publish Latest Evaluation Dataset Version ->
PublishEvaluationDatasetDraftResponse
post/v4/evaluation-datasets/{evaluation_dataset_id}/publish

Publish Latest Evaluation Dataset Version

Delete Evaluation Dataset ->
GenericDeleteResponse
delete/v4/evaluation-datasets/{evaluation_dataset_id}

Description

Deletes the dataset, and all other entities associated with the dataset, such as test cases, evaluations and results.

Details

This is a permanent and destructive action that cannot be undone.

Get Evaluation Dataset -> { id, account_id, created_at, 11 more... }
get/v4/evaluation-datasets/{evaluation_dataset_id}

Description

Gets the details of a dataset.

Update Evaluation Dataset ->
EvaluationDataset
patch/v4/evaluation-datasets/{evaluation_dataset_id}

Description

Updates a evaluation dataset

Details

This API can be used to update the evaluation dataset that matches the ID that was passed in as a path parameter. To use this API, pass in the id that was returned from your Create Evaluation Dataset API call as a path parameter.

Review the request schema to see the fields that can be updated.

Domain types

AutoGeneratedDraftTestCaseApproveBatchResponse = { autogenerated_draft_test_cases, success }
AutoGeneratedDraftTestCaseMissingChunkInfo = { artifact_content_modification_identifier, artifact_id, artifact_name, 1 more... }
EvaluationDataset = { id, account_id, created_at, 10 more... }
PaginatedEvaluationDatasets = { current_page, items, items_per_page, 1 more... }
PublishEvaluationDatasetDraftResponse = { autogenerated_draft_test_cases, success }
Evaluation Datasets

Autogenerated Draft Test Cases

evaluation_datasets.autogenerated_draft_test_cases

Methods

Approve Autogenerated Draft Test Case ->
ApproveAutoGeneratedDraftTestCaseResponse
post/v4/evaluation-datasets/{evaluation_dataset_id}/autogenerated-draft-test-cases/{autogenerated_draft_test_case_id}/approve

Approve Autogenerated Draft Test Case

Create Autogenerated Draft Test Case ->
AutoGeneratedDraftTestCase
post/v4/evaluation-datasets/{evaluation_dataset_id}/autogenerated-draft-test-cases

Description

Creates a autogenerated draft test case

Details

This API can be used to create a autogenerated draft test case. To use this API, review the request schema and pass in all fields that are required to create a autogenerated draft test case.

Delete Autogenerated Draft Test Case ->
GenericDeleteResponse
delete/v4/evaluation-datasets/{evaluation_dataset_id}/autogenerated-draft-test-cases/{autogenerated_draft_test_case_id}

Description

Delete a specified draft test case. This version will not be retained.

List Autogenerated Draft Test Cases -> PageResponse<{ id, account_id, approved, 9 more... }>
get/v4/evaluation-datasets/{evaluation_dataset_id}/autogenerated-draft-test-cases

List Autogenerated Draft Test Cases

Get Autogenerated Draft Test Case ->
AutoGeneratedDraftTestCase
get/v4/evaluation-datasets/{evaluation_dataset_id}/autogenerated-draft-test-cases/{autogenerated_draft_test_case_id}

Description

Get the test case by its id.

Update Autogenerated Draft Test Case ->
AutoGeneratedDraftTestCase
patch/v4/evaluation-datasets/{evaluation_dataset_id}/autogenerated-draft-test-cases/{autogenerated_draft_test_case_id}

Description

Update a test case draft.

Domain types

ApproveAutoGeneratedDraftTestCaseResponse = { autogenerated_draft_test_case_id, failed_chunks, success }
AutoGeneratedDraftTestCase = { id, account_id, approved, 8 more... }
AutoGeneratedDraftTestCasesList = { current_page, items, items_per_page, 1 more... }
Evaluation Datasets

Evaluation Dataset Versions

evaluation_datasets.evaluation_dataset_versions

Methods

Create Evaluation Dataset Version ->
EvaluationDatasetVersion
post/v4/evaluation-datasets/{evaluation_dataset_id}/evaluation-dataset-versions

Description

Creates a evaluation dataset version

Details

This API can be used to create a evaluation dataset version. To use this API, review the request schema and pass in all fields that are required to create a evaluation dataset version.

List Evaluation Dataset Versions -> PageResponse<
EvaluationDatasetVersion
>
get/v4/evaluation-datasets/{evaluation_dataset_id}/evaluation-dataset-versions

Description

Lists all evaluation dataset versions accessible to the user.

Details

This API can be used to list evaluation dataset versions. If a user has access to multiple accounts, all evaluation dataset versions from all accounts the user is associated with will be returned.

Publish Evaluation Dataset Version ->
PublishEvaluationDatasetDraftResponse
post/v4/evaluation-datasets/{evaluation_dataset_id}/evaluation-dataset-versions/{evaluation_dataset_version_id}/publish

Publish Evaluation Dataset Version

Get Evaluation Dataset Version ->
EvaluationDatasetVersion
get/v4/evaluation-datasets/{evaluation_dataset_id}/evaluation-dataset-versions/{evaluation_dataset_version_id}

Description

Gets the details of a evaluation dataset version

Details

This API can be used to get information about a single evaluation dataset version by ID. To use this API, pass in the id that was returned from your Create Evaluation Dataset Version API call as a path parameter.

Review the response schema to see the fields that will be returned.

Domain types

EvaluationDatasetVersion = { id, account_id, created_at, 6 more... }
PaginatedEvaluationDatasetVersions = { current_page, items, items_per_page, 1 more... }
Evaluation Datasets

Generation Jobs

evaluation_datasets.generation_jobs

Methods

Cancel Evaluation Dataset Generation Job -> unknown
post/v4/evaluation-datasets/{evaluation_dataset_id}/generation-jobs/{generation_job_id}/cancel

Cancel Evaluation Dataset Generation Job

Create Evaluation Dataset Generation Job ->
EvaluationDatasetGenerationJobResponse
post/v4/evaluation-datasets/{evaluation_dataset_id}/generation-jobs

Create Evaluation Dataset Generation Job

Get Evaluation Dataset Generation Jobs -> GenerationJobsPagination<
EvaluationDatasetGenerationJob
>
get/v4/evaluation-datasets/{evaluation_dataset_id}/generation-jobs

Get Evaluation Dataset Generation Jobs

Get Evaluation Dataset Generation Job ->
EvaluationDatasetGenerationJob
get/v4/evaluation-datasets/{evaluation_dataset_id}/generation-jobs/{generation_job_id}

Get Evaluation Dataset Generation Job

Domain types

EvaluationDatasetGenerationJob = { created_at, generation_job_id, status, 5 more... }
EvaluationDatasetGenerationJobResponse = { created_at, generation_job_id, status, 5 more... }
EvaluationDatasetGenerationJobsList = { generation_jobs }
Evaluation Datasets

Test Cases

evaluation_datasets.test_cases

Methods

Batch Create Test Cases -> Array<
TestCase
>
post/v4/evaluation-datasets/{evaluation_dataset_id}/test-cases/batch

Description

Creates a batch of test cases

Details

This API can be used to create multiple test cases so users do not have to the incur the cost of repeated network calls. To use this API, pass in a list of test cases in the request body.

Create Test Case ->
TestCase
post/v4/evaluation-datasets/{evaluation_dataset_id}/test-cases

Description

Create a test case for a selected dataset.

Delete Test Case ->
GenericDeleteResponse
delete/v4/evaluation-datasets/{evaluation_dataset_id}/test-cases/{test_case_id}

Description

Delete a specified test case -- retaining older version belonging to older dataset versions.

Details

This request shouldn't be used when deleting due to compliance reasons. See Wipe action.

List Test Cases -> PageResponse<
TestCase
>
get/v4/evaluation-datasets/{evaluation_dataset_id}/test-cases

Description

List all test cases for a selected dataset.

Get Test Case ->
TestCase
get/v4/evaluation-datasets/{evaluation_dataset_id}/test-cases/{test_case_id}

Description

Get the test case by its id.

Update Test Case ->
TestCase
patch/v4/evaluation-datasets/{evaluation_dataset_id}/test-cases/{test_case_id}

Description

Update a test case.

Domain types

ArtifactSchemaGeneration = { artifact_ids_filter, input, expected_extra_info, 1 more... }
FlexibleChunk = { text, metadata }
FlexibleMessage = { content, role } | { content, role } | { content, role }
FlexibleTestCaseSchema = { input, expected_extra_info, expected_output }
GenerationTestCaseSchema = { input, expected_extra_info, expected_output }
PaginatedTestCases = { current_page, items, items_per_page, 1 more... }
TestCase = { id, account_id, autogenerated, 8 more... } | { id, account_id, autogenerated, 8 more... }
Evaluation DatasetsTest Cases

History

evaluation_datasets.test_cases.history

Methods

Wipe Full History Of Test Cases ->
GenericDeleteResponse
delete/v4/evaluation-datasets/{evaluation_dataset_id}/test-cases/{test_case_id}/history

Description

Delete a specified test case and remove all its history from past dataset versions too.

List Historical Test Cases -> PageResponse<
TestCase
>
get/v4/evaluation-datasets/{evaluation_dataset_id}/test-cases/history/{num}

Description

List all test cases for a specified dataset version.

Get Historical Test Case ->
TestCase
get/v4/evaluation-datasets/{evaluation_dataset_id}/test-cases/{test_case_id}/history/{num}

Description

Get the test case by its id for a specified dataset version.