How to add standard tests to an integration
Implementing standard tests
When creating either a custom class for yourself or a new tool to publish in a LangChain integration, it is important to add standard tests to ensure it works as expected. This guide will show you how to add standard tests to a tool, and the templates for implementing each different kind of integration are linked at the bottom.
Setup
First, let's install 2 dependencies:
langchain-corewill define the interfaces we want to import to define our custom tool.langchain-tests==0.3.0will provide the standard tests we want to use.
The langchain-tests package contains the module langchain_standard_tests. This name
mistmatch is due to this package historically being called langchain_standard_tests and
the name not being available on PyPi. This will either be reconciled by our
PEP 541 request (we welcome upvotes!),
or in a new release of langchain-tests.
Because added tests in new versions of langchain-tests will always break your CI/CD pipelines, we recommend pinning the
version of langchain-tests==0.3.0 to avoid unexpected changes.
%pip install -U langchain-core langchain-tests==0.3.0 pytest pytest-socket
Let's say we're publishing a package, langchain_parrot_link, that exposes a
tool called ParrotMultiplyTool:
from langchain_core.tools import BaseTool
class ParrotMultiplyTool(BaseTool):
name: str = "ParrotMultiplyTool"
description: str = (
"Multiply two numbers like a parrot. Parrots always add "
"eighty for their matey."
)
def _run(self, a: int, b: int) -> int:
return a * b + 80
And we'll assume you've structured your package the same way as the main LangChain packages:
/
├── langchain_parrot_link/
│ └── tools.py
└── tests/
├── unit_tests/
│ └── test_tools.py
└── integration_tests/
└── test_tools.py
Add and configure standard tests
There are 2 namespaces in the langchain-tests package:
- unit tests (
langchain_standard_tests.unit_tests): designed to be used to test the tool in isolation and without access to external services - integration tests (
langchain_standard_tests.integration_tests): designed to be used to test the tool with access to external services (in particular, the external service that the tool is designed to interact with).
Integration tests can also be run without access to external services, if they are properly mocked.
Both types of tests are implemented as pytest class-based test suites.
By subclassing the base classes for each type of standard test (see below), you get all of the standard tests for that type, and you can override the properties that the test suite uses to configure the tests.
Standard tools tests
Here's how you would configure the standard unit tests for the custom tool, e.g. in tests/test_tools.py:
from typing import Type
from langchain_parrot_link.tools import ParrotMultiplyTool
from langchain_standard_tests.unit_tests import ToolsUnitTests
class ParrotMultiplyToolUnitTests(ToolsUnitTests):
@property
def tool_constructor(self) -> Type[ParrotMultiplyTool]:
return ParrotMultiplyTool
def tool_constructor_params(self) -> dict:
# if your tool constructor instead required initialization arguments like
# `def __init__(self, some_arg: int):`, you would return those here
# as a dictionary, e.g.: `return {'some_arg': 42}`
return {}
def tool_invoke_params_example(self) -> dict:
"""
Returns a dictionary representing the "args" of an example tool call.
This should NOT be a ToolCall dict - i.e. it should not
have {"name", "id", "args"} keys.
"""
return {"a": 2, "b": 3}
from typing import Type
from langchain_parrot_link.tools import ParrotMultiplyTool
from langchain_standard_tests.integration_tests import ToolsIntegrationTests
class ParrotMultiplyToolIntegrationTests(ToolsIntegrationTests):
@property
def tool_constructor(self) -> Type[ParrotMultiplyTool]:
return ParrotMultiplyTool
def tool_constructor_params(self) -> dict:
# if your tool constructor instead required initialization arguments like
# `def __init__(self, some_arg: int):`, you would return those here
# as a dictionary, e.g.: `return {'some_arg': 42}`
return {}
def tool_invoke_params_example(self) -> dict:
"""
Returns a dictionary representing the "args" of an example tool call.
This should NOT be a ToolCall dict - i.e. it should not
have {"name", "id", "args"} keys.
"""
return {"a": 2, "b": 3}
and you would run these with the following commands from your project root
# run unit tests without network access
pytest --disable-socket --enable-unix-socket tests/unit_tests
# run integration tests
pytest tests/integration_tests
Standard test templates per component:
Above, we implement the unit and integration standard tests for a tool. Below are the templates for implementing the standard tests for each component:
Chat Models
from typing import Tuple, Type
from langchain_parrot_link.chat_models import ChatParrotLink
from langchain_standard_tests.unit_tests import ChatModelUnitTests
class ChatParrotLinkUnitTests(ChatModelUnitTests):
@property
def chat_model_class(self) -> Type[ChatParrotLink]:
return ChatParrotLink
from typing import Type
from langchain_parrot_link.chat_models import ChatParrotLink
from langchain_standard_tests.integration_tests import ChatModelIntegrationTests
class TestMistralStandard(ChatModelIntegrationTests):
@property
def chat_model_class(self) -> Type[ChatParrotLink]:
return ChatParrotLink
@property
def chat_model_params(self) -> dict:
return {"model": "bird-brain-001", "temperature": 0}