Tutorial

The Toolkit Component

In the previous chapter, I have shown a Python program to manually exchange messages with an LLM API. I introduced a data structure that contains inputs and outputs to interact with the LLM.

In this chapter, I introduce the Toolkit component. The Toolkit contains definitions for function tools for the LLM API. But it does not explicitly perform a tool call (that will be done in later chapters).

Here are my two goals for this chapter. The Toolkit, that I implement, must:

  • define a Python REPL tool call for an LLM API,
  • export all tools as a list that is ready to be sent to an LLM API.

Implementing the Toolkit

I will step back for a moment and consider how do LLMs use tools. It is useful to keep that in mind while implementing the toolkit.

How do LLMs Use Tools?

Recall that the LLM API accepts a tools field in the JSON input data. When you provide the API tools, the server constructs a special system prompt. The prompt is designed to instruct the model to use the specified tool(s).

For example, here is the Claude API example prompt that is constructed for tool use.

In this environment you have access to a set of tools you can use to answer the
user's question.
{{ FORMATTING INSTRUCTIONS }}
String and scalar parameters should be specified as is, while lists and objects
should use JSON format. Note that spaces for string values are not stripped.
The output is not expected to be valid XML and is parsed with regular
expressions.
Here are the functions available in JSONSchema format:
{{ TOOL DEFINITIONS IN JSON SCHEMA }}
{{ USER SYSTEM PROMPT }}
{{ TOOL CONFIGURATION }}

Setup

mkdir infer_tk && cd infer_tk
python3 -m venv venv
source venv/bin/activate
pip3 install requests ipython
ipython

Starting with the Class

The basic Toolkit I will implement will not be a function, but a class. This is because it has state. (Though, functions can have state, but that is not on the agenda here.)

To store state, the Toolkit class keeps a variable named table.

In [1]: class Toolkit:
   ...:     def __init__(self):
   ...:         self.table = {}
   ...: 

In [2]: tk = Toolkit()

In [3]: tk.table
Out[3]: {}

Variable table is a Python dictionary. Inside of it, will be tool definitions for the LLM API. But, first, I have to recall the schemas for those definitions.

API Tools Input Schema

Recall the tools schema for the OpenAI API.

{
    ...
    "tools": [ properties ...]
    ...
}

The properties schema contains a definition of one function tool.

{
    "name": string,
    "type": "function",
    "description": string,
    "parameters": parameters
}

The parameters schema contains the definitions for all arguments.

{
    "type": "object",
    "properties": {
        arg: {
            "type": string,
            "description": string
        }, ...
    },
    "required": [ strings ... ]
}

The objects placed in the toolkit table shall follow the schemas for:

  • properties, and
  • parameters.

Defining Tools

I wish for a method to add new tools to the Toolkit.

In [5]: class Toolkit:
   ...:     def __init__(self):
   ...:         self.table = {}
   ...:     def deftool(self, name, description, parameters):
   ...:         if name in self.table:
   ...:             raise ValueError(f"Tool '{name}' already defined.")
   ...:         self.table[name] = {
   ...:             "name": name,
   ...:             "type": "function",
   ...:             "description": description,
   ...:             "parameters": parameters
   ...:         }
   ...: 

In [6]: tk = Toolkit()

In [7]: tk.deftool("name0", "desc0", {})

In [8]: tk.table
Out[8]: 
{'name0': {'name': 'name0',
  'type': 'function',
  'description': 'desc0',
  'parameters': {}}}

That is weird. Maybe I need to define a variable and set it’s value into the dictionary? Also it seems better to return the result.

In [11]: class Toolkit:
    ...:     def __init__(self):
    ...:         self.table = {}
    ...:     def deftool(self, name, description, parameters):
    ...:         if name in self.table:
    ...:             raise ValueError(f"Tool '{name}' already defined.")
    ...:         r = {
    ...:             "name": name,
    ...:             "type": "function",
    ...:             "description": description,
    ...:             "parameters": parameters
    ...:         }
    ...:         self.table[name] = r
    ...:         return r
    ...: 

In [12]: tk = Toolkit()

In [13]: tk.deftool("name0", "desc0", {})
Out[13]: 
{'name': 'name0', 
 'type': 'function',  
 'description': 'desc0', 
 'parameters': {}}

That worked.

Defining an Evaluator Tool

The requirement is an evaluator tool. A Python REPL for the LLM.

In [21]: def add_py_repl_tool(toolkit):
    ...:     p = {
    ...:         "type": "object",
    ...:         "properties": {
    ...:             "code": {
    ...:                 "type": "string",
    ...:                 "description": "Python code to evaluate in the REPL."
    ...:             }
    ...:         },
    ...:         "required": ["code"]
    ...:     }
    ...:     d = (
    ...:         "Evaluate Python code in a persistent Python REPL. "
    ...:         "Accepts code. Returns stdout, stderr, and result."
    ...:     )
    ...:     return toolkit.deftool("py_repl_runsource", d, p)
    ...: 

In [22]: tk = Toolkit()

In [23]: add_py_repl_tool(tk)
Out[23]: 
{'name': 'py_repl_runsource',
 'type': 'function',
 'description': 'Evaluate Python code in a persistent Python REPL.
Accepts code. Returns stdout, stderr, and result.',
 'parameters': {'type': 'object',
  'properties': {'code': {'type': 'string',
    'description': 'Python code to evaluate in the REPL.'}},
  'required': ['code']}}

Converting to Tools

The second requirement is that the tools is a list.

This means that the tools dictionary, inside the Toolkit, must be converted to a list.

In [24]: class Toolkit:
    ...:     def __init__(self):
    ...:         self.table = {}
    ...:     def deftool(self, name, description, parameters):
    ...:         if name in self.table:
    ...:             raise ValueError(f"Tool '{name}' already defined.")
    ...:         r = {
    ...:             "name": name,
    ...:             "type": "function",
    ...:             "description": description,
    ...:             "parameters": parameters
    ...:         }
    ...:         self.table[name] = r
    ...:         return r
    ...:     def tools(self):
    ...:         return [x for x in self.table.values()]
    ...:     def match(self, name):
    ...:         if name in self.table:
    ...:             return name
    ...:         else:
    ...:             return False
    ...: 

In [25]: tk = Toolkit()

In [26]: add_py_repl_tool(tk)
Out[26]: 
{'name': 'py_repl_runsource',
 'type': 'function',
 'description': 'Evaluate Python code in a persistent Python REPL. Accepts code.
 Returns stdout, stderr, and result.',
 'parameters': {'type': 'object',
  'properties': {'code': {'type': 'string',
    'description': 'Python code to evaluate in the REPL.'}},
  'required': ['code']}}

In [27]: tk.tools()
Out[27]: 
[{'name': 'py_repl_runsource',
  'type': 'function',
  'description': 'Evaluate Python code in a persistent Python REPL. Accepts
  code. Returns stdout, stderr, and result.',
  'parameters': {'type': 'object',
   'properties': {'code': {'type': 'string',
     'description': 'Python code to evaluate in the REPL.'}},
   'required': ['code']}}]

In [28]: tk.match("py_repl_runsource")
Out[28]: 'py_repl_runsource'

In [29]: tk.match("must be False")
Out[29]: False

Toolkit Code

class Toolkit:
    def __init__(self):
        self.table = {}

    def deftool(self, name, description, parameters):
        if name in self.table:
            raise ValueError(f"Tool '{name}' already defined.")
        r = {
            "name": name,
            "type": "function",
            "description": description,
            "parameters": parameters
        }
        self.table[name] = r
        return r

    def tools(self):
        return [x for x in self.table.values()]

    def match(self, name):
        if name in self.table:
            return name
        else:
            return False

def add_py_repl_tool(toolkit):
    p = {
        "type": "object",
        "properties": {
            "code": {
                "type": "string",
                "description": "Python code to evaluate in the REPL."
            }
        },
        "required": ["code"]
    }
    d = (
        "Evaluate Python code in a persistent Python REPL. "
        "Accepts code. Returns stdout, stderr, and result as one string."
    )
    return toolkit.deftool("py_repl_runsource", d, p)