Refraction Home

Refraction is a low-cost (no LLMs!), low-latency, domain-agnostic, data-agnostic, model-agnostic approach towards validation and repair for a sequence of tool calls, based on classical AI planning techniques.

The process of refraction accepts an LLM response and makes appropriate edits to it before passing it off to a downstream applications (in contrast to reflection, where we send back the response for the LLM to reason with again).

Compared to the reasoning step, this process must be:

1. Quick, in relation to the reasoning step itself; and
2. With certain guarantees, since we are passing on the input to downstream applications.

In the context of agentic systems, refraction appears as a middleware component to be executed between the generation and the execution of a tool call, as well as offline to improve model performance based on cached refraction outputs.

Pre-call and sequence-level verification

The refraction API allows an agentic system to validate a tool call or a sequence of tool calls against a reference specification and memory. The whole process works on two simple but key ideas:

💡 A large section of issues with tool calls is already identifiable from the specification of the tool (and an operating memory in the context of a sequential execution of multiple tools) without trying to execute it, or sending it back again to a large model for reflection.

💡 Planning and validating at the sequence level, while not necessary, significantly saves on costs, latency, and on occasions, accuracy of agentic systems rather than doing so one step at a time.

Debugging as an optimization process

To find out if there are any issues with a sequence of tool calls, we cast the debugging task into an optimization process. Consider the following sequence.

var2 = SkyScrapperSearchAirport(query="London")
var3 = SkyScrapperFlightSearch(originSkyId="$var1.skyId$", ..., date="2024-08-15")

We can extract a series of tokens describing the following different aspects of this sequence:

1. SkyScrapperSearchAirport is called with the parameter query.
2. The parameter query has been assigned to "London" which has, presumably, come from the user.
3. The output of the call to SkyScrapperSearchAirport is assigned to var2.
4. SkyScrapperFlightSearch is called with parameters originSkyId, ..., date.
5. The parameter originSkyId is assigned to the key skyId of var1 produced previously in the sequence.
6. And so on so forth.

💡 Given a sequence of API calls, we extract all such tokens. Then the task of the optimizer is: Given the specification of those APIs, enforce as many of the tokens as possible.

In order to achieve this, we use your favorite NL2Flow package to check the extracted tokens for soundness i.e. if the tokens represent a valid sequence of API calls. Check the validator API of NL2Flow for more details on this.

Example

Here is an example use case from the NESTFUL dataset:

User: Find flights from New York to London that depart on August 15, 2024, and return on August 18, 2024 and find hotels in London.

var1 = SkyScrapperSearchAirport(query="New York")
var2 = SkyScrapperSearchAirport(query="London")
var3 = SkyScrapperFlightSearch(originSkyId="$var1.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
var4 = TripadvisorSearchLocation(query="London")
var5 = TripadvisorSearchHotels(geoId="$var4.geoId$", checkIn="2024-08-15", checkOut="2024-08-18")

We have introduced several errors in the previous example before making the call above. This includes, wrong parameters, wrong assignment to parameters, wrong api names, missing variables, etc.

from data.data_handler import get_nestful_catalog
from refraction import refract

catalog = get_nestful_catalog(executable=True)

sequence = [
    'var1 = SkyCrapperSearchAirport(query="New York")',
    'var2 = SkyScrapperSearchAirport(query="London")',
    'var3 = SkyScrapperFlightSearch(originalSkyId="$var1.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$")',
    'var45 = TripadvisorSearchLocation(query="London")',
    'var5 = TripadvisorSearchHotels(geoId="$var4.ID$", checkIn="2024-08-15", checkOut="2024-08-18")',
]

refract(sequence, catalog)

Here is a pretty print of a sample response from the refraction process (it also provides a structured response) on the example above. Notice how different edits have been made to the input sequence. This includes fixes to parameter names, parameter assignments, API names, output variables, missing function calls, and so on.

+ var7 = NewsAPISearchByKeyWord()
- var1 = SkyCrapperSearchAirport(query="New York")
?    ^      ^

+ var2 = SkyScrapperSearchAirport(query="New York")
?    ^      ^^

- var3 = SkyScrapperFlightSearch(originalSkyId="$var1.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$")
?                                      --           ^                                                               ^

+ var3 = SkyScrapperFlightSearch(originSkyId="$var2.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var2.entityId$", destinationEntityId="$var2.entityId$", date="$var7.date$")
?                                                 ^                                                               ^                                                  ++++++++++++++++++++

  var45 = TripadvisorSearchLocation(query="London")
- var5 = TripadvisorSearchHotels(geoId="$var4.ID$", checkIn="2024-08-15", checkOut="2024-08-18")
?                                              ^

+ var5 = TripadvisorSearchHotels(geoId="$var45.geoId$", checkIn="2024-08-15", checkOut="2024-08-18")
?                                            + +++ ^

Next Steps

1. Continue reading on to the next chapter to find out more on the categories of errors handled by this process and the different recovery patterns for different types of errors.

2. Head over to the refraction API on detailed usage instructions.