Skip to content

Refraction Examples

We started by saying that one of the key ideas for refraction is that a large section of issues with tool calls is already identifiable from the specification of the tool (and an operating memory in the context of a sequential execution of multiple tools) without trying to execute it, or sending it back again to a large model for reflection.

This means that there are certain things it can do, and there are many things it cannot. Depending on your needs, you can make a decision based on the following.

What it does

If it is in the input spec, we will try to cover it -- we are in the process of implementing full coverage. This means common issues with parameters, memory references, tool names, etc. as we detail below.

  • [x] Made up API
  • [x] Wrong label
  • [x] Missing label
  • [x] Missing input parameter
  • [x] Made up input parameter
  • [x] Made up input parameter assignment to variable
  • [x] Made up input parameter assignment to property
  • [x] Made up assignment recovery with ask
  • [x] Made up assignment recovery with mapping
  • [x] Made up assignment recovery with function call
  • [x] Missing step
  • [ ] Type mismatch
  • [ ] Inadmissible assignments to enum inputs
  • [x] Optional items in the parameters
  • [x] Ordering of parameters
  • [ ] Transformation of parameters

What it doesn't do

This also means that we DO NOT at all consider issues with the values assigned to parameters. In other words, a mistaken tool call with an issue with, for example, the format of a date string assigned to a date input, is out of scope. Only an LLM can fix those issues through a traditional reflection approach.

1.1 Categories of errors 🐞

In this page, we will cover some common mistakes in a sequence of API calls (generated from an LLM) and the result of refraction on each of them, in isolation. The cost model of the debugging API determines the response when many such errors can encountered simultaneously.

We will start with the following sequence of API calls and make modifications to illustrate the refraction process.

var1 = SkyScrapperSearchAirport(query="New York")
var2 = SkyScrapperSearchAirport(query="London")
var3 = SkyScrapperFlightSearch(originSkyId="$var1.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
var4 = TripadvisorSearchLocation(query="London")
var5 = TripadvisorSearchHotels(geoId="$var4.geoId$", checkIn="2024-08-15", checkOut="2024-08-18")

Messed up API / function name

In this example, the API or function call has been misspelled 🐦. The refractor corrects it to the correct name. Note that this is NOT done by computing the closest valid API or function name. As explained before, the debugging process is one of optimizing the set of valid tokens. Here, given all the parameters mentioned, the least costly edit turns out to be this API that can make use of the other tokens around it. As a result, this fix is agnostic to how messed up the wrong token was.

  var1 = SkyScrapperSearchAirport(query="New York")
  var2 = SkyScrapperSearchAirport(query="London")
- var3 = SkyCrapperFlightSearch(originSkyId="$var1.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
  var4 = TripadvisorSearchLocation(query="London")
+ var3 = SkyScrapperFlightSearch(originSkyId="$var1.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
  var5 = TripadvisorSearchHotels(geoId="$var4.geoId$", checkIn="2024-08-15", checkOut="2024-08-18")

Note that the fixed sequence is not exactly the same as the starting sequence -- it is in a different order. This is because the API calls mentioned in it can occur in a couple of equivalent orders. Executing this sequence will produce the same outcome as the original one.

Wrong output label

Now we have messed up the output label var2 to var20. This messes up the following steps as well. For example, the assignment of destinationSkyId to $var2.skyId$ is inadmissible now. The debugger has injected an extra line with the correct output label as a result. Again, from the optimization point of view, this edit is not syntactic but is made because this single edit is enough to make all the other tokens valid.

  var1 = SkyScrapperSearchAirport(query="New York")
  var20 = SkyScrapperSearchAirport(query="London")
+ var2 = SkyScrapperSearchAirport(query="London")
  var3 = SkyScrapperFlightSearch(originSkyId="$var1.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
  var4 = TripadvisorSearchLocation(query="London")
  var5 = TripadvisorSearchHotels(geoId="$var4.geoId$", checkIn="2024-08-15", checkOut="2024-08-18")

Notice that while introducing the new step, the debugger has also kept the old vestigial step. It's hanging around there like an appendicitis. This is because we are interpreting a token as "The output of SkyScrapperSearchAirport was assigned to var20" as one to enforce. We may look into ignoring this token in the future.

Missing output label

Instead of a wrong label, here we have removed the label altogether. This has the same impact (inadmissible assignments) on the remaining sequence as in the previous case. The debugger restores the missing token.

  var1 = SkyScrapperSearchAirport(query="New York")
- SkyScrapperSearchAirport(query="London")
+ var2 = SkyScrapperSearchAirport(query="London")
? +++++++

  var3 = SkyScrapperFlightSearch(originSkyId="$var1.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
  var4 = TripadvisorSearchLocation(query="London")
  var5 = TripadvisorSearchHotels(geoId="$var4.geoId$", checkIn="2024-08-15", checkOut="2024-08-18")

Missing input parameter

Now we have removed an input (required) parameter from the API call. Notice that the debugger has put it back. Notice that it has also introduced a slot-fill for it, instead of the assignment in the original sequence.

More about such recovery patterns later.

+ ask(originSkyId)
  var1 = SkyScrapperSearchAirport(query="New York")
  var2 = SkyScrapperSearchAirport(query="London")
- var3 = SkyScrapperFlightSearch(destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
+ var3 = SkyScrapperFlightSearch(originSkyId=originSkyId, destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
?                                +++++++++++++++++++++++++

  var4 = TripadvisorSearchLocation(query="London")
  var5 = TripadvisorSearchHotels(geoId="$var4.geoId$", checkIn="2024-08-15", checkOut="2024-08-18")

Made up input parameter

Similar to the previous example, now we have messed up the name of the parameter instead removing it altogether. The effect is the same. Note that, as discussed previously, this is not an edit based on the text of the mistake but rather on the tokens required by the underlying API specs. That is why a missing parameter and a made-up parameter are almost identical in practice (with certain caveats in recovery patterns).

+ ask(originSkyId)
  var1 = SkyScrapperSearchAirport(query="New York")
  var2 = SkyScrapperSearchAirport(query="London")
- var3 = SkyScrapperFlightSearch(originalSkyId="$var1.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
?                                      --      ^^^^ ^^^    --

+ var3 = SkyScrapperFlightSearch(originSkyId=originSkyId, destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
?                                            ^ ^^^^^

  var4 = TripadvisorSearchLocation(query="London")
  var5 = TripadvisorSearchHotels(geoId="$var4.geoId$", checkIn="2024-08-15", checkOut="2024-08-18")

Assignment to made up variable

Now we have messed up the variable in the parameter assignment, by changing var2 to a non-existent var20. The debugger puts it back.

  var1 = SkyScrapperSearchAirport(query="New York")
  var2 = SkyScrapperSearchAirport(query="London")
- var3 = SkyScrapperFlightSearch(originSkyId="$var1.skyId$", destinationSkyId="$var20.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
?                                                                                   -

+ var3 = SkyScrapperFlightSearch(originSkyId="$var1.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
  var4 = TripadvisorSearchLocation(query="London")
  var5 = TripadvisorSearchHotels(geoId="$var4.geoId$", checkIn="2024-08-15", checkOut="2024-08-18")

Assignment to made up variable parameter

Now, instead of the variable name being wrong, the assignment of the parameter is being made to an invalid field i.e. the output object var2 does not contain that field. The debugger has identified the issue and recommended a slot-fill for the missing value. It could have also mapped to an existing key -- we will study this pattern in more detail below among the recovery patterns.

+ ask(skyayeId)
  var1 = SkyScrapperSearchAirport(query="New York")
  var2 = SkyScrapperSearchAirport(query="London")
- var3 = SkyScrapperFlightSearch(originSkyId="$var1.skyId$", destinationSkyId="$var2.skyayeId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
?                                                                             -------        --

+ var3 = SkyScrapperFlightSearch(originSkyId="$var1.skyId$", destinationSkyId=skyayeId, originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
  var4 = TripadvisorSearchLocation(query="London")
  var5 = TripadvisorSearchHotels(geoId="$var4.geoId$", checkIn="2024-08-15", checkOut="2024-08-18")

Missing step

Finally, we have removed an entire step from the original sequence here. The debugger has suggested a fix. It happens to be perfect here, but there is a lot of intrigue to what this fix may be in practice. There are many things up in the air here:

  1. The debugger does not know that something is missing in the first place. Recall that the debugger here has no higher level abstract goal other than to enforce whatever it is given. So whether a new step will be suggested at all is dependent on whether there are unrequited tokens in the remaining sequence (in this case, $var4.geoId$) and additionally from the optimization point of view, whether it is cheaper to just edit out the unrequited tokens instead of introducing a new one.

  2. Furthermore, what the assignment of the parameter in the new step is, is also up for debate. It could have been "New York" (50-50 chance). While it might be possible to improve upon random assignments in the future, In such situations it is advisable to use defensive recovery mechanisms discussed below.

  var1 = SkyScrapperSearchAirport(query="New York")
  var2 = SkyScrapperSearchAirport(query="London")
+ var4 = TripadvisorSearchLocation(query="London")
  var3 = SkyScrapperFlightSearch(originSkyId="$var1.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
  var5 = TripadvisorSearchHotels(geoId="$var4.geoId$", checkIn="2024-08-15", checkOut="2024-08-18")

1.2 Recovery Patterns

So far, we have been concentrating on the individual mistakes made in a sequence of API / function calls. Now we will look at the suggestions recommended by the debugger in a bit more detail.

1.2.1 Self-contained

The minimal case is when all the information to fix one or more problematic tokens is there in the input sequence itself. Then the recovery pattern is confined to an in-situ fix. This is the case with the first 3 examples above.

1.2.2 Slot-filling

A slot-filling response is when there is an issue with assigning a value to a parameter based on the available tokens and so the debugger suggests we ask its value from the user. This is the case in the next 2 examples. Ideally, we want to reuse the existing tokens, so a slot-fill response is a less preferred recovery compared to maps and function calls.

+ ask(originSkyId)
  var1 = SkyScrapperSearchAirport(query="New York")
  var2 = SkyScrapperSearchAirport(query="London")
- var3 = SkyScrapperFlightSearch(destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
+ var3 = SkyScrapperFlightSearch(originSkyId=originSkyId, destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
?                                +++++++++++++++++++++++++

  var4 = TripadvisorSearchLocation(query="London")
  var5 = TripadvisorSearchHotels(geoId="$var4.geoId$", checkIn="2024-08-15", checkOut="2024-08-18")

1.2.3 Mapping

A mapping is an assignment of a parameter to a value (e.g. query="London") or to another variable (e.g. destinationSkyId="$var20.skyId$"). A fix involving a mapping can appear in two different forms.

Reusing existing maps

Let's go back to the example of an assignment to a made up variable. Here the fix to an invalid assignment has been done based on assignments already available from the existing tokens.

Specifically, var2 already exists and the original tokens showed that the parameter destinationSkyId can be assigned to skyId -- so all the debugger needs to do is to produce a skyId from a call to SkyScrapperSearchAirport (as promised by its API spec) and then assign the output of that call to var2. 😌

  var1 = SkyScrapperSearchAirport(query="New York")
  var2 = SkyScrapperSearchAirport(query="London")
- var3 = SkyScrapperFlightSearch(originSkyId="$var1.skyId$", destinationSkyId="$var20.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
?                                                                                   -

+ var3 = SkyScrapperFlightSearch(originSkyId="$var1.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
  var4 = TripadvisorSearchLocation(query="London")
  var5 = TripadvisorSearchHotels(geoId="$var4.geoId$", checkIn="2024-08-15", checkOut="2024-08-18")

Using a new map

We can also ask the debugger to think of new maps if possible, and not ask the user unless possible. This is an optional feature and has impact on performance.

Compare the results with that in the same situation in a previous example.

  var1 = SkyScrapperSearchAirport(query="New York")
  var2 = SkyScrapperSearchAirport(query="London")
- var3 = SkyScrapperFlightSearch(originalSkyId="$var1.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
?                                      --           ^

+ var3 = SkyScrapperFlightSearch(originSkyId="$var2.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
?                                                 ^

  var4 = TripadvisorSearchLocation(query="London")
  var5 = TripadvisorSearchHotels(geoId="$var4.geoId$", checkIn="2024-08-15", checkOut="2024-08-18")

As noted previously, currently the debugger picks a valid map at random so this map could have (and should have) gone to the parameter skyId from var1 as well. This might be possible to improve upon.

This fix is not entirely syntactic. The newly introduced maps are ones that are close to the unrequited variable in some text embedding space (computed offline for a given catalog of APIs).

1.2.4 Function Calls

Introducing a new map may also involved introducing a new step with an new function call. This recovery pattern can appear in two different contexts.

Missing step

We saw this example in the example above with a missing step. As we discussed previously, the debugger does not know that a step is missing. So whether a new step is introduced depends on whether introducing new things are preferred over deleting already existing stuff per the cost model being used.

  var1 = SkyScrapperSearchAirport(query="New York")
  var2 = SkyScrapperSearchAirport(query="London")
+ var4 = TripadvisorSearchLocation(query="London")
  var3 = SkyScrapperFlightSearch(originSkyId="$var1.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
  var5 = TripadvisorSearchHotels(geoId="$var4.geoId$", checkIn="2024-08-15", checkOut="2024-08-18")

Enabler of a new mapping

A new call is not be confined to missing steps only. If a invalid token can be possibly fixed with a different API, e.g. by producing a new variable that can be mapped to an existing variable whose assignment token needs to be satisfied, then the debugger will introduce a new step if, as in the previous case, introducing new things are preferred over deleting already existing stuff per the cost model being used.

We have now removed the (required) date parameter and the debugger has introduced a call to acquire the current date. This invocation may be dangerous if unsupervised at runtime. This brings us next to the topic of defensive recovery.

  var1 = SkyScrapperSearchAirport(query="New York")
  var2 = SkyScrapperSearchAirport(query="London")
+ var7 = NewsAPISearchByKeyWord()
- var3 = SkyScrapperFlightSearch(originSkyId="$var1.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$")
+ var3 = SkyScrapperFlightSearch(originSkyId="$var1.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="$var7.date$")
?                                                                                                                                                                    ++++++++++++++++++++

  var4 = TripadvisorSearchLocation(query="London")
  var5 = TripadvisorSearchHotels(geoId="$var4.geoId$", checkIn="2024-08-15", checkOut="2024-08-18")

1.2.5 Defensive Recovery

In systems that execute on the results of their reasoning, it is advisable to use "defensive execution" where the system can confirm certain aspects of its reasoning (in this context, the component of an API call) before acting on it. The NL2Flow package offers some off-the-shelf options for defensive execution such as confirming newly introduced maps, newly determined variables, and so on, as described here.

We can re-use the same for debugging, as shown below. Note that, unlike defensive execution in NL2Flow, here the defensive operations will only show up for extra stuff that the debugger has added in.

This feature is optional and has impact on performance.

refract(tokens, catalog, defensive=True)

Previously, we saw how newly introduced maps may pick the wrong mapping when picking from available maps at random. We also saw how a new operator may be introduced that the user may not like. Let's revisit the same situations with the defensive guardrails on.

Defense against new mapping

Contrast with the corresponding example above.

  var1 = SkyScrapperSearchAirport(query="New York")
  var2 = SkyScrapperSearchAirport(query="London")
+ confirm(originSkyId="$var1.skyId$")
- var3 = SkyScrapperFlightSearch(destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
+ var3 = SkyScrapperFlightSearch(originSkyId="$var1.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
?                                ++++++++++++++++++++++++++++

  var4 = TripadvisorSearchLocation(query="London")
  var5 = TripadvisorSearchHotels(geoId="$var4.geoId$", checkIn="2024-08-15", checkOut="2024-08-18")

Defense against using information from new operation

Contrast with the corresponding example above.

  var1 = SkyScrapperSearchAirport(query="New York")
  var2 = SkyScrapperSearchAirport(query="London")
+ var7 = NewsAPISearchByKeyWord()
+ confirm(date="$var7.date$")
- var3 = SkyScrapperFlightSearch(originSkyId="$var1.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$")
+ var3 = SkyScrapperFlightSearch(originSkyId="$var1.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="$var7.date$")
?                                                                                                                                                                    ++++++++++++++++++++

  var4 = TripadvisorSearchLocation(query="London")
  var5 = TripadvisorSearchHotels(geoId="$var4.geoId$", checkIn="2024-08-15", checkOut="2024-08-18")