4.1 Debugging as an Optimization Problem¶

While introducing the concept of refraction, we talked about how we cast the debugging task as one of optimizing how many of the tokens in a sequence of API calls can be enforced given a set of API specifications. Of course, there may be multiple ways to fix an invalid token and hence the cost model used in the optimization process is crucial.

In a previous chapter, we talked about two considerations in the refraction process: 1) the nature of the mistake being edited; and 2) the nature of the recovery response. Unsurprisingly, the cost model impacts both.

4.1.1 Preferred Edits¶

One of the critical considerations in trading off alternative fixes for a mistake is how we rank the preference of an edit in terms of whether it is a positive or negative edit, i.e. whether we want to fix stuff by adding or removing items; and which items positive and negative edits are operating on, e.g. are we editing the name of an API or a parameter within an API.

Let us take the example of a missing label again and turn down the cost of a negative edit. In other words, this allows the debugger to remove stuff if it wants in order to make all the remaining tokens valid. The same input sequence now produces a delete operation on the problematic step and reuse an existing map, instead of trying to produce a new label that maintains the existing tokens.

  var1 = SkyScrapperSearchAirport(query="New York")
- SkyScrapperSearchAirport(query="London")
- var3 = SkyScrapperFlightSearch(originSkyId="$var1.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
?                                                                                  ^                                                                     ^

+ var3 = SkyScrapperFlightSearch(originSkyId="$var1.skyId$", destinationSkyId="$var1.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var1.entityId$", date="2024-08-15")
?                                                                                  ^                                                                     ^

  var4 = TripadvisorSearchLocation(query="London")
  var5 = TripadvisorSearchHotels(geoId="$var4.geoId$", checkIn="2024-08-15", checkOut="2024-08-18")

4.1.2 Preferred Recovery Patterns¶

Similar trade-offs apply to recovery patterns. We saw previously in the case of the made up input parameter, how we can switch between slot-filling and mapping and function calling recovery modes on the same input.

Slot-filling preferred¶

+ ask(originSkyId)
  var1 = SkyScrapperSearchAirport(query="New York")
  var2 = SkyScrapperSearchAirport(query="London")
- var3 = SkyScrapperFlightSearch(originalSkyId="$var1.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
?                                      --      ^^^^ ^^^    --

+ var3 = SkyScrapperFlightSearch(originSkyId=originSkyId, destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
?                                            ^ ^^^^^

  var4 = TripadvisorSearchLocation(query="London")
  var5 = TripadvisorSearchHotels(geoId="$var4.geoId$", checkIn="2024-08-15", checkOut="2024-08-18")

Internal maps preferred¶

  var1 = SkyScrapperSearchAirport(query="New York")
  var2 = SkyScrapperSearchAirport(query="London")
- var3 = SkyScrapperFlightSearch(originalSkyId="$var1.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
?                                      --           ^

+ var3 = SkyScrapperFlightSearch(originSkyId="$var2.skyId$", destinationSkyId="$var2.skyId$", originEntityId="$var1.entityId$", destinationEntityId="$var2.entityId$", date="2024-08-15")
?                                                 ^

  var4 = TripadvisorSearchLocation(query="London")
  var5 = TripadvisorSearchHotels(geoId="$var4.geoId$", checkIn="2024-08-15", checkOut="2024-08-18")

As we have established so far, the results of refraction depends on the cost model of each edit both in terms of what edits are made and what recovery pattern is used in response to an edit. The refraction package currently operates under the following cost model:

Edits on parameter names are preferred over function calls.
Edits on parameter to value maps are preferred over edits to parameter names.
New maps are considered at all and how they are preferred over other edits.
Recovery operations include new maps, slot fills, and new functions calls.