Skip to content

Refraction on the NESTFUL Dataset

All our examples so far has been using samples from the NESTFUL dataset. Of course, as ground truth, there are no mistakes to fix here. Nevertheless, we ran it on the data to measure some preliminary scaling characteristics, as well as catch some mistakes with the ground truth along the way! 😉

6.1 Executable Dataset

Two (three?) kinds of errors were noticed quite frequently.

Wrong parameter name

For this example, the API TripadvisorSearchRestaurants is being called with wrong parameters. The sequence is invalid because of a missing required parameter.

  var1 = TripadvisorSearchLocation(query="Rome")
+ ask(locationId)
- var2 = TripadvisorSearchRestaurants()
+ var2 = TripadvisorSearchRestaurants(locationId=locationId)
?                                     +++++++++++++++++++++

Similarly, for the following example, the API Tripadvisor_Search_Restaurant_Location does produce location and location.locationId but not directly locationId.

  var1 = Tripadvisor_Search_Restaurant_Location(query="Paris")
+ ask(locationId)
- var2 = Tripadvisor_Search_Restaurants(locationId="$var1.locationId$")
?                                                  -------          --

+ var2 = Tripadvisor_Search_Restaurants(locationId=locationId)
  var3 = Tripadvisor_Get_Restaurant_Details(restaurantsId="$var2.restaurantsId$")

Assignment to non-existent parameter

Another kind of frequent error is an assignment to an incorrect parameter of the result of an API call. In this example, the API Spotify_Scraper_Get_Artist_ID_By_Name does not produce an id.

  var1 = Spotify_Scraper_Get_Artist_ID_By_Name(name="Ed Sheeran")
+ var1 = Instagram_Search_User()
  var2 = Spotify_Scraper_Get_Artist_Overview(artistId="$var1.id$")

We have fixed several dozens (🥲) of such mistakes in the "ground truth" of NESTFUL using the refraction package. The newest version of the data is made available here and is also available on PyPI here.

Miscellaneous 🙈

We currently don't deal with operations on the parameters. So examples like these are false positives.

  var1 = Alpha_Vantage_CURRENCY_EXCHANGE_RATE(function="CURRENCY_EXCHANGE_RATE", from_currency="INR", to_currency="JPY")
- var2 = CipherCircuit_Math_Assistant_CalculateAllArithmeticOperations(numbers="100 * $var1.Exchange Rate$")
?                                                                               ------

+ var2 = CipherCircuit_Math_Assistant_CalculateAllArithmeticOperations(numbers="$var1.Exchange Rate$")