Coding Challenge for Generative AI, Part 4: HuggingChat

This post extends the series of code interpretation challenges for Generative AI for the new Generative AI Chat service HuggingChat from HuggingFace. They didn’t announce any particular support for coding or code interpretation, but I figured it would be interesting to see how it did on day 1.

Previous parts in the series gave the same challenge as given below, but to different services, Part 1 was GPT-3.5, Part 2 was GPT-4, and Part 3 was Google Bard.

Session Start

What will be the result?
c = [False if int(i) % 2 != 0 else True for i in str(24901)]
b = [False if False in c else True]
print(b[0])

I think this code is written incorrectly. As you can see there seems to be some missing parts which would prevent this from functioning correctly

Thoughts

Maybe just a little adjustment to the prompt?

Trying Again

What will be the result of this python script:
c = [False if int(i) % 2 != 0 else True for i in str(24901)]
b = [False if False in c else True]
print(b[0])

I apologize but that line of code is still incorrect, here is the corrected version : c = list(map(lambda x: False if x%2!=0 else True ,str(24901))) print(next(filter(None,c),False))

Thoughts

Interesting! You can see an attempt here to reinterpret into a more functional style of programming (explicit nested functions with iteration).

Part of it is actually close..

This is only slightly off:

list(map(lambda x: False if x%2!=0 else True ,str(24901)))

The problem is that x in the above is a character (from map iterating over stringified number “24901”), so simply converting to int would fix that:

list(map(lambda x: False if int(x)%2!=0 else True ,str(24901)))

That would then evaluate to the correct array for c that we are expecting!

The next part is more fundamentally off however.

This:

print(next(filter(None,c),False))

Is not a good replacement. filter(None,c) will filter out False-y values. So if there are any True values in c then next() will return the first element and it will be True. If all of the values are False and they are all filtered out then next() will return False. Whereas the original script results in False if any of the elements of c are False this translated construct will only return False if all of the elements of c are False.

While it’s not right, it feels like a damn good try.

Evaluating?

what is the result of that python script?

The output of this script will vary depending on whether len(str(24901)) is even or odd. If it is even, then the length of the resulting list b will be 1, containing the value True. If it is odd, then the length of the resulting list b will be 2, containing True at index 0 and False at index 1.

Thoughts

Since it has tried to “correct” my python script instead of really interpret it, I figured I’d try to get it to interpret. It somehow brought in the notion of the length of the number of digits in the number mattering, so I figured it was too far afield to try and take it further. It didn’t quite interpret to a concrete answer, but rather provided an incorrect explanation of what the code would do under different circumstances.

Conclusion

It’s not clear that HuggingChat is quite ready to interpret code to provide an answer of stepping through code.

I think instead what happened is my question about the script taken as an instruction to create a script. Or close to that; it stated it was “correcting” the script that I gave it.

I was impressed by how close it was to providing an alternative implementation to evaluating the c variable. This hints at behind the scenes interpretation and then generation of an (almost) isomorphic implementation. But for a missing conversion to int it was spot on. In human terms it came down to making a switch in coding style.

This is probably not entirely surprising given we are working with a language model. While the language is more formal than English, it’s equivalent to finding different ways of saying the same thing.

Unfortunately its equivalent to evaluating the b variable was further off. It correctly would boil down to a single value, but the semantics were interestingly off for how the value would be interpreted.

I think putting each of these services through the same simple challenge exposed something different about each one, and from HuggingChat the “rewording” of my script to a different style provides a good hint at exposing the language model nature and how it applies to the formalized language of code.