Fuck You, Show Me The Prompt [UPDATE]
· Originally published on Medium
Intercepting LLM API calls
If you clicked, there’s high chance that you have already read Fuck You, Show Me The Prompt by Hamel
Lets take this a step further, and take a closer look at LLM observability, and the various frameworks that are available for the same.
I wanted something like LangSmith but opensource and I came accross Phoenix
Example Trace in Phoenix
Pros:
++ can create multiple projects
++ can export traces easily
++ good ui for displaying conversations (doesn’t display images tho)
Cons:
— doesn’t have inbuilt cost monitoring
— bugs when exporting traces, for a project I was not able to export user messages and it was super hard to debug why
— can’t selectively delete traces in the UI
Next, I tried Portkey:
Pros:
++ inbuilt ui for cost tracking
++ inbuilt caching and guardrails
Cons:
— many essential features requires enterprise upgrade
— falls behind pheonix with respect to free tier
— not opensource
I do not recommend portkey
What I am using now?
pros:
++ shows both traces and high level dashboard
++ generous free tier
++ opensource
since its opensource and provides a good starting point for LLM observebility, you can use it as a starting point and make changes in the code as necessary
Bottom Line: LLM observebility and evals are crutial while building Generative AI apps, use any tool that you like, but monitor how people are using your GenAI Apps