We are excited to announce the release of the Proxati llm_proxy v2.1.0, with some new logging features, bug fixes, and performance improvements. This version brings enhancements that make the llm_proxy more reliable, efficient, and easier to use.
Major Changes and Improvements
This release focuses on logging, including the removal of the dir_logger mode in favor of logging traffic from any mode.
We've also renamed the 'simple' mode to 'run' for clarity and cleaned up the logging addon code for better maintainability. Various other cleanups and workflow improvements have been implemented in response from our users. This is laying the foundation for future improvements in logging and observability.
This version is the first being used by several community members and should be considered a stable release.
Detailed Update Highlights
apiAuditor Mode
One of the most exciting features of this release is the apiAuditor mode, which shows real-time tracking and visualization of your generative AI cost while developing.
Real-Time Cost Tracking Example:
$ ./llm_proxy apiAuditor
URL: https://api.openai.com/v1/chat/completions Model: gpt-3.5-turbo inputCost: $0.000102 outputCost $0.000252 = Request Cost: $0.000354 Grand Total: $0.000354
URL: https://api.openai.com/v1/chat/completions Model: gpt-3.5-turbo inputCost: $0.000102 outputCost $0.000432 = Request Cost: $0.000534 Grand Total: $0.000888
URL: https://api.openai.com/v1/chat/completions Model: gpt-3.5-turbo inputCost: $0.000102 outputCost $0.000138 = Request Cost: $0.00024 Grand Total: $0.001128Implementation Guide
To use the apiAuditor mode, see the previous blog post about this feature which includes example code. Only the OpenAI chat completion API is currently supported, but we are actively working on adding more providers and endpoints based on your feedback.
The power of Open Source
The introduction of the apiAuditor mode and other logging enhancements in v2.1.0 is a direct response to user feedback. We invite you to try out this feature and share your thoughts with us. Our goal is to make AI APIs easier to operate and manage, and your feedback is crucial to our continuous improvement.
Check out the new release on GitHub: Proxati LLM_Proxy v2.1.0