Guide to building your first LLM-powered app
  • Building an LLM-powered Application
    • Who this guide is for
  • Before you build
    • As a Business User
    • As a Product Manager
    • As a Designer
  • Implementing LLMs
    • Model Selection
    • Prompt Engineering
    • Model Failures
  • Important Next Steps
    • On LLM Usage Analytics
    • On LLM Fraud, Waste & Abuse
    • On Pricing LLM-powered features
    • Beyond V1
Powered by GitBook
On this page
  1. Important Next Steps

On LLM Usage Analytics

PreviousModel FailuresNextOn LLM Fraud, Waste & Abuse

Last updated 1 year ago

You’re not done when you release your LLM feature. You’re not done when you add tests for it either. You are done when your LLM features help you achieve your original business objective. This is the most important step that almost everyone ignores.

Most companies are still trying to add and distribute LLM-powered features. Since there weren’t enough best practices on how to monitor our LLM usage in the wild, the team at LogicLoop has best practice dashboards ourselves, to:

  1. Help Sales understand usage for specific accounts

  2. Help Product understand usage trends

  3. Help Engineering understand load patterns, monitor incidents etc.

These dashboards allowed anyone to look up an account and understand # users, # returning users, # interactions and tokens used (as a precursor to On Pricing LLM-powered features), most popular times of day and specific prompts (to optimize sales outreach for conversion).

It's not fancy, but all stakeholders at LogicLoop can understand just how much our AI features are being used, who is using them and for what.

If you want to set up your own dashboards for monitoring AI feature activity in 10 minutes, feel free to copy our templates .

here
LogicLoop AI Activity Dashboard for Product, Sales and Engineering