Skip to content
Sortup
Book a call

5 days · Production Audit

Your LLM feature is live. Find out what it costs, where it fails and what to fix first.

For teams with an LLM feature in production and complaints they cannot reproduce.

Book a 20-minute fit callGet a free AI Teardown

Start to finish

5days

  • Evals from your traffic
  • Cost and latency
  • Ranked fix list

The system

Production Audit, end to end.

  • Already yours
  • We build
  • Human step
How the Production Audit works, in 5 steps
  1. 01

    Your live app

    Traces and a traffic sample

  2. 02

    Eval set

    Built from real requests

  3. 03

    Review

    Cost, latency, security

  4. 04

    Fix list

    Ranked, with evidence

  5. 05

    Walkthrough

    60 minutes with your team

Builds layer

  • 04Evals and guardrails
All five layers

Use cases

What you get

Six deliverables you can check.

  • An eval set built from your real traffic, with a baseline score
  • Cost per request and per user, with the biggest drivers
  • A latency breakdown across retrieval, model and tools
  • A security review: prompt injection, data leakage, access control
  • A ranked fix list with effort estimated in days
  • A 60-minute walkthrough with your engineers

How it works

5 days, four stages.

Production Audit timeline: Access and a traffic sample, day 1; Evals and tracing, days 2–3; Cost, latency and security, day 4; Report and walkthrough, day 5.

  1. 01Day 1

    Access and a traffic sample

  2. 02Days 2–3

    Evals and tracing

  3. 03Day 4

    Cost, latency and security

  4. 04Day 5

    Report and walkthrough

What it is, and what it is not.

It is

  • Findings ranked by impact, with the evidence for each
  • A second opinion from engineers who build and run LLM systems

It is not

  • A rewrite, or a pitch for one
  • A penetration test of your whole stack
  • A report of generic best practice

AI work

Similar systems, already shipped.

Built by Sortup or one of our co-founders, as labelled. Not results of this offer.

All client work
  • momentswriter.ai
    MomentsWriter: Family (Answers guided questions), then Draft (LangGraph, several models), then Revisions (In the same conversation), then Evaluation (Quality checks on the draft), then Obituary (Final text).

    Co-founder project · Writing assistant

    MomentsWriter case study

    Multi-step drafting with revision loops and quality evaluation.

The team delivered a complex super app with e-wallet, e-commerce, logistics integrations, and a dynamic commission engine. Their communication was clear, execution was reliable, and they consistently provided solution-driven results throughout the project.
Simon SprouleCo-founder, Gather

Scope

What changes the scope.

  • LLM features or agents in scope
  • Whether tracing already exists
  • Traffic history we can sample
  • Compliance rules for production data

FAQ

Questions about this one.

Do you need production access?

Read access to traces and a traffic sample, under your NDA, is enough. If that is not possible, we work from exported logs.

Will you try to sell us a rebuild?

No. The fix list estimates each item in days and says which ones your team can do in-house.

What do we keep?

The report, the eval set and the harness that runs it, so you can re-score after every change.

Is the Production Audit the right first step?

No pitch deck. An engineer, and a straight answer.

Book a 20-minute fit callGet a free AI Teardown
Hours
We work yours, in the GCC, the UK and EU or the US
NDA
We sign yours before the first call
IP
You own the code from the first commit