Notice when your LLM app gets quietly worse.

DriftWatch is an LLM drift monitor: it watches a running LLM app for quality that drops without any error or crash. It scores five signals, including embedding drift and canary accuracy, combines them into one score and alerts when that score crosses a threshold. It shows that quality moved, though not always why.

It watches several signals at once instead of trusting one metric. Embedding drift shows answers changing in meaning, canary probes catch wrong answers to known questions, and an LLM judge grades a sample against a rubric. When the combined score crosses a threshold, you get an alert. The alert says something moved; finding the cause is still your job.

Partly. The hallucination score asks the same question several times and measures how much the answers' wording disagrees. It is a self-consistency heuristic based on word overlap, not fact checking. A model that gives the same wrong answer every time can pass it, so treat a clean score as weak evidence.

Install the package with pip install driftwatch-llm[agent], then run driftwatch init inside your project folder. It signs you in through the browser, creates a project and stores its key in your system keychain. From then on, a small background agent starts whenever your Python app runs. Run driftwatch dashboard to see it.

Yes. Every score the local agent computes syncs to your DriftWatch account, so the same charts appear on the website after you log in. If you would rather skip the local install entirely, connect your model from the web dashboard and DriftWatch runs its scheduled checks against it from the cloud.

One: embedding drift, meaning how far recent responses have moved from a baseline of your app's first 200. It is computed on your machine and works with the OpenAI, Anthropic, Groq and Google GenAI Python SDKs, streaming included. The other signals need extra model calls and keys, and embedding drift alone raises no alerts yet.

With the free allowance, a redacted sample of a response (emails, keys and numbers masked, cut to 1500 characters) goes to the DriftWatch relay, which asks a small language model to grade it and keeps nothing. It is off until you run driftwatch judge enable, and you get 50 graded samples before it stops.

Your key travels over an encrypted connection, is checked once with your provider, then encrypted with a key that only our scheduled runner holds, so the website and database cannot read it. The runner uses it for fixed questions on your schedule and budget. You pay your provider for those calls, and you can delete the key any time.

Each account gets a small free allowance on our key, enough to try DriftWatch on a real app. When it runs out, add your own provider key and keep going on your own account with your provider. You can also bring your own key from the start and skip the allowance entirely if you would rather not use ours.

Not by default. Scores are computed on your machine and only numbers reach your account. Prompts are never read. The one exception is the optional judge: once you turn it on, a small sample of responses, with emails and keys masked first, goes to the model provider you chose.

The demo is a small question-answering bot that DriftWatch monitors in real time. Buttons such as Simulate model downgrade and Simulate prompt injection creep push it into drift, and you can watch the five signals and the combined score react. Every visitor shares the same demo, so you may see other people's runs.

No. Every visitor shares the demo, and scenarios you trigger are visible to others. Any question you type goes through the demo server to an LLM provider so the bot can answer it. Do not type anything sensitive into the demo, such as personal data, passwords or confidential company details.

An account lets you create projects, make API keys and send events to a project with a plain HTTP call. Your scores update live on your dashboard. You can export your data or delete your account yourself in Settings. The live demo needs no account.

DriftWatch cannot prove your model is correct, and it cannot always explain why quality changed. It watches statistical signals and a sampled LLM judge, so a quiet score means nothing obvious moved, not that nothing is wrong. Treat an alert as a reason to look closer, then read real examples yourself.