The request that should scare you is not “delete my account.” It’s “clean up my duplicate contacts.” One is obviously destructive. The other sounds like a helpful feature, gets approved in a product meeting without a second thought, and quietly requires an LLM to decide which rows in your production database no longer deserve to exist. We built that feature for a client-facing admin console last year. It shipped. It has not deleted the wrong row yet, and the reason isn’t that the model got smarter — it’s that we never let the model’s judgment be the last line of defense.
Most write-enabled AI features fail in one of two ways: either the team gets nervous and restricts the model to read-only summaries (safe, and also not the feature anyone asked for), or they wire the model’s output straight into an ORM call with a system-level credential and hope the prompt holds. The prompt will not hold. Prompt injection through user-supplied content, tool-call argument drift, and plain old model confusion are not edge cases at scale — they’re Tuesday. The fix isn’t a better prompt. It’s treating the LLM as an untrusted caller, the same way you’d treat a third-party API consumer, and building the permission model accordingly.
Scoped credentials, not scoped prompts
The first decision we made was to stop asking the model to be careful and start making carelessness impossible at the database layer. The LLM never receives your application’s normal database credential. It gets a dedicated Postgres role, created specifically for this feature, with privileges granted per-table and per-operation:
- UPDATE and INSERT on the specific tables the feature touches — contacts and contact_merge_log in our case — and nothing else.
- No DELETE grant at all. Every “delete” the model performs is actually an UPDATE that sets a status column to archived. Reversibility is a permission decision, not an application-logic promise.
- Row-level security policies scoped to the tenant_id of the session making the request, enforced at the database, so a cross-tenant write isn’t something the application code has to remember to check.
- No DDL privileges, no access to the users, billing, or auth schemas, full stop.
This means that even in the worst case — the model is fully manipulated by adversarial input and emits an arbitrary write — the blast radius is bounded by what a Postgres GRANT statement allows, not by what a system prompt discouraged. That’s a much smaller trust surface, and it’s one you can audit with \dp in psql instead of by re-reading prompt history.
The validation layer the model never sees
Scoped credentials stop the model from touching the wrong table. They don’t stop it from writing garbage to the right one. For that we put a validation layer between the model’s tool call and the actual SQL execution — and critically, the model doesn’t know this layer exists or what its rules are, because a sufficiently motivated prompt injection will try to talk its way around any rule it can see.
Concretely, every write the model proposes goes through:
- Schema-bound tool definitions. The model doesn’t write SQL. It calls a typed function like
merge_contacts(primary_id: uuid, duplicate_ids: uuid[], reason: string). There is no code path from model output to a raw query string, which eliminates SQL injection as a category, not just as a mitigated risk. - Pre-write invariant checks. Before the merge executes, application code — not the model — verifies the IDs exist, belong to the same tenant, and aren’t already archived. These checks are ordinary code review artifacts, testable with normal unit tests, independent of anything the LLM decided.
- A dry-run diff returned before commit. The function computes what would change and returns it. For anything above a low-confidence threshold, that diff goes to a human queue rather than executing immediately. We tuned “low confidence” empirically — name similarity plus overlapping email domain auto-executes; fuzzy matches on name alone don’t.
- An append-only audit log with the model’s stated reasoning attached to every write. When a merge looks wrong three weeks later, we can see exactly what evidence the model cited, which matters more for debugging than for compliance.
What this costs you
Being candid about the trade-off: this is slower to build than “call the API and pipe the response into a query.” You’re writing a typed function per allowed operation, a confidence-scoring pass, and a review queue UI, before the AI feature does anything a user notices. For a feature with two or three write operations, that’s a reasonable week of backend work. For a feature that wants to write to fifteen tables, this pattern is a signal that the feature is too broad and should be split, not a reason to skip the validation layer.
The other cost is latency — a dry-run-then-commit flow is two round trips instead of one. We’ve found users tolerate this fine when the UI shows the diff as a confirmation step rather than a spinner, which is a product decision as much as an engineering one.
This is the kind of problem we like taking on at orithLabs — not “add an AI chatbot,” but “let AI take real actions on real data without becoming the thing your security review flags.” We’ve built this pattern into production systems like Crumb Count and Kompete, and we’re happy to walk through the specific grants, tool schemas, and audit design with your team if you’re weighing a similar feature.