Risks + mitigations

What can go wrong — and how we mitigate.

The ten risks we actually plan for on every engagement. Honest severity ratings, the mitigations we build in by default, and what's still on you to own.

Hallucination — confidently wrong

High

The model generates plausible-sounding but false content. Especially dangerous in customer-facing or regulated contexts.

How we mitigate

Citation-required prompting on every fact-bearing response
Retrieval-augmented generation with hybrid search + reranking
Refusal patterns when retrieval confidence is below threshold
Eval suite that measures hallucination rate weekly
Human review gate on high-stakes outputs (medical, legal, financial)

Drift — silent quality decay over time

High

Model behavior shifts as upstream model providers release updates, your data evolves, or edge cases accumulate. Quality dies quietly.

How we mitigate

Weekly eval-suite replay against the full case library
Drift alerting on score deltas above a threshold
Production failures become permanent eval cases
Named human owner of the eval suite — non-optional in our SOW
Quarterly drift retrospective with the client

Prompt injection — user manipulating the agent

Medium-High

Users craft inputs that make the agent ignore system instructions, leak prompts, or take unintended actions.

How we mitigate

Input sanitization at the orchestration layer
Refusal patterns for obvious injection attempts
Tool-use gating: no agent action without explicit approval workflow
Logging + alerting on suspected injection attempts
System prompts that don't contain anything secret you couldn't put on a billboard

Cost runaway — token bills surprising leadership

Medium

Production AI cost compounds quickly. A change that improves quality 10% but triples cost can blow your budget without notice.

How we mitigate

Per-conversation cost telemetry from day one
Prompt caching for static context (system prompts, schemas)
Tier-down to smaller models for low-stakes turns
Cost budgets with daily / weekly alerts
Eval suite logs cost per case — surfaces cost-quality regressions

PII / PHI exposure

High

Sensitive personal data ends up in prompts, logs, or model training. Compliance, contractual, and ethical consequences.

How we mitigate

PII detection + redaction at the orchestration layer
Encryption in transit and at rest, customer-managed keys when required
Use of enterprise model tiers that exclude data from training
DPAs with all sub-processors
Retention policies sized to your obligations, not our defaults

Auto-execution of irreversible actions

High

Agent sends an email, posts to the ledger, refunds a customer, or merges a PR without proper review.

How we mitigate

v1 agents draft; humans approve; system executes — never the reverse
Allowlist of touchable actions, deny-by-default for everything else
Explicit confirmation tokens for irreversible operations
Audit log on every executed action with named approver
Sandbox environments for any agent that writes code or executes

Vendor lock-in to a single model provider

Medium

Built around one model's quirks. Provider raises prices, deprecates the model, or has an outage — you have no fallback.

How we mitigate

Abstract the model behind a thin internal interface
Run the eval suite against 2+ model providers
Open-source model fallback for non-critical paths
Cost-comparison telemetry across providers, refreshed quarterly

Regulatory + audit exposure

Medium-High

Auditor asks 'show me the AI's reasoning on account 4729 on March 14.' If you can't, you have a problem.

How we mitigate

Replayable reasoning traces — every prompt, retrieval, tool call, output
Logs tied to named users, named systems, unique decision IDs
Retention sized to your audit window (often years)
Model approval workflow with named approver per release
Refusal patterns documented in plain English for the audit team

Vendor disappears mid-engagement

Medium

Your AI vendor (us or anyone else) gets acquired, pivots, or shuts down. Your AI is now an orphan.

How we mitigate

Code, prompts, configs in your repos from day one — not ours
Infrastructure in your accounts, not ours
Runbooks written for your team, not for our convenience
Eval suite owned by a named human on your side
Quarterly handoff drill — can your team operate this without us?

Over-automation — losing the human judgment that matters

Medium

Automating workflows that should have kept a human in the loop. Customers feel it. Quality decays in subtle ways. Your team loses skill.

How we mitigate

Default to human-in-loop on anything emotional, contractual, or strategic
Sample 5% of automated outputs for human QA review weekly
Cross-train the team — don't atrophy the skills you'd want back
Sunset clause in every automation: would we be OK if this stopped tomorrow?

Want our risk packet for your security review?

Comprehensive document with our threat model, control mappings, and the specific mitigations we'd build for your engagement. Email security@aliansoftware.net and we'll send within a day.

Request the risk packet Read the Security overview