Your Software Has an AI Mode. Is It Safe?

The Quiet Risk

A normal software feature waits for a defined input and follows a defined path. An AI feature may interpret a broad request, choose among tools, read several sources, and decide which action seems useful next. That flexibility is why it feels powerful. It is also why the risk cannot be understood from the button label alone.

The most capable AI features usually ask for access to something valuable: email, documents, source code, customer records, calendars, payment systems, cloud accounts, or production databases. They do not inherently need access to everything. They need the minimum permissions required for the task. When a team grants broad access because it makes the first demo easier, it quietly transfers real authority to a system that can misunderstand context.

This is where we see the danger growing. More founders, operators, and subject-matter experts can now write software through natural-language tools. That access can be genuinely useful. What is not useful is shipping code nobody can explain, test, restore, or own. Generating an application is not the same as operating one, and a polished interface tells us almost nothing about how safely the system handles data behind it.

AI also changes the scale of a mistake. A person may delete one record and notice the confirmation dialog. An agent with the same credentials can repeat an action across thousands of records before anyone opens the dashboard. The model is not malicious. It is simply fast, connected, and capable of being wrong.

Documented Incidents

In July 2025, Replit's development agent deleted data from a production database during Jason Lemkin's public test. Replit CEO Amjad Masad acknowledged the incident and wrote that it was “unacceptable and should never be possible.” Replit responded with automatic separation between development and production databases, staging environments, and restore controls. Those are governance decisions added after the failure: separate environments, constrain access, and make recovery real.

A user report in Google's official Gemini CLI repository described an agent running a destructive command while installing packages. The user said it began deleting Documents, Downloads, and Desktop files, and stated the expected behaviour plainly: “Not delete all my files in unrelated directories.” Google marked the issue high priority and linked it to follow-up work. This is a public bug report, not an independent forensic finding, but it shows the practical risk of combining a broad instruction, persistent approval, and a destructive system tool.

In April 2026, a detailed report in Anthropic's official Claude Code repository documented subagent commits that included unintended deletion of production modules, tests, scripts, and planning files. The issue was labelled both “data-loss” and “has repro.” Its author described the impact as “deleting production source code — not just documentation.” The report also contains the lesson we keep returning to: an agent should not be able to stage or delete outside the scope it was given without a hard confirmation.

The loss does not need to be a deleted database. In Moffatt v. Air Canada, a customer relied on incorrect bereavement-fare information supplied by the airline's chatbot. The British Columbia Civil Resolution Tribunal held Air Canada responsible and awarded the customer damages, interest, and fees. The tribunal's wording was blunt: “It should be obvious to Air Canada that it is responsible for all the information on its website.” The business owns the outcome even when the words came from software.

These cases are different. One involved a vendor-acknowledged production deletion, two are public technical reports filed by users, and one is a legal decision about incorrect information. We should not flatten them into a claim that every AI feature is unsafe. Together they show the same operational truth: once software can act, the quality of its boundaries matters as much as the quality of its answers.

Before the Incident

An AI audit begins with an inventory. Which products contain AI? Which employees have connected personal assistants to company accounts? What data can each system read? What can it create, change, send, approve, publish, purchase, or delete? Many businesses cannot answer those questions because AI arrived through ordinary software updates rather than a formal implementation project.

From there, we examine consequences rather than model marketing. If the assistant is wrong, what is the largest plausible loss? Can the action be reversed? Does a human see it first? Is there a record of the request, source, action, and approval? Who can stop the system, and who is accountable for recovery? The answers tell us where a friendly convenience has become an operational control problem.

The audit does not exist to slow every use of AI or automation. It separates low-risk help from high-impact authority. Summarising an internal meeting is not the same as sending a contractual promise. Drafting a database query is not the same as running it against production. Suggesting a code change is not the same as merging and deploying it. Governance lets the easy work stay easy while putting friction exactly where a mistake becomes expensive.

What Governance Means

The word governance sounds larger than it needs to be. For us, it means the business has decided what the AI is for, what it may access, which actions require approval, how activity is recorded, how performance is reviewed, and what happens when the system fails. Someone owns those decisions. They do not disappear into a vendor contract or an employee's private account.

NIST's AI Risk Management Framework organises the work into four continuing functions: govern, map, measure, and manage. The sequence is useful because it refuses the fantasy that safety is a one-time launch checklist. Context changes. Vendors update models. Employees connect new tools. Data moves. A control that worked six months ago may no longer protect the same boundary.

Runtime products can help turn those decisions into enforcement. Clyro, incubated by PromptCloud, provides a governance layer for AI agents with policy checks, cost and step limits, approval controls, activity records, and tools intended to stop unsafe actions while an agent is running. Clyro's own materials describe a default-deny approach and per-action evidence. Those are product claims that still need to be evaluated against the buyer's systems, risk, and deployment requirements, but the category solves a real problem: written rules are weak if nothing checks them before execution.

OWASP calls the underlying risk “excessive agency”: giving an AI system more functionality, permissions, or autonomy than it needs. Its guidance recommends minimum privileges and human approval for high-impact actions. That should be normal business practice. An agent that drafts a refund can be useful. An agent that issues any refund, to anyone, without a limit or review is a financial control failure wearing an AI label.

Why We Care

We have watched people lose faith in software for ordinary reasons: a form that discards an hour of work, a migration that changes records, a dashboard nobody can reconcile, an update that removes the one feature the team depended on. The technical defect may be fixed in a day. The human response lasts longer. People keep shadow spreadsheets, avoid the new workflow, and ask a colleague to check what the system already checked because they no longer believe it.

AI can deepen that problem because its mistakes often arrive wrapped in confident language, and because we are connecting it to more of the business. If a chatbot invents a policy once, staff will question every answer. If an agent deletes the wrong files once, developers will watch every future change. If a customer sees private information once, “we fixed the prompt” will not restore the relationship.

This is why we are uneasy with the idea that everyone writing their own code is automatically progress. More people being able to shape software is good. More unreviewed software handling customer data, payments, employment decisions, or production operations is not. We want AI to widen participation without erasing engineering discipline: tests, access boundaries, version control, backups, review, monitoring, and clear ownership.

At Brownsmith Dynamics, that belief shapes both our AI implementation work and our Security Operations Console. We would rather tell a client that an AI action needs one more approval than explain why nobody noticed it running overnight. We would rather scope the permissions tightly and expand them later than start with total access because the demonstration looks better.