When AI remembers the wrong thing

When yesterday’s claim becomes today’s fact

A finance manager at a manufacturer receives a well-organized supplier-payment review pack. The AI assistant has brought together the invoice, the supplier’s details and a note confirming a new bank account. Everything appears ready for approval.

But who confirmed the account?

In this hypothetical example, nobody did. The assistant read an unverified email several days earlier and saved its contents as a supplier fact. When it prepared the pack, it described that remembered claim as a previously approved change.

The assistant has not transferred money. It has nevertheless changed the information on which a person may rely. That is why AI security needs to cover the decisions an assistant influences, as well as the actions it performs.

Our view is that a business should treat saved AI memory as information with a source, an owner and limits on its use. Convenience should not turn an unverified statement into organizational authority.

The value of memory comes with a responsibility

Some AI applications save preferences, notes or summaries and reuse them during later tasks. This is what we mean here by persistent memory. It does not require the underlying model to be retrained after every conversation, and not every product uses memory in the same way.

Used appropriately, memory can reduce repeated explanations and help employees pick up unfinished work. The business benefits from continuity. The responsibility is to preserve the distinction between an observation, a request and an approved record.

Consider the difference between “this supplier requested a bank change” and “this supplier’s bank change was approved.” The wording is similar; the business consequences are not. A useful summary must retain that distinction.

Researchers have demonstrated ways to manipulate agent memory so that hostile instructions influence later sessions.^[1] Our supplier scenario illustrates the broader decision problem. An outdated note may be a quality issue; deliberate manipulation to redirect future work is a security concern. Both require correction, but an attack also requires investigation.

Decide what memory is allowed to mean

A smaller organization does not need an elaborate AI program to make this decision. Start with the information that could affect money, customer commitments, confidential records or access to services.

For the manufacturer in our example, finance remains responsible for approving supplier changes. The assistant can organize requests and highlight discrepancies. It can remember that a change was requested, but only the controlled finance record establishes which account is approved.

This arrangement preserves much of the assistant’s usefulness. It also gives the employee reviewing the pack something concrete to verify. A general instruction to “check the AI’s work” is less useful than specifying which record settles a conflict and who may change it.

The same principle applies to remembered discounts, customer consents, contractual commitments and employee access. The question is not simply whether the assistant remembers correctly. It is whether the remembered statement is entitled to influence this particular decision.

Three decisions to make with your team

1. Agree what may be retained. Ask the business owner and provider to identify what the application saves and who can use it. Keep sensitive approvals and authoritative records in systems with appropriate control. Retain an unverified claim only where there is a defined reason and a clear status.
2. Set the rule for conflicting information. Decide what happens when memory disagrees with a current business record. In the supplier example, the assistant should flag the discrepancy for verification, rather than present the remembered account as approved. An AI-generated statement cannot replace a required approval.
3. Ask for a demonstration across two conversations. Have your technical team introduce a harmless, synthetic claim, open a fresh conversation and show how the assistant uses it. Then ask them to correct or remove the entry and demonstrate the later result. A safe first response alone does not establish what was saved.

OWASP’s agent-security guidance supports validation of saved information, separation between users and appropriate retention limits.^[2] The leadership task is to translate those controls into a clear rule for the workflow you intend to use.

A practical starting point

Choose one assistant and one consequential type of information. Ask its owner to explain where that information comes from, how its status is preserved and what happens when it is disputed. If those questions cannot be answered, narrow the use of memory while the gap is resolved.

The objective is useful continuity that employees can trust for a defined purpose. That trust should rest on traceable information and dependable business controls.

Before a remembered claim influences tomorrow’s decision, can your team establish who verified it?