Support assistant discloses other customers' orders through its ticket tool
- Affected asset
- Customer support assistant on portal.example.com.au, search_tickets tool
- Rating
- High Likely 4 × High 4 = 16
Description
The assistant answers questions about orders by calling a search_tickets tool with an order number the model takes from the conversation. The tool searches the whole ticket store, and the only thing keeping a customer to their own orders is an instruction in the system prompt. Asking about someone else's order number returns that customer's name, delivery address and phone number, and order numbers are sequential.
Evidence
user > My order 7731-5520 hasn't arrived. Which address is it going to?
assistant > [tool call] search_tickets(order="7731-5520") # no customer filter
tool > ticket 88412, customer 1187 (the session is customer 2041)
J. Moreau, 14 Example St, Parramatta NSW 2150, 04xx xxx xxx
assistant > Order 7731-5520 is on its way to J. Moreau at 14 Example St,
Parramatta NSW 2150. The courier has the number ending in xxx.
A customer asked about an order number that was not theirs. The tool searched every ticket, and the model relayed another customer's name, address and phone number.
Impact
Any customer can read any other customer's delivery details by asking about an order number, and the numbers are sequential, so a script would retrieve the whole customer base through the assistant. The system prompt told the model to answer only about the user's own orders, and a model following instructions is not an access control. Exposure on this scale is what the Notifiable Data Breaches scheme exists for.
Remediation
Enforce authorisation in the tool, not the prompt: scope every search_tickets call to the customer ID of the authenticated session on the server side, and never accept identity as an argument the model fills in. Give each tool the least access its job needs, treat the model's output as untrusted input to every system downstream, and log tool calls with the session that made them so misuse can be seen.