"We Can't Check"

Executive Summary

For months, OpenAI told a federal court that it had no practical way to determine what ChatGPT was reproducing from its training data. The argument was technical: the model generates probabilistic outputs, so checking whether any specific output matches any specific training input is computationally infeasible. The court largely accepted this framing.

Then the New York Times filed a motion alleging that OpenAI had, in fact, already searched its own training data, retained 78 million user conversations as part of that process, and deleted outputs that the court had ordered it to preserve. These are the Times's allegations -- OpenAI disputes them -- but the filing reframes a fundamental question about AI accountability.

What does 'we can't check' actually mean when an AI company says it? Does it mean the technical capability doesn't exist? Or does it mean the company would prefer not to look?

This distinction matters for every user of cloud AI. If the company hosting your AI cannot -- or will not -- audit what its model does with data, then the only architecture where you can verify what happens to your inputs is one where the model runs on your hardware, against data you control, with logs you own.

1. What OpenAI Told the Court

1.1 The 'Black Box' Defense

In the New York Times v. OpenAI copyright case, OpenAI's central technical argument was straightforward: large language models don't store or retrieve training data the way a database does. They learn statistical patterns across billions of parameters. Asking whether a specific output 'came from' a specific training document is, they argued, like asking which raindrop caused the river.

This is technically defensible as a description of how transformers work. A model with 175 billion parameters does not contain a lookup table of its training corpus. It contains learned weights that encode patterns across that corpus. The relationship between any single training document and any single output is genuinely probabilistic and diffuse.

OpenAI used this argument to resist discovery requests. The Times wanted to see training data, model outputs, and internal testing. OpenAI argued that producing such materials would be technically burdensome and ultimately uninformative, because the model's behavior could not be traced back to specific inputs in any meaningful way.

1.2 The Court's Initial Acceptance

For a period, this argument worked. The technical complexity of large language models is genuine, and courts are understandably cautious about ordering discovery that may not yield useful evidence. The 'black box' framing positioned OpenAI as a company that would cooperate if it could, but was constrained by the fundamental nature of its technology.

This framing held until the Times alleged otherwise.

2. What the Times Alleges

2.1 Training Data Searches

According to the Times's court filing, OpenAI had already conducted internal searches of its training data -- the very data it told the court was impractical to search. The implication is that OpenAI had the technical infrastructure to query its training corpus, had used that infrastructure, and had not disclosed this capability to the court when arguing that such searches were infeasible.

If true, this means the 'black box' defense was not a description of technical reality. It was a litigation strategy.

2.2 78 Million Conversations

The filing also alleges that OpenAI retained approximately 78 million ChatGPT user conversations. The context matters: OpenAI's public messaging around data retention has emphasized user control, conversation deletion, and opt-out training. The existence of a 78-million-conversation archive raises questions about what 'deletion' means in practice when a company's business model depends on training data.

For every ChatGPT user who assumed their conversations were ephemeral, this number is a data point worth considering.

2.3 Deleted Evidence

The most serious allegation is that OpenAI deleted outputs that the court had ordered it to preserve. In federal litigation, once a preservation order is in effect, destroying relevant evidence is sanctionable -- potentially case-ending. If the court finds that OpenAI destroyed preserved materials, the consequences could range from adverse inference instructions (the jury may assume the destroyed evidence was harmful to OpenAI) to default judgment.

OpenAI disputes these allegations. The case is ongoing. But the filing itself changes the conversation about AI transparency in a way that affects every user, not just the litigants.

3. The Transparency Gap

3.1 'Can't Check' vs. 'Won't Check'

The distance between 'we technically cannot determine what our model does with your data' and 'we have the tools to check but would prefer not to disclose what we find' is the entire transparency gap in cloud AI.

Every major AI provider makes some version of the first claim. Training data is too large to audit. Model behavior is too probabilistic to trace. Outputs cannot be reliably attributed to inputs. These claims are partially true -- the probabilistic nature of language models is real -- but they are also convenient. A company that genuinely cannot audit its model's behavior has no obligation to report what it finds, because by definition it finds nothing.

The Times's allegations suggest that the 'can't' framing may obscure a 'won't.' If OpenAI had tools to search its training data, the question is not whether auditing is technically possible. The question is whether AI companies have incentives to audit -- and whether their users have any way to verify the answer.

3.2 The User's Position

When you use a cloud AI service, you occupy a specific position in the information architecture: you provide input, you receive output, and you have no visibility into what happens in between.

You cannot verify:

• Whether your input was used for training
• Whether your conversation was retained after you 'deleted' it
• Whether the output contains material from copyrighted training data
• Whether the model's behavior on your input was logged, analyzed, or shared
• Whether any of the provider's public claims about data handling match their internal practices

This is not a flaw in cloud AI. It is the architecture. The model runs on the provider's servers, processes data in the provider's memory, and stores logs in the provider's databases. The user's only interface is the input/output boundary. Everything behind that boundary is the provider's unilateral domain.

3.3 Trust Without Verification

The security community has a phrase for this: trust without verification. It means you are relying entirely on the provider's self-reporting about their own behavior. You trust that they delete what they say they delete. You trust that they don't train on what they say they don't train on. You trust that their technical limitations are genuine and not strategic.

The Times's filing is significant precisely because it alleges that at least one major provider's self-reporting did not match its internal practices. Whether the allegations are ultimately proven true is a question for the court. But the existence of the allegations highlights a structural problem: in cloud AI, users have no independent means of verification.

4. What This Means for Professionals

4.1 The Data You Feed Your AI

Knowledge workers in 2026 routinely paste sensitive material into AI chatbots: legal briefs, financial analyses, medical notes, strategic plans, client communications, proprietary code. Each paste is an act of trust. The user trusts that the provider will handle the data according to its stated policies.

But stated policies are not auditable by users. And if the Times's allegations hold, stated policies may not even be auditable by courts.

For professionals in regulated industries -- attorneys, financial advisors, healthcare providers -- this is not an abstract concern. It is a compliance exposure. If you cannot verify that your AI provider actually deletes data it claims to delete, you cannot certify compliance with data retention regulations. You are signing attestations based on someone else's unverifiable claims.

4.2 The Conversation Archive Problem

78 million conversations is not a rounding error. It is an archive. And archives have properties that individual conversations do not: they can be searched, they can be analyzed in aggregate, and they persist beyond any individual user's awareness.

If you have used ChatGPT for any purpose involving non-public information, the existence of a large conversation archive means your data exists in a context you did not choose and cannot audit. The data may be anonymized. It may be encrypted. It may be access-controlled. But 'may' is not 'verified,' and you have no mechanism to verify.

4.3 The Evidence Destruction Question

For organizations considering AI deployment in contexts where data may be subject to legal hold -- essentially any organization with active or anticipated litigation -- the allegation that a major AI provider destroyed court-ordered evidence is a material risk factor. If your AI provider is capable of destroying data it was ordered to preserve, the question of what it does with data it was not ordered to preserve becomes significantly more urgent.

5. The Architecture That Solves This

5.1 Local-First AI

There is exactly one architecture where you can independently verify what happens to your data: one where the data never leaves your machine.

Local-first AI means the model runs on your hardware. Your inputs are processed in your memory. Your conversations are stored in your database. Your logs are written to your filesystem. There is no remote server to trust, no provider to rely on for self-reporting, and no conversation archive accumulating on someone else's infrastructure.

This is not about distrust. It is about verifiability. In a local-first architecture, every claim about data handling is auditable by the user because the user controls the entire stack.

5.2 How Kent Implements This

Kent's Private Mode runs inference locally via Ollama on the user's machine. Zero network calls. Zero data transmission. Zero conversation logging on remote servers. The model runs on localhost, the knowledge graph lives in a local SQLite database, and the logs are plain text files on the user's filesystem.

When a user asks Kent a question in Private Mode:

1. The input stays on the local machine
2. The model processes it in local memory
3. The output is returned through local IPC
4. The conversation is stored in the user's local database
5. No network request is made at any point

Every step is verifiable. The user can inspect the network traffic (there is none). The user can read the database directly. The user can grep the log files. The user can run a packet sniffer and confirm zero outbound connections.

This is what verifiable data handling looks like. Not a privacy policy. Not a terms-of-service update. A network monitor showing zero packets.

5.3 Cloud When You Choose, Local When It Matters

Kent does not force users to choose between cloud capability and local privacy. Cloud mode accesses Anthropic, OpenAI, Gemini, and other providers when the user explicitly opts for it. Private Mode runs locally when the user decides the data should not leave their machine.

The key is that the choice is the user's. Not the provider's. Not the provider's legal team's. The user decides, per query, whether the data leaves the machine. And when they choose local, they can verify the decision was honored.

6. The Broader Pattern

6.1 This Is Not Just About OpenAI

The Times v. OpenAI case is the most visible example, but the underlying dynamic applies to every cloud AI provider. When a company operates the model, stores the data, and self-reports its practices, users are structurally unable to verify any claim the company makes about data handling.

Google's Gemini processes data on Google's servers. Anthropic's Claude processes data on Anthropic's servers. Every cloud AI interaction is, by architecture, a trust exercise. The question is not whether you trust any specific company. The question is whether trust-without-verification is an acceptable architecture for sensitive data.

6.2 Regulation Is Not a Solution

Regulation can impose penalties for violations. It cannot prevent them. GDPR has been in effect since 2018, and data breaches, unauthorized retention, and policy violations continue to be discovered years after they occur. The EU AI Act adds AI-specific requirements, but enforcement depends on detection, and detection depends on transparency that cloud architecture structurally prevents.

Regulation is necessary. But it is not sufficient. The only reliable protection is architectural: keep the data on your machine, and verification becomes trivial.

6.3 The Shifting Definition of 'Delete'

One of the most consequential ambiguities in cloud AI is the definition of 'delete.' When a user deletes a ChatGPT conversation, what actually happens? The conversation disappears from the user's interface. But does it disappear from the company's backup systems? From their training pipelines? From their analytics databases? From the 78-million-conversation archive?

'Delete' in a cloud context means 'remove from the user-facing view.' It does not necessarily mean 'remove from all storage systems, backups, training sets, and derived datasets.' The gap between user-facing deletion and actual data elimination is one of the least examined problems in cloud AI, and the Times's filing brings it into sharp focus.

7. Practical Guidance

7.1 For Individual Professionals

Audit your AI usage. List every type of sensitive data you have pasted into a cloud AI service in the last 12 months. Client names. Financial figures. Legal strategies. Medical information. Proprietary code. For each category, ask: if this data were retained in a 78-million-conversation archive, what would the consequences be?

Adopt a tiered approach. Use cloud AI for general queries where data sensitivity is low. Use local AI for anything involving non-public information, client data, or material subject to regulatory requirements.

Verify, don't trust. If your AI provider claims to delete data, ask for the technical documentation that describes the deletion process. If they claim not to train on your data, ask for the architectural description that enforces this. If neither is available, assume the claims are aspirational rather than architectural.

7.2 For Organizations

Reassess cloud AI risk. The Times's allegations -- whether ultimately proven or not -- demonstrate that AI providers' self-reported data practices may not match their internal practices. Update your risk assessment to account for this possibility.

Implement local-first options. For use cases involving regulated data, client information, or material subject to legal hold, deploy local inference capability. The cost of running a local model is a fraction of the cost of a data breach or compliance violation.

Document your AI data flows. For any cloud AI service, document exactly what data enters the service, what the provider claims happens to it, and what verification mechanisms exist. If the verification column is empty -- which it will be for every cloud service -- that is your risk surface.

7.3 For the Industry

Build for auditability. The next generation of AI infrastructure should treat verifiable data handling as a core architectural requirement, not a policy afterthought. Users should be able to independently verify every claim a provider makes about their data.

Separate capability from custody. The AI model and the user's data do not need to live on the same infrastructure. Architectures that bring the model to the data -- rather than the data to the model -- eliminate the custody problem entirely.

Expect the litigation to expand. The Times case is the beginning, not the end. As more training data practices are disclosed through discovery, the gap between public messaging and internal practices will either narrow (if the practices were always compliant) or widen (if they were not). Either way, the companies that built for transparency from the start will be the ones still standing.

Conclusion

OpenAI spent months telling a court that it could not check what ChatGPT reproduces from its training data. The New York Times now alleges that OpenAI had already searched that data, retained 78 million user conversations, and destroyed evidence it was ordered to preserve. OpenAI disputes these allegations, and the case continues.

But the question the case raises is bigger than either party. When an AI company says 'we can't check,' what does that actually mean? And when the only way to verify the answer is to take the company at its word, is that acceptable for sensitive data?

The local-first alternative is not a retreat from AI capability. It is a demand for verifiable architecture. Run the model locally. Store the data locally. Log everything locally. Verify everything independently.

You should not have to trust a company's litigation strategy to know what happened to your data.

This article discusses allegations made by the New York Times in court filings in New York Times v. OpenAI (S.D.N.Y.). OpenAI disputes these allegations. The case is ongoing, and no court has made findings of fact on these matters. Source: TechCrunch, July 9, 2026.

Kent Research | July 2026