Context is All or Nothing

I have an obsession with giving my agents as much context as possible. Why? Every gap could lead to a bad experience or misinformed decision and when individual agents are getting less and less supervision that becomes really hard to debug.

The context iceberg

The problem is that certain types of context are much harder to access and give up than others. I believe the iceberg analogy is apt here:

• Top: Slack, Email, Linear, etc
• Mid: production logs, internal systems, etc
• Lower mid: WhatsApp, Telegram, iMessage
• Bottom: your location, your bank accounts, threads from colleagues’ agents, personal documents (ID, passport)

There’s two factors that make a context area move lower and lower in the iceberg:

1. Data sensitivity: how bad would it be if this leaked?
2. Ease of integration: how do I actually get this context to my agents?

Data sensitivity

This is a tough one for me. I live in Germany which is possibly the most data privacy focused country on earth with GDPR, the EU AI act and hundreds of other structures in place to ensure that your data stays under your control. The reality is though that all the serious AI labs are based in the US and as a consumer you generally don’t have much control over data residency.

My take is that it’s simply unavoidable in this day and age to have most of your private information uploaded somewhere, where you have very little say in what happens to it. The alternatives are, to be frank, bloody inconvenient. So what’s the decision here? I think it comes down to if you mistrust the AI labs significantly more than big tech. Do I trust Anthropic less with my WhatsApp chats than Meta? Not really, if I’m honest. Of course with LLMs there’s the added dimension of model training (despite the labs mostly claiming they won’t). My controversial thought here is that you actually want to be in the weights of LLMs as much as possible, and it will be perceived as beneficial in the coming years.

Ease of integration

The majority of standard SaaS has an official MCP server with a one-click OAuth flow, and it takes about 2 minutes to set up. It’s typically where the “official” version of stuff lives but in my opinion is becoming an increasingly weak signal due to slop.

Below that we get into the actual sources of truth: prod logs, databases, internal tools. This is usually a bit tougher because engineering and security have to get involved. Someone has to set up a read-only replica, scope some tokens, wire up a logs MCP and make sure nothing sensitive is exposed and so on.

Then it gets tough. WhatsApp, Telegram and iMessage have no official way in for your personal account. What you do have is sketchy unofficial libraries that might get your number banned, bridges that sit in the middle of your end-to-end encryption or manual labour.

At the bottom there’s some fundamental constraint (either perceived or technical):

• Location is technically trivial, but are you okay with your location being accessible at any time?
• Bank accounts: in the best case it’s super annoying to set up but for many it’s basically impossible.
• Personal documents: also trivial, but if there were to be a leak it would be bad.
• Your colleagues’ agents: both a technical and social challenge.

All or nothing

Partial context doesn’t give you a partially right agent. It gives you an agent that’s wrong. And because most of us aren’t in the business of reading through every tool call / agent trace, we never quite know why and the experience simply gets worse.

Some examples:

• The most annoying tasks in life require sensitive information (tax, visa forms etc).
• My agent stalls because it needs input from me. I’m driving so there’s no way it will get the info. If it could see my location, it would know that. It could pick the safe option, or move on to something else and come back later.
• It books a 9am call the morning my flight lands. My work calendar says I’m free. The flight confirmation is in my personal inbox.
• It picks the wrong bug to fix. The ticket looks trivial / low-impact but the enterprise customer texted our founder and is going to churn over it.
• It spends an hour rediscovering something which a colleague’s agent root-caused yesterday.

The models are smart enough to do the right thing in all of these cases, we simply don’t let them.