Making Composio production-ready

Composio is an aggregator. You integrate once and your AI can suddenly act in Gmail, Google Calendar, Linear, Notion and GitHub, without you writing a line against any of those APIs. That part took an afternoon.

Then came four months of building everything around it. None of this is Composio's failing. Reach is the product, and the reach is real. But reach is not the same as something you would let loose on a person's actual email, and the gap between those two is made of a lot of small parts. Here they are.

Deciding what the AI is even allowed to see

GitHub offers 893 separate actions through Composio. Gmail offers 63. Hand those over as they come and you have given a model a delete button because it was in the box.

So every connected app gets a list we maintain: which of its actions we offer, sorted by what kind of thing they do (reading, creating, editing, deleting, sending), with the refusals written down and the reason next to them. That list is enforced twice. Once when the AI goes looking for a tool, so it can't find what we don't offer. Once again at the moment of execution, so naming a forbidden action directly doesn't work either.

Letting people set the rules, without 893 switches

The same five categories became the settings. Nobody wants a toggle per action, and it isn't the question people actually have. The question is coarser: can it read my mail, can it send mail. So there's one setting per app per category, and every action inside inherits it.

Sending is its own category for one reason. It reaches other people under your name and there is no undo.

The distinction that keeps this from getting confusing: what we offer is our decision, whether it asks first is yours. Something we don't offer isn't reachable and has no setting. A setting only ever decides whether we pause and show you the action before it runs. So a bug in the settings can't widen what the AI can touch. The worst it does is skip a confirmation.

Not showing the AI everything at once

Every tool description you put in front of a model gets read on every single request, whether or not it's relevant. With hundreds of actions available that gets expensive fast.

So the connected apps sit behind a search step: the model asks for the kind of tool it needs, gets that one, and calls it. Our own features stay directly visible, because those are the common path. The result is a constant three entries for all connected apps, however many you've connected, instead of a list that grows every time someone links a new account. It took about 10,000 tokens off every request.

The cost is one extra round trip when the AI writes to a connected app. Those are rare compared to everything else, so the smaller request pays for itself many times over.

Teaching it your names

The AI shouldn't be handling raw IDs. So we keep a copy of the named things in each workspace, your Linear teams and projects, your pinned Notion pages, and let the model work in names while we swap in the IDs at the last moment.

The policy differs per app, deliberately. A Linear workspace is bounded, so we fetch all of it. A Notion or GitHub account can reach thousands of things, so those start empty and fill only with what you pin.

Checking its work before anything leaves

Here's the failure that started this one. Someone says "file a bug for the Windows team". There is no Windows team. The model picks "Roma Web", which starts with the same letter, and files it. Silently. Same shape when "assign this to me" lands on a colleague.

Prompting did not fix that. A model that is confidently wrong is not listening to instructions about being careful.

Our first real attempt was mechanical: before writing, require that the name the model chose shares a word with what the person actually said. That catches the Windows case immediately, and breaks on everything indirect. "Assign it to me" shares no word with my own name. "The one you just used" shares no word with the page it means. Every miss wanted its own exception, and the exceptions kept multiplying.

What replaced it works backwards from a better question. When we swap a name for an ID, we record why we picked it: the person said it out loud, or they referred back to something used earlier in the conversation, or they said "me", or they gave no name at all and the model had to choose something. Or none of those, and it was a guess. Only the guess gets blocked, with an error listing the real options so the model asks instead of trying again.

The word check is still in there. It just went from being the whole test to being one of the signals.

Handling what comes back, twice

A tool's response has two audiences that want opposite things, and it took us a while to stop treating them as one.

The screen wants something small and predictable. So every response gets converted into a single standard shape, which means the app can draw a result card for any action from any app without us hand-writing one per action.

The model wants the response too, for its next step, and there the problem is size. Responses pile up across a chain of calls, and that pile is the single biggest cost in a turn. One Notion writing session hit 213,000 tokens of accumulated history and blew straight through our provider's rate limit. So the model's copy gets cut down, per app, and only where an app is measurably wordy.

The exception we learned the hard way: we do not cut down Composio's own responses. We tried it the same week. Its search results include suggested steps that look like padding, and removing them broke the harder cases, because the model had been leaning on an instruction buried in there about fetching a page's structure before writing to it. Without it, one job went from a couple of calls to seven or more, rediscovering that on its own. Reverted the same day.

Telling it app-specific things at the right moment

Every app has quirks worth explaining. But anything you put in the standing instructions gets paid for on every request, including the thousands that never touch that app.

So those notes ride along with the search result instead. Notion's quirks only reach the model on a request that went looking for Notion.

Assuming the shape of the data will change

Twice now, a response has quietly changed which field holds the list. A calendar list moved. A team list came back wrapped differently, with no name on the collection at all.

The second one is the one worth telling. Three of those lists get fetched at once, and each is written to fail quietly on its own so that one outage doesn't take down the other two. Teams silently saved zero while members and projects filled in normally. Nothing errored. The only trace was a count on a settings screen nobody had reason to look at, and it sat wrong for months.

Two rules came out of it. Read the actual response from a real call, never the documentation. And anything built to fail quietly needs to say something when it finds nothing in a response that plainly contained something, or you have built a fault you cannot see.

Composio's own notes describe the opposite shape for that same endpoint. Both are real. So we handle both.

The part I'd still want before recommending it

The check that stops the AI writing to the wrong team is the piece I would least want to be wrong about, and it is the piece I have the least evidence for. I know it catches what it was built for. I know the version before it produced false alarms, because one of them was mine. What I don't have is a number, or a test that would tell me if a change quietly broke it.

The same is true one floor down, for the app-specific notes. I can prove they cost nothing. I can't yet prove they help.