• Taylor Brooks
• 29 Mar, 2026
• • Building in Public

Most AI Agents Aren't Actually Agents

Everyone's building "AI agents" right now. The timeline is full of them. Companies are raising millions to ship them. The problem? Most of them aren't actually agents. They're chatbots with API access. That's it. What People Call Agents Here's the pattern I see everywhere:User types a message LLM decides which function to call Function returns some data LLM formats a response DoneThat's not agency. That's function calling with a conversational wrapper. The LLM picks a tool, the tool runs, the result comes back. If it works, great. If it breaks, the conversation dies. If the user needs three things done in sequence, they're manually prompting through each step. This is useful. It's even impressive sometimes. But it's not an agent. What Real Agents Need Real agentic systems operate with autonomy. They handle the messy parts without constant human supervision. That means: Error recovery. When something breaks (and it will), the agent doesn't just apologize and give up. It retries with backoff. It falls back to alternative approaches. It routes around failures without making the user debug what went wrong. State management. The agent needs to remember what it's doing across multiple tool calls. Not just "what did the user ask for?" but "what have I tried, what worked, what's left to do, and what's blocking me right now?" Retry logic. APIs timeout. Rate limits hit. Sometimes data isn't ready yet. A real agent knows when to try again, when to wait, and when to give up. Supervision and checkpointing. For multi-step work, the agent should be able to pause, show you what it's done so far, and resume if something goes sideways. You don't want it to redo 20 steps because step 21 failed. Context persistence. If the system restarts, the agent should be able to pick up where it left off. Not "sorry, you'll need to start over." Graceful degradation. When a preferred tool is unavailable, the agent should try another approach. When data is incomplete, it should work with what it has or ask for the missing pieces. This is infrastructure work. It's not fun. It's not what people demo. But without it, you don't have an agent. You have a chatbot that calls APIs. The Infrastructure Problem The hard part of building agents isn't the LLM. That's the easy part. The hard part is everything around it. You need a task queue that can handle retries. You need a way to checkpoint progress so work doesn't get lost. You need monitoring so you know when an agent is stuck. You need logging so you can debug failures after the fact. You need to handle rate limits from every API your agent touches. You need to deal with inconsistent error responses. You need to decide what to do when a tool returns malformed data or no data at all. You need a way to supervise long-running workflows. You need to surface status updates without spamming the user. You need to decide when to ask for help and when to keep trying. None of this is LLM work. It's systems engineering. What I'm Seeing in Practice I build agents daily. The pattern is always the same. I spend 10% of my time writing prompts and configuring LLM calls. I spend 90% of my time on infrastructure:Handling tool failures Managing state across multiple turns Implementing retry logic Building supervision layers Writing recovery flows for when things go wrongThe prompt is never the problem. The problem is making the system robust enough to actually finish the job. When I look at "AI agent" demos online, I see polished function calling. I don't see error handling. I don't see state management. I don't see retry logic. That's fine for demos. It's not fine for production. The Real Opportunity If most "AI agents" are just chatbots with API access, there's a huge opportunity for anyone willing to build the infrastructure. The companies that win won't be the ones with the best prompts. They'll be the ones with the most resilient execution layers. They'll build systems that:Recover from failures without human intervention Maintain state across sessions and restarts Coordinate multi-step workflows reliably Degrade gracefully when things break Surface meaningful status without overwhelming usersThis is less glamorous than training models or writing clever prompts. But it's what separates working agents from chatbots. LinksOpenClaw - agent infrastructure I'm actively building with LangGraph documentation - one approach to stateful agent workflows Modal - infrastructure for long-running agent workloadsBuilding agents that actually work means caring more about the infrastructure than the LLM. The wrapper matters more than the model. Most people aren't ready for that conversation yet.