The Interface Gap: Why LLMs Still Struggle with OpenAPI

When large language models started calling APIs, it looked like the future had finally arrived.  Agents could use live data, trigger actions, even close support tickets without a human in the loop. Yes! We, for one, welcome our robot overlords.  And there were already thousands of APIs documented in OpenAPI. Surely that would mean  instant compatibility.

Annd, there was that record scratch moment. Because, unfortunately, even with beautifully written specs, agents often choke. They call the wrong endpoint, they miss a required field, or they just full-on fail to understand what the API is for. Are our robot overlords bad at following instructions? Or is the problem more foundational? 

The truth is, unfortunately our interfaces were not built for reasoning systems. OpenAPI provides a predictable structure for endpoints and methods, machine-readable schemas for parameters and responses, and auto-generated client libraries and documentation. But LLMs need to know more than just “what” an endpoint is - they need to know when and how to use it, and how the output might fit into the surrounding tasks.  That’s the interface gap: the missing semantic layer between mechanical description and cognitive intent.

Why Agents Guess… And Get It Wrong

Let’s take a peek at two nearly identical endpoints: 

# OpenAPI excerpt
paths:
  /users/search:
    get:
      summary: Search for users by name
  /users/find:
    get:
      summary: Retrieve user by email

‍

To a human developer, the difference between these two snippets would  be intuitive: one queries by name, the other by email. Unfortunately, an LLM sees both of these as GET requests. Hey, both mention “users,” so it’s pretty close, right? Neither includes any embedded clue about intent or context. So what is the agent going to do? It’s not going to reason its way to the right choice. It’s going to make its best probabilistic guess, which may mean that it gets it right, but could also mean calling the wrong function or straight-up hallucinations. On an enterprise scale, when you’re dealing with hundreds, if not thousands, of endpoints across multiple services, those small mistakes will start to add up. 

Back to the human in the loop. We add wrappers with helpful docstrings. We inject system prompts that say things like, “If the user wants to find a customer by email, use /users/find.” We stuff the context into YAML, JSON, Markdown, or whatever the orchestration framework will let us get away with.

It works for a hot minute, until  the API changes, or the model updates, or a new tool is added to the stack. Hours of patching code, prompt tuning, and trying not to explode as the LLM sheepishly agrees “You’re right!” and then goes right back to hallucinating its next bit of non-working code.

What Agents Actually Need From an Interface

To an agent, a list of endpoints is just that: a list. What’s missing is why each one exists, when to use it, what it connects to, and what constraints apply. Without that kind of context, agents are just slotting inputs into patterns they’ve seen before.

Now imagine if instead of just reading an OpenAPI spec, your agent had access to something like this:

{
  "name": "find_user",
  "description": "Retrieve a user account by email address",
  "intents": ["customer_lookup", "account_recovery"],
  "constraints": {
    "auth_scope": "read:users",
    "safe_to_call_first": true
  }
}

This goes beyond syntax into semantics. You’re telling the agent what the tool means in business terms, when it should be used, and under what conditions it’s safe to run. You’re making intent explicit.

When interfaces are built this way, agents don’t have to guess. They can reason, decide between similar tools, chain them logically, and avoid calls they’re not authorized to make. You don’t have to jam all that context into a prompt or hope your system prompt survives token truncation. The meaning travels with the method.

From Endpoints to Intent: Rethinking the Interface Layer

We’ve written before about why MCP is essential for agentic AI, but here’s the TL;DR: APIs were built for programs. Programs are deterministic systems with perfect memory, byte-level precision, and strict control flow. Sadly, our (robot overlords) LLMs are none of those things. LLMs are goal-driven, language-based, and probabilistic. 

So for what’s needed for smooth agent to enterprise integration is a semantic interface layer that adds:

• Purpose metadata – what a tool actually does in business terms
• Usage constraints – when and how it should be called
• Cross-tool relationships – which actions logically follow others
• Governance hooks – security, access, and observability baked in

The Model Context Protocol has reimagined what an interface is when the consumer is a reasoning system. MCP defines tools in terms of their purpose, not just their payloads. With MCP, integration becomes something more semantic, declarative, and adaptable.

• Tools can be selected based on natural language intent
• Inputs are labeled with user-facing concepts
• Execution is abstracted and mediated
• Outputs are structured for follow-up

In an MCP-enabled world, agents don’t have to reverse-engineer the API’s logic from syntax. They can rely on interfaces that describe intent, expose constraints, persist relevant context, and return structured results that make downstream decisions easier.

Integration for the Age of Agents

If the past few years were about getting LLMs to talk to tools, the next wave is going to focus on designing tools they can actually use. That shift will require thinking differently about what an interface should offer. We’re going beyond structure and access to meaning and alignment. 

Imagine an interface that speaks the same language as your model, while surfacing constraints, relationships, and context. This is the kind of foundation that enables reasoning systems to integrate cleanly, act reliably, and scale intelligently. Pretty soon we’ll also have the jetpacks we were promised!

‍