What a forward deployed engineer actually does, and why AI made the role matter

People keep asking me what is a forward deployed engineer, usually right after I have spent a week inside their codebase fixing an agent that looked perfect in the demo and fell apart on real data. Here is the honest version of the job.

It is a Tuesday and I am sitting in a client's Slack, watching their support agent confidently tell a customer that a refund was processed. It was not. The tool call failed silently three weeks ago and nobody noticed because the happy path still looked happy. That is the moment the job actually starts. Not the build. The part after, where the thing you shipped meets the mess of a real business.

So when people ask me what is a forward deployed engineer, that scene is my answer. A forward deployed engineer is someone who embeds inside a client's team, ships working software into their production environment, and then stays close enough to the users to catch the failures that never show up in a spec. The name comes from the military idea of forward deployment, putting capability at the edge instead of back at headquarters. Palantir made the title well known. AI made it matter to everyone else.

The definition most people get wrong

A forward deployed engineer is not a solutions architect who draws diagrams and hands them to a delivery team. It is not a sales engineer who demos and disappears. And it is not consulting in the slide deck sense, where you leave a PDF of recommendations and an invoice. You write the code. You deploy the code. You own whether it works in front of real users, on their infrastructure, with their weird data. If it breaks at 6pm on a Friday because their CRM returns a null where you expected a string, that is your problem, not a ticket for someone else.

The distinguishing trait is proximity. A normal product engineer is one or two layers removed from the person using the software. A forward deployed engineer sits next to them. You hear the actual complaint, in the actual words, before it becomes a sanitized feature request that lost all its useful detail on the way up the chain. That closeness is the whole point. It is also the hard part, because you cannot hide behind a backlog.

Why AI agents made this role suddenly matter

For years you could build software in a clean loop. Gather requirements, build, test against those requirements, ship. The behavior was deterministic. Given the same input you got the same output, and if it passed the tests it mostly held in production. You could throw it over a wall and it would stay standing.

AI agents broke that loop. An LLM does not do the same thing twice. You cannot fully specify its behavior up front, and you cannot test your way to confidence with a fixed suite because the input space is every sentence a human might type. The demo works because the demo is three curated prompts. Production is ten thousand messages a day, and somewhere in there is a customer writing in broken English asking for something your prompt never anticipated, and the model, trained to be helpful, invents an answer.

This is why the build then hand off model falls apart with agents. The failures are behavioral, not just functional. They live in the gap between what you imagined users would do and what they actually do. You only close that gap by being there, reading the transcripts, watching the traces, and tuning against reality. That work does not fit in a statement of work with a clean end date. It needs someone forward deployed.

What the job actually looks like day to day

The first week is mostly listening and reading. Their codebase, their data, their support tickets, the three failed automation attempts they are slightly embarrassed about. I want to know where the bodies are buried before I write a line. Usually the real problem is not the one they hired me for. They think they need a smarter model. They actually need their product data cleaned up and a retrieval layer that returns the right document instead of a plausible looking wrong one.

Then I build the smallest thing that touches production. Not a prototype in a notebook. A narrow slice that runs against real traffic, even if it is shadowed and does nothing but log what it would have done. I would rather ship an agent that handles one intent correctly than a general one that handles twenty intents at seventy percent. Seventy percent is a disaster in customer facing work. It means one in three people gets a wrong answer with total confidence.

The tools are boring on purpose. OpenAI or Claude for the model, depending on the task and what the client already pays for. Postgres with pgvector for retrieval, because most companies do not need a dedicated vector database and do need one fewer system to operate. n8n or a few cron jobs for the orchestration, LangGraph when the control flow genuinely has branches and loops worth modeling as a graph. Zendesk or their existing help desk as the surface, so the agent lives where the humans already work instead of in yet another tab nobody opens.

The part everyone skips: observability

You cannot fix what you cannot see, and an agent's reasoning is invisible by default. The first real infrastructure I add on almost every engagement is logging that captures the full picture of every run: the input, the retrieved context, the tool calls, the raw model output, and whether a human had to step in. Without this you are debugging by vibes. With it you can go from a customer complaint to the exact trace in under a minute.

def log_agent_turn(session_id, user_msg, retrieved, tool_calls, output, escalated):
    # one row per turn. boring, and it saves you every single time.
    db.execute(
        """
        insert into agent_traces
            (session_id, user_msg, retrieved_ctx, tool_calls,
             model_output, escalated, created_at)
        values (%s, %s, %s, %s, %s, %s, now())
        """,
        (session_id, user_msg, json.dumps(retrieved),
         json.dumps(tool_calls), output, escalated),
    )
    # anything the model refused, hedged, or escalated gets sampled
    # into a review queue a human actually reads each morning.
    if escalated or looks_uncertain(output):
        review_queue.push(session_id)

That review queue is where the real improvement comes from. Every morning someone reads the escalations and the low confidence answers, and those become the next round of prompt fixes, new tool guardrails, or a note that a whole category of question should never be automated at all. That last one is important. Part of the job is telling a client that no, this should stay human, and here is the data showing why.

The stance I bring to every engagement

Ship narrow, then widen. A working agent for one workflow beats an impressive demo for ten. I will say no to scope that sounds good in a meeting and dies in production. And I treat the model as the least reliable component in the system, because it is. The reliability comes from everything around it: the retrieval, the validation, the fallback to a human, the loud alert when a tool call fails instead of a silent shrug like the refund bot at the top of this post.

One more opinion, and I will hold it firmly. The value of a forward deployed engineer is not the code. Anyone can generate code now. The value is judgment applied in context, the hundred small decisions about what to automate, what to leave alone, where the model will embarrass you, and when good enough is actually good enough. That judgment only exists close to the work. You cannot outsource it to a spec, and you definitely cannot outsource it to the model itself.

So, what is a forward deployed engineer, really

It is the person who moves toward the problem instead of away from it. Who ships into the mess and stays until the thing holds. AI did not invent the role, but it made the role unavoidable, because agents fail in ways that only reveal themselves in production, in front of real people, on a random Tuesday. Someone has to be standing there when they do.

I like the work because it is honest. The software either helps a real person do their job or it does not, and you find out fast. There is no wall to throw it over. There is just you, the users, and whatever the model decided to say next.