I put an agent inside chrome. chrome did not like it.

What building Browsergent taught me, which may extend to your harness development.

You can learn more here: https://browsergent.com/

I have been developing a browser extension that runs an agent inside. The motivation is simple: everything else sucks. Every single extension, or Gemini on the top right of your Chrome, feels bad.

They can summarize a page, answer a question, maybe click a few buttons, but none of them feel like Claude Code for the browser.

So naturally I thought, damn I'm a genius, I'm gonna build a much better one.

About one week in, my arrogance met browser engineering, and it hit hard.

Building an agent that clicks a button is easy. Building a harness that lets the model actually work reliably and efficiently without getting completely confused is a whole different beast.

That's what's driving me to write this post, in the belief that sharing it benefits you all, and honestly I think the lessons go beyond browser automation — probably apply to anyone building a harness around an LLM.

The landscape of today's browser automation

Browser automation is facing problems for captcha, fingerprints, and just in general, the interests of Google, Cloudflare and a million other service providers who'd rather your automation not exist.

I tried browser use. It's selfhostable, but simply tedious for the small things I wanted to automate inside the browser I was already using. I didn't want to open another hosted browser, migrate my session, spin up a fresh automation profile just to organize some tabs or fill out a form.

Then it's Atlas or its peers, the visual driven browser use harness. They all face the same problem: we cannot build a repetitive script or skill for tasks we want to automate.

It looks at the page, decides what to click, acts, looks again, repeats. Works surprisingly well for one-off tasks, but good luck turning that into a reliable reusable workflow. Ask it to do the same thing tomorrow and it rediscovers the whole page from scratch.

Burning compute against a chrome-rendered HTML tree over and over is not a sustainable plan.

Then here comes Codex and similar harnesses that embed a browser inside the app itself. Works much better today, and I'd say it genuinely solves a lot of what I was originally trying to solve with Browsergent. ChatGPT's chrome integration for example can just work with a page depending on your already signed in browser state.

Browsergent still holds tiny edges though.

You don't need to open another process, and you can work across multiple tabs and windows simultaneously — the ones already involved in the workflow you already have.

But the thing that actually mattered, way more than I expected, is that it gives the model a real programmatic interface.

Why agentic extensions suck

The biggest reason is all those extensions don't have a programmatic interface.

Unlike Claude Code, or a coding agent running in your terminal, they can't run whatever shell or REPL they want. Have you noticed how many lines, how many operations a single tool call to shell can do at the same time?

It's massive, and more importantly it's reusable. It's the native language of the model.

Most browser extensions instead give you atomic tools: click this, fill this, read the page, open a tab, click again, read the page again.

The model isn't programming the browser. It's remotely operating it, one button press at a time.

When I first noticed this I thought damn, I'm a genius, I'm gonna build a chrome extension with a runtime embedded so the model can actually write automation code instead of pressing buttons one at a time.

Then, as the preface suggests, about a week in I hit the second wall. Chrome Web Store does not allow extensions to embed a programming runtime — or more precisely, Manifest V3 bans building an interpreter that runs complex commands fetched remotely, which was uncomfortably close to exactly what I'd built.

It almost felt personal. Sucks. I almost gave up.

But I was surprised how much the agent could already do sitting right next to my webpage. Felt like a different animal, like the first time I used Cursor's Composer mode (not the post-trained Kimi K2.5 model) — not because the model suddenly got smarter, but because the interface finally let it express what it already knew how to do.

So I kept going.

CDP plugins are good, except for that ugly banner

CDP based tools are actually pretty good. Real programmatic access: the model can inspect the DOM, run JavaScript, watch network traffic, read console output, actually do multiple things instead of reducing everything to screenshot-click-screenshot-click.

But there's always a catch.

With Chrome's modern remote debugging flow you get the lovely "chrome is being controlled by automated test software" banner sitting at the top of your browser the entire time. Technically harmless, emotionally devastating. It never lets you forget you're not using your normal browser anymore — you're driving some suspicious lab edition of Chrome.

Some setups also want you launching Chrome with debug flags, a separate profile, approving debug access, running a local bridge process. None of it's fatal. CDP tools are still very capable. But Browsergent doesn't touch CDP at all: no remote debug port, no external process attached to Chrome, no permanent banner. The runtime and browser APIs live inside the extension itself.

Tiny advantage, probably the smallest competitive moat ever documented in a technical blog post, but when the tool's supposed to live inside the browser you use all day, tiny annoyances turn into actual product decisions.

Lesson 1: make your harness fit the model's habits, not your handbook

If you can't post train your model, do not expect it to work properly on your harness just because you put nice instructions in the context.

I initially put a Lua runtime inside the extension, https://github.com/Irvingouj/web-lua. It works fine, loads pretty fast, but the problem is the model has perhaps in all its training data never used Lua to manipulate the DOM.

It works, but only after bumping around, inventing function names, guessing argument shapes, hitting errors, correcting itself, hitting more errors, and eventually getting there.

Very natural next step, I said to my Codex, or GLM 5.1, can't remember which: fuck it, let's put a JavaScript runtime inside a JavaScript runtime. And god it actually worked, like a hundred times better.

I swapped Lua for a QuickJS runtime and deliberately designed the API to mimic Playwright as closely as I could, also granted almost all Chrome extension APIs pure pass through without transforming the function shape at all.

I don't have a proper bench yet so a hundred times better is currently measured in the highly scientific unit of how annoyed I felt watching the agent work, but the number of failed calls dropped a lot.

The model already knows JavaScript, the DOM, promises, selectors, loops, Playwright style APIs. Instead of spending intelligence learning my weird abstraction it could spend it finishing the task.

Don't design the interface that looks cleanest to you. Design the interface the model already knows.

Lesson 2: the accessibility tree is not the whole web

Accessibility snapshots are genuinely useful — roles, labels, states, names, way cleaner than raw HTML.

But they're not the whole application.

Take file uploads. A lot of React apps hide the real <input type="file"> and show a styled upload button instead. Depending on how it's hidden it might not even show up in the accessibility tree. display: none usually strips it out entirely while other hiding tricks leave it in.

The agent sees a beautiful button labeled "upload résumé", clicks it, a native file picker opens, and the agent spiritually leaves the room.

If your harness only exposes accessibility labels and visible controls — no underlying file input, no file-chooser primitive — the model has literally nothing to attach the file to.

Playwright handles this by letting you target the real input directly with setInputFiles(), or if the input's created dynamically, intercepting the filechooser event instead.

A browser harness needs the same kind of escape hatch.

Accessibility info isn't bad. It's just a deliberately filtered view of the page, built for accessibility semantics not for every implementation detail an automation system might need.

So you need both: a semantic representation for normal reasoning, and access to the underlying document for when that representation isn't enough.

Lesson 3: never feed the entire DOM to the agent

And by never I mean never make the full DOM the default observation.

Dumping an entire modern webpage into context is expensive, noisy, and usually makes the model worse, not better.

A page can have thousands of wrapper divs, hidden menus, duplicate mobile layouts, SVG paths, analytics junk, framework attributes, off screen content, text that has nothing to do with the task.

More context isn't automatically more intelligence. Sometimes it's just a bigger pile of garbage to dig through.

The default should be a selective semantic snapshot: visible text, interactive controls, form fields, nav, meaningful regions, current states like selected/expanded/disabled/invalid, stuff related to the task at hand.

Playwright's aria snapshots do something similar — a compact tree of roles, names, values, states, instead of the full HTML, and it can be scoped to a locator instead of the whole page.

But any snapshot can miss something. That's why it can't be the whole interface. The model also needs to walk the underlying tree dynamically: ask for a subtree, filter by tag, attribute, text, role, visibility, input type, relationship to another element, whether to include hidden stuff.

The snapshot should answer what looks important right now.

The dynamic tree should answer what did the snapshot miss.

Don't make the model inspect the whole DOM every single step. Give it a decent default view and let it dig deeper when that view falls short.

Wrapping up, for now

Building Browsergent taught me making an agent click buttons is the easy part. Building a harness that doesn't constantly confuse the model is the actual work.

The model isn't some magic brain that adapts to whatever beautiful abstraction you invented at 2am. Give it tools it already understands, show it a semantic version of the page but never assume that's everything, and definitely don't just dump the whole DOM in and pray more tokens equals more intelligence.

Still don't have the benchmarks to back a lot of this up properly, and newer browser/coding agents have already solved a surprising chunk of what I originally set out to fix with Browsergent.

But running code right next to my existing tabs still feels different. Doesn't feel like an agent remotely driving a browser somewhere else. Feels more like giving the browser a programmable coworker that's slightly stupid, extremely fast, and occasionally decides an invisible React file input just doesn't exist.

Maybe Browsergent becomes something real. Maybe it stays a weird extension that can never make it into the Chrome Web Store. Either way it already taught me one thing worth keeping.