Gemini Computer Use API: Build a Safe Browser Agent

Gemini Computer Use is now generally available. Google moved the Computer Use tool to GA on 30 October 2026, for both the Interactions API and the GenerateContent API. You give the model a task and a screenshot, it replies with UI actions like click or type, and your code carries them out in a real browser. The model never touches the browser itself. That part is your job, and so are the guardrails. In this guide I build a small typed Python agent with Playwright that handles the parts the quickstart skips: a domain allowlist, a step limit, human confirmation for risky actions, and a realistic cost estimate.

What GA means for Gemini Computer Use

Computer Use has been around in preview for a year. Google launched Gemini 2.5 Computer Use Preview on 7 October 2025, added it to gemini-3-pro-preview and gemini-3-flash-preview in January 2026, and on 24 June 2026 shipped the version this post uses, first in Gemini 3.5 Flash: actions that carry an intent, browser, mobile and desktop environments, configurable safety policies and prompt injection detection. The 30 October release note makes that tool generally available.

For a client project, the practical change is that you can plan on it. The API shape in the guide is the one I tested against below. The guide still warns that the tool "may contain errors and security vulnerabilities" and should be supervised for important tasks, so GA doesn't mean you can leave it unattended.

Which model to use

The guide lists four models. Prices are the paid-tier standard rates per 1M tokens from the pricing page. Batch and Flex are half price on the models that offer them.

I default to gemini-3.8-flash and read the model ID from an environment variable, so switching to Flash-Lite for simple, repetitive flows is a config change. If you are weighing Argon for other work as well, my Gemini 4 Carbon vs Argon guide covers its pricing change on 1 December.

How the agent loop works

Each turn follows the same pattern. Your code sends the task and a screenshot. The model returns one or more function_call steps. You check each one, run it in Playwright, take a new screenshot and send it back as a function_result. The loop ends when the model replies with text instead of an action, or when your code stops it.

Follow-up turns pass previous_interaction_id, so the API keeps the history and you only send the new results. Two details from the guide matter here. If the model returns several actions at once, you must send one function_result per call. And there is no screen size parameter in the request: the model works in normalised coordinates and you scale them to your viewport.

Building a Gemini Computer Use agent in Python

The full file is about 180 lines. These are the parts that need care. Everything below passes mypy --strict.

The tool config

TOOL: gi.ComputerUseParam = {
    "type": "computer_use",
    "environment": "browser",
    "enable_prompt_injection_detection": True,
}

environment accepts browser, mobile or desktop. Prompt injection detection is off by default. When it is on, the model checks each screenshot for hidden instructions (the guide's example is "Ignore previous commands") and blocks execution if it finds one. I turn it on for anything that browses pages I don't control. The SDK's ComputerUseParam type also has excluded_predefined_functions, for removing actions such as click and replacing them with your own functions, and disabled_safety_policies.

Scaling coordinates

def denorm(value: int, size: int) -> int:
    """Model coordinates are normalised to 0-999; scale to viewport pixels."""
    return min(int(value / 1000 * size), size - 1)

Handling safety_decision

This is the part I would not skip. An action can arrive with a safety_decision inside its arguments. When its decision is require_confirmation, you ask a person. If they approve, you add safety_acknowledgement: true to the result for that call. If they decline, you stop.

def handle_call(page: Page, call: gi.FunctionCallStep, confirm: Confirm) -> dict[str, Any]:
    args = dict(call.arguments)
    result: dict[str, Any] = {}
    decision = args.pop("safety_decision", None)
    if isinstance(decision, dict):
        if decision.get("decision") != "require_confirmation":
            raise StopAgent(f"safety decision {decision.get('decision')!r}")
        if not confirm(str(decision.get("explanation", ""))):
            raise StopAgent("user declined the action")
        result["safety_acknowledgement"] = True
    print(f"-> {call.name}: {args.pop('intent', '')}")
    result.update(run_action(page, call.name, args))
    if not host_allowed(page.url):
        raise StopAgent(f"left the allowlist: {page.url}")
    return result

The guide describes three outcomes (allowed, require_confirmation and blocked), but its JSON example only shows require_confirmation. So my code treats any other value it receives as a reason to stop. That's deliberately strict. run_action returns {"error": ...} instead of navigating when a navigate URL is not on the allowlist. That way the model sees why it failed and can try something else. The host check after every action catches links that leave the allowlist through a click.

The safety categories the model checks are FINANCIAL_TRANSACTIONS, SENSITIVE_DATA_MODIFICATION, COMMUNICATION_TOOL, ACCOUNT_CREATION, DATA_MODIFICATION, USER_CONSENT_MANAGEMENT and LEGAL_TERMS_AND_AGREEMENTS.

Sending results back

def function_results(page: Page, done: list[tuple[gi.FunctionCallStep, dict[str, Any]]]
                     ) -> list[gi.FunctionResultStepParam]:
    png = base64.b64encode(page.screenshot(type="png")).decode()
    return [{
        "type": "function_result",
        "name": call.name,
        "call_id": call.id,
        "result": [
            {"type": "text", "text": json.dumps({"url": page.url, **res})},
            {"type": "image", "data": png, "mime_type": "image/png"},
        ],
    } for call, res in done]

Each result carries the current URL, any error or acknowledgement, and a fresh PNG. The loop itself is short:

for _ in range(MAX_STEPS):
    calls = [s for s in interaction.steps or []
             if isinstance(s, gi.FunctionCallStep)]
    if not calls:
        return final_text(interaction)
    done = [(c, handle_call(page, c, confirm)) for c in calls]
    interaction = create(client, function_results(page, done), interaction.id)
return f"stopped: hit MAX_STEPS={MAX_STEPS}"

create is a three-line wrapper around client.interactions.create that checks the return value is an Interaction and not a stream. Without it, mypy complains because the SDK's return type covers both.

Guardrails I add to every browser agent

The guide has a solid safety best practices section. This is how I apply it:

• Run it in a sandbox. A container or VM, or at least a separate browser profile with no saved logins. Google points to a Docker-based reference implementation as a starting point.
• Allowlist, not blocklist. The guide calls an allowlist "even more secure" than a blocklist. For client work I list only the staging hosts the task needs.
• Cap the steps. A confused agent will happily scroll forever, and every turn costs money.
• Write your own system instruction. The guide includes a long example that makes the model stop before sending, buying, accepting terms or solving CAPTCHAs. Start from it.
• Log everything. Prompts, screenshots, each function_call, safety responses and what your code actually executed. You will need it the first time an agent does something odd.
• Start from a clean state. Pop-ups and changed layouts confuse the model, so reset the browser for each task.

Screenshots also end up in Google's logs by default, because the Interactions API stores requests unless told otherwise. If the pages show customer data, read my post on Gemini API logging and store=false before you run this on production screens.

What a Gemini Computer Use run costs

The token counting page says images larger than 384 pixels are split into 768x768 tiles of 258 tokens each. A 1440x900 screenshot is 2 x 2 tiles, about 1,032 tokens. The page also says usage on a chained interaction includes the earlier turns. My working assumption is that turn N bills N screenshots. Treat this as an upper bound and check usage.total_input_tokens on your own runs.

Screenshot input only, before text, output and thinking tokens:

The rates on 3.8 Flash double on 1 January 2027 and on Argon on 1 December 2026, so double those columns after those dates. Per run it's cheap. At thousands of runs a day, step count is the number that matters, which is the same lesson as in my cost per task comparison. A smaller viewport also means fewer tiles per screenshot. The guide adds that lower thinking levels usually work well for standard automation, which cuts output tokens.

How I tested it without an API key

I don't have a production Gemini key on my test machine, so I pointed the SDK at a local mock of the Interactions endpoint (genai.Client(api_key="test-key", http_options={"base_url": ...})) and a local test page with a text box and a Buy button. The mock replays a fixed script: type into the box, click Buy with a require_confirmation safety decision, try to navigate to an outside domain, then finish with text.

Every check passed on three runs in a row:

• The typed text and the click landed on the right elements after coordinate scaling (the next URL carried q=computer+use).
• The click result carried safety_acknowledgement: true after approval, and a declined confirmation stopped the run with no further API calls.
• The outside navigation came back as an allowlist error instead of loading.
• Each function_result matched its call_id and carried a valid PNG, and follow-up turns sent previous_interaction_id.
• A model that never finishes was stopped by MAX_STEPS.

One thing I noticed: the Python SDK posted to /v1beta/interactions, while the guide's curl sample uses /v1/interactions. If you use the SDK, that's handled for you. I'm mentioning it because a mock or proxy that only accepts one path will fail. What this test can't show is how well the real model reads a real page. Try that on a staging site first.

Checklist before you point it at a real site

• Paid Gemini API project, google-genai 2.7.0 or newer (I used 2.34.0).
• Model ID in config, starting with gemini-3.8-flash.
• Browser running in a container or throwaway profile, fixed viewport.
• Domain allowlist checked on navigate and after every action.
• MAX_STEPS set and a spend cap on the Cloud project.
• safety_decision handled for every call, with a real person answering.
• Prompt injection detection on for pages you don't control.
• Custom system instruction that stops before irreversible actions.
• Logs of screenshots, actions and safety responses, plus a decision about AI Studio request storage.
• Unknown actions return an error result instead of crashing the loop.