How to set up ADB and screen capture for a phone-use agent

A phone-use agent is a loop with four beats: grab a screenshot, ask a vision model what to do, send a tap or swipe, check that the screen changed. All four run through ADB, the Android Debug Bridge, and a lot of the failures have nothing to do with the model. A charge-only USB cable. A phone stuck on “unauthorized”. A PNG that arrives corrupted because of how a shell redirect treated the bytes.

This is for teams running an Android agent in production against apps they own, or apps and accounts they have permission to automate, within each app’s terms. By the end you’ll have one phone that a Python script can screenshot, tap, swipe and type into, with retry and reconnect logic for when ADB drops. Android only. iOS uses a different toolchain and I haven’t covered it.

No CAPTCHA solving and no ban evasion here.

What you need

• an Android phone you can dedicate to the agent. Not your personal one. A test device with throwaway accounts is easier to wipe, and how to sandbox a computer-use agent explains the isolation thinking.
• a USB cable that carries data. Plenty of drawer cables are charge-only. Add a powered USB hub once you have more than two phones.
• Android SDK Platform-Tools from Google, free, from the Platform-Tools download page. That is where adb comes from.
• Python 3.10 or newer, plus Pillow (pip install pillow) for resizing images.
• scrcpy, optional and free, from its GitHub repository. I use it to watch and record runs, not as the agent’s eyes.
• a vision-capable model API. Pricing is per call and changes, so check your provider’s price sheet before budgeting.

Software costs nothing. The phone and cable are the whole hardware bill.

Step by step

Step 1: install platform-tools and check adb

Unzip Google’s platform-tools somewhere permanent and add the folder to PATH. I prefer Google’s zip over a distro package, which can lag behind. The adb documentation is the reference for every command below.

# macOS or Linux (on Windows: $env:Path += ";C:\tools\platform-tools")
export PATH="$PATH:$HOME/platform-tools"
adb version

Expected output: the first line reads Android Debug Bridge version 1.0.41, followed by a Version line.

If it breaks: “adb is not recognized” means the PATH change didn’t reach this terminal. Open a new one or call adb by its full path.

Step 2: turn on developer options and USB debugging

On the phone, open Settings, About phone, and tap Build number seven times (Samsung hides it under Software information). Then go to Settings, System, Developer options and switch on USB debugging and Stay awake. Menu paths differ by maker.

Expected output: a Developer options entry appears in Settings with USB debugging on.

If it breaks: as of September 2026, many Xiaomi builds need a second toggle, “USB debugging (Security settings)”, before input commands work. Without it you get a SecurityException about INJECT_EVENTS.

Step 3: connect, authorize and list the device

Plug the phone in and run:

adb kill-server
adb start-server
adb devices

Accept the “Allow USB debugging?” prompt on the phone, tick “Always allow from this computer”, then run adb devices again.

Expected output (your serial will differ):

List of devices attached
R58M12ABCDE    device

If it breaks: “unauthorized” means the prompt was missed. Use “Revoke USB debugging authorizations” in Developer options and replug. “offline” usually clears with a replug and adb kill-server. An empty list is nearly always the cable or the port.

Android 11 and later can also pair over Wi-Fi with adb pair and adb connect. Fine for a bench test. I keep production on USB, because as of September 2026 wireless debugging can switch off when the phone changes network, and the older adb tcpip 5555 mode does not survive a reboot.

Step 4: read the screen size and keep the phone awake

adb shell wm size
adb shell settings put global stay_on_while_plugged_in 3
adb shell settings put system screen_off_timeout 600000
adb shell settings put system accelerometer_rotation 0

Expected output: Physical size: 1080x2400, or whatever your panel is. Write it down, because every coordinate you send lives in that space. If an Override size line appears, run adb shell wm size reset so screenshots and taps agree.

If it breaks: some OEM builds refuse settings put. Use the Stay awake toggle from step 2 and set the timeout by hand.

Step 5: capture a screenshot from Python

adb exec-out screencap -p streams a PNG off the device. In bash you can redirect it to a file. In Windows PowerShell 5.1 the > operator re-encodes the stream and corrupts the PNG, so read the bytes in Python instead.

import io, subprocess
from PIL import Image

def adb(serial, *args, timeout=15):
    return subprocess.run(["adb", "-s", serial, *args],
        capture_output=True, timeout=timeout, check=True).stdout

def screenshot(serial):
    img = Image.open(io.BytesIO(adb(serial, "exec-out", "screencap", "-p")))
    img.load()
    return img

Expected output: screenshot(serial).size gives (1080, 2400), matching step 4. Save one with img.save("check.png") and open it.

If it breaks: a CalledProcessError means adb itself failed, so read e.stderr. A black image means the screen is off, or the app sets FLAG_SECURE, which banking and payment apps often do to block capture. Treat that as a stop sign. Don’t hunt for a way round it.

For a live view while you debug, run scrcpy --no-audio --max-size=1024 --no-control. The --no-control flag stops a stray mouse from fighting the agent, and --record=run.mp4 keeps a video of the run.

Step 6: send taps, swipes, text and keys

def tap(s, x, y):
    adb(s, "shell", "input", "tap", str(x), str(y))

def swipe(s, x1, y1, x2, y2, ms=300):
    adb(s, "shell", "input", "swipe", *map(str, (x1, y1, x2, y2, ms)))

def type_text(s, text):
    adb(s, "shell", "input", "text", text.replace(" ", "%s"))

def key(s, name):
    adb(s, "shell", "input", "keyevent", name)  # KEYCODE_BACK, KEYCODE_HOME

Expected output: nothing prints on success. key(s, "KEYCODE_HOME") returns to the launcher, and a tap on an icon opens the app.

If it breaks: ignored taps mean the wrong coordinate space or the Xiaomi toggle from step 2. input text handles simple ASCII only, and the phone’s shell will read characters like & and ;, so keep strings plain. For anything else, use a keyboard app that accepts broadcast input, such as ADBKeyBoard.

Step 7: shrink the image and map coordinates back

Full-resolution PNGs are slow to upload and cost more tokens. Shrink before sending, and convert the model’s coordinates back before tapping.

def for_model(img, max_side=1280):
    scale = min(1.0, max_side / max(img.size))
    small = img.resize((round(img.width * scale), round(img.height * scale)))
    return small, scale

def to_device(x, y, scale):
    return round(x / scale), round(y / scale)

I start at 1280 pixels on the long side. I haven’t benchmarked that across models, so tune it against your provider’s image guidance and your own accuracy runs.

Expected output: a point at the centre of the small image maps back to the centre of the device, (540, 1200) on a 1080x2400 panel.

If it breaks: taps that land off by a constant ratio mean the scale was applied twice or not at all. Rotation swaps width and height, which is why step 4 locks it.

Step 8: act, verify and recover

Wrap every action so you compare the screen before and after, wait for a change, and reconnect when ADB drops.

import time

def changed(a, b):
    top = int(a.height * 0.05)  # skip the status bar clock
    box = (0, top, a.width, a.height)
    return a.crop(box).tobytes() != b.crop(box).tobytes()

def recover(s):
    subprocess.run(["adb", "-s", s, "reconnect"], timeout=20)
    subprocess.run(["adb", "-s", s, "wait-for-device"], timeout=30)

def act_and_verify(s, action, settle=0.8, tries=3):
    before = screenshot(s)
    for _ in range(tries):
        try:
            action()
            time.sleep(settle)
            after = screenshot(s)
        except (subprocess.SubprocessError, OSError):
            recover(s)
            continue
        if changed(before, after):
            return after
    raise RuntimeError(f"{s}: no screen change after {tries} tries")

Expected output: the new screenshot when the screen changed, or a RuntimeError after three tries. The 0.8 second settle is a starting guess, not a measured value.

If it breaks: a slow cold start fails the check because nothing has changed yet, so raise settle for that step. If recover keeps failing, run adb kill-server and adb start-server, and after that page a human instead of looping. Never put a send, pay or delete tap behind auto-retry, since a tap that worked but changed nothing visible would fire twice. Those belong behind human approval checkpoints.

Common pitfalls

• two adb versions on one machine: scrcpy’s Windows release ships its own adb.exe. If its version differs from your platform-tools, each can kill the other’s server, and your agent sees phones drop at random. Set the ADB environment variable so scrcpy uses the same binary you do.
• overlays you didn’t plan for: notifications, permission dialogs and update prompts cover the thing you meant to tap. Turn on Do Not Disturb, and have the model report a blocking system dialog instead of tapping through it.
• a fixed sleep tuned on a fast screen. 0.8 seconds works on the home screen and fails on a cold app start. Poll for a stable screen with a timeout.
• logging the test phone into personal accounts. A phone holding your own email and banking apps is one bad tap from an incident. Use purpose-made accounts on a dedicated device.

Scaling this

At 10 phones, one adb server can see all of them. Route by serial with -s, run one worker per phone and keep a serial-to-job map in config. Use powered hubs, because unpowered ones cause the random disconnects that look like software bugs. Poll adb devices every minute and alert on anything that isn’t device.

At 100, you’re past one host. USB controllers and CPU set the ceiling per machine, and I don’t have a number for you. Find yours by adding phones until screenshot time climbs. Give each host its own adb server and let an orchestrator hand out jobs. Phones held on a charger around the clock age their batteries, so budget for replacements. Network matters too. A hundred phones on one Wi-Fi network share one public IP, and how IP reputation is earned and lost covers what that means for the sites you talk to.

At 1000, this is a hardware operation: racks, cabling, spares, and someone on site for reboots. I’d price owning against renting before building it. If you’d rather not hold the hardware, cloudf.one rents real Android phones in Singapore on dedicated hardware, each with a persistent Singapore mobile IP, controlled from the browser. It’s ours, so weigh that, and it’s Singapore only. Because you drive it from the browser rather than over a cable you hold, the ADB steps above are for phones you own.

Where to go next

• How to sandbox a computer-use agent: isolating the device and the host that drives it.
• How to add human approval checkpoints to an AI agent: gating the send, pay and delete taps.
• How to route a Playwright agent through a mobile proxy: written for browsers, but the network reasoning carries over.

Everything else is in the blog index.

Written by Xavier Fok

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-09-27.