Running a fleet of phone-use agents: device health at scale

one phone-use agent is a demo. ten is a hobby with a power strip. fifty is a small operations job, and the thing that breaks is rarely the model. it is the hardware under it: a battery that swelled, a thermal governor that quietly halves the CPU clock, an adb connection that says “device” while the screen is stuck on a system dialog nobody dismissed.

I run Android devices for agent workloads from Singapore, and the pattern I keep seeing is that teams spend their first month on prompts and tool schemas, then lose their second month to devices that fail in ways the agent cannot report. an agent that taps the wrong coordinates on a throttled phone does not raise an exception. it just does a worse job, slowly, and your success-rate chart sags with no obvious cause.

this piece is about the unglamorous layer: how to measure device health, how to decide a device is unfit for work, and how to pull it out of rotation before it burns tokens and task retries. I assume you already know adb, you have run at least one UI-automation loop against a real handset, and you have felt the pain of a flaky device. I will stay with documented Android behaviour and say plainly where I am describing my own operating policy rather than a measured result.

background and prior art

device farms are not new. DeviceFarmer/stf (formerly OpenSTF, from the Smartphone Test Farm project) has exposed fleets of Android devices through a browser for years, with a provider process that watches adb and tracks which devices are present, ready, or in a bad state. Appium’s UiAutomator2 driver is the usual way to drive the UI on top of that. both were built for QA teams running test suites, where a failed run is re-queued and a human looks at the red dashboard in the morning.

phone-use agents change the load profile. a test suite runs for minutes and then the device idles. an agent loop keeps the screen on, holds a radio connection, captures screenshots every few seconds, and may run for hours. that is a sustained thermal and battery load that QA farms mostly did not have to model. the second difference is that the consumer of the device is a model that cannot tell “the app is slow” from “the phone is throttling”. so health has to be measured outside the agent and enforced by the scheduler. the same split exists on the browser side, and I covered the orchestration half of it in the best ways to orchestrate many Playwright browser sessions for a fleet of agents. phones add physical failure modes that a headless browser pool simply does not have.

the core mechanism

the model I use has four parts: a probe, a score, a state machine, and a scheduler that only reads the state. keep them separate. the moment the agent harness starts deciding for itself whether a device is fine, you get optimistic devices that stay in rotation too long.

the probe

a probe is a function that takes a device serial and returns a small dict of facts, collected over adb with no cooperation from the agent. everything below is documented in Android’s adb documentation or the dumpsys reference.

connection state: adb -s <serial> get-state returns device, offline or bootloader, and adb devices -l also lists unauthorized for devices that have not accepted the host key. only device is usable.
battery: adb -s <serial> shell dumpsys battery prints level, status, plugged source and temperature. temperature is reported in tenths of a degree Celsius, so a value of 312 means 31.2 C.
thermal status: adb -s <serial> shell dumpsys thermalservice dumps the thermal service state. the underlying API is PowerManager.getCurrentThermalStatus(), documented in the PowerManager reference, with levels from THERMAL_STATUS_NONE (0) through light, moderate, severe, critical, emergency and shutdown (6).
storage: adb -s <serial> shell df /data gives free space on the user partition.
foreground state: adb -s <serial> shell dumpsys window includes the currently focused window, which tells you whether the app you expect is actually on screen or a system dialog is.
screen state: adb -s <serial> shell dumpsys power reports wakefulness, so you can tell asleep from awake.

a minimal probe in Python looks like this:

import re
import subprocess

def adb(serial, *args, timeout=8):
    return subprocess.run(
        ["adb", "-s", serial, *args],
        capture_output=True, text=True, timeout=timeout,
    ).stdout

def probe(serial):
    facts = {"serial": serial}
    try:
        facts["state"] = adb(serial, "get-state").strip()
        batt = adb(serial, "shell", "dumpsys", "battery")
        facts["level"] = int(re.search(r"level: (\d+)", batt).group(1))
        facts["temp_c"] = int(re.search(r"temperature: (\d+)", batt).group(1)) / 10
        therm = adb(serial, "shell", "dumpsys", "thermalservice")
        m = re.search(r"Thermal Status: (\d+)", therm)
        facts["thermal"] = int(m.group(1)) if m else None
        df = adb(serial, "shell", "df", "/data").splitlines()[-1].split()
        facts["free_kb"] = int(df[3])
    except (subprocess.TimeoutExpired, AttributeError, ValueError, IndexError):
        facts["state"] = "probe_failed"
    return facts

two warnings. first, dumpsys output formats are not a stable API. the field names above have been consistent in my experience but vary by vendor skin and Android version, so treat every regex as something that can return None and handle it as “unknown”, not as zero. second, put a hard timeout on every call. a wedged adb transport will hang a probe forever, and a hung probe loop is its own silent failure.

the score and the state machine

I do not compute a smooth health score. I use hard gates that map to four states.

ready: connected, thermal status 0 or 1, battery temperature under the limit, free storage above the floor, battery above the floor or charging.
degraded: still connected, but one soft limit is breached. the device can finish its current task but takes no new ones.
quarantined: a hard limit is breached or a probe failed twice in a row. no tasks, and a recovery action is queued.
retired: failed recovery three times in a window, or a physical flag (swollen battery, cracked screen, charge port loose) was set by a human. stays out until a person clears it.

the thresholds are policy, and you set them from your own hardware. Android documents the thermal levels but not “the temperature at which your particular handset’s agent loop gets unreliable”. I start from the platform’s own signal, thermal status 2 (moderate) or higher means degraded and 3 (severe) or higher means quarantined, and then adjust per model after watching real task logs.

the scheduler reads state only

the scheduler gets exactly one question per device: is it ready. it never probes, never reasons about temperature, and never overrides a quarantine. this is what lets you change thresholds in one place and have the whole fleet obey. it also keeps the agent harness honest. if you want the agent to know about device trouble, pass it a single boolean at task start, not the raw numbers.

recovery actions

each quarantine reason has a cheap first action and a more expensive second one:

thermal: stop scheduling, turn the screen off with adb shell input keyevent KEYCODE_SLEEP (keycode 223), and wait for status to fall back to 0. do not restart the agent app on a hot device. a restart adds CPU load for no benefit.
offline: adb reconnect offline, then adb kill-server and restart on the host, then a USB port power cycle if you have a controllable hub.
low storage: clear the agent’s own screenshot cache and app caches. never pm clear an app that holds a logged-in session unless you are ready to log in again.
wrong foreground app: send KEYCODE_HOME (3), relaunch the target app, and re-verify the focused window before releasing the device.
battery floor: keep it quarantined until charged. repeated trips of this on one unit point to a charger or cable fault, not a software one.

every recovery action is logged with the device serial, the reason and the outcome. you will use that log later to find the repeat offenders.

worked examples

these are illustrative setups with numbers I chose to show the arithmetic. they are not benchmark results.

example 1: the thermal gate on a 24-device rack

say a rack holds 24 handsets in a shelf with one fan and USB hubs behind it. each runs an agent loop with the screen on. you probe every 30 seconds and apply this policy:

thermal status 2 or higher: degraded
thermal status 3 or higher, or battery temperature at or above 42.0 C: quarantined
release back to ready only when status is 0 or 1 and temperature is at or below 38.0 C for two consecutive probes

the gap between 42 and 38 is deliberate hysteresis. without it, a device sitting right at the limit flaps between ready and quarantined, and the scheduler hands it a task every other cycle. the specific numbers (38, 42) are my own starting values. check what your handset’s vendor documents and adjust.

the point of this example is the exit condition, not the entry. most teams write the entry rule and forget that the release rule needs to be stricter, or they release a still-warm phone straight into another long task.

example 2: capacity planning with quarantine in the model

suppose you need 20 devices working concurrently and you want to know how many to rack. a quick calculation:

target concurrency: 20
expected share of devices quarantined or in recovery at any moment: you do not know this yet. say you assume 15 percent for the first month as a planning number, and replace it with your own measurement from the recovery log
devices to rack: 20 / (1 - 0.15) = 23.5, so 24

the 15 percent is an assumption I am labelling as one. the useful part is the method: the unavailable fraction comes straight out of your state log (time spent in quarantined divided by total time), so after two weeks you can replace the guess with a real figure per device model. if one model sits at a visibly higher quarantine share than another, that is a purchasing signal, and you get it for free from data you were collecting anyway.

this also ties into spend. a degraded device that keeps getting tasks burns model tokens on retries. I went through that accounting in cost engineering for computer-use agents, and the short version is that a health gate pays for itself by removing wasted attempts before they happen.

example 3: the stuck dialog that adb cannot see

here is a failure that every connection check says is fine. a handset reports device, battery 80 percent, thermal 0. the agent keeps failing. the focused window turns out to be a system permission prompt or an OS update notice sitting over the target app, so every tap lands on the wrong layer.

the check is cheap: read the focused window from dumpsys window and compare it to the package you expect. in practice:

def focused_package(serial):
    out = adb(serial, "shell", "dumpsys", "window")
    m = re.search(r"mCurrentFocus=.*? ([\w.]+)/", out)
    return m.group(1) if m else None

def foreground_ok(serial, expected_pkg):
    return focused_package(serial) == expected_pkg

run it before each task starts, and again if two consecutive agent steps produce no screen change. a mismatch sends the device to quarantine with reason wrong_foreground and triggers the home-and-relaunch recovery. the mCurrentFocus line format also varies across Android versions, so test the regex on every model in your fleet and fall back to “unknown, treat as degraded” when it does not match.

edge cases and failure modes

the probe lies by omission. a phone that returns a perfectly healthy dumpsys while its app is frozen will pass every infrastructure check. counter this with a task-level canary: a cheap known action (open the app, read one element, close it) run on a schedule, with the result fed into the same state machine. infrastructure health and task health are different signals and you need both.
adb itself becomes the bottleneck. at a few dozen devices, a probe that shells out five times per device every 30 seconds is a lot of process spawns on one host, and a single stuck call can stall the whole loop if you run it serially. run probes concurrently with per-call timeouts, cap concurrency, and alert when a full probe cycle takes longer than your interval. if the host is the weak link, split devices across several hosts, which is how STF’s provider model is meant to scale.
identity drift after reboots. serial numbers are stable over USB but network adb endpoints change, and a device that reboots may reappear under a new address. key your state on the hardware serial, never on the IP and port. otherwise you get a ghost entry stuck in quarantine forever and a “new” device with no history.
network position hides as device trouble. if several phones on one mobile network start failing in the same minute, that is not seven bad phones. it is one network event. cellular addressing often sits behind carrier-grade NAT (the mechanism is specified in RFC 6888), which affects what the outside world sees and how long connections survive, and I wrote about the practical consequences in carrier-grade NAT explained for AI agents. correlate failures across devices before you blame hardware. a simple rule helps: if more than a set share of the fleet trips the same reason inside one window, raise a fleet-level incident and pause scheduling, instead of quarantining everything one by one.
recovery loops that make things worse. restarting a crashed app on a hot, low-storage phone, over and over, is how you turn a recoverable device into a retired one. cap recovery attempts per device per window, back off between them, and make the third failure terminal until a human looks. this is also where idempotency matters, because a task interrupted by a quarantine will be retried on another device. if it is not safe to run twice, you will double-submit something. see idempotency for AI agents explained for how I key task attempts so a requeue is harmless.
charging as a health risk. a device on a charger with the screen on and a long agent loop is under constant heat from both. charging speed and thermal behaviour differ by handset, and some phones limit charge current when warm. do not assume plugged in means healthy. read the battery temperature, and treat a device that sits at a low level while plugged in as a cable or charger fault.

what we learned in production

the biggest shift for me was moving every health decision out of the agent and into a small supervisor that owns the device state. before that, each task did its own half-hearted checks and each had slightly different thresholds. once the scheduler only reads ready and nothing else, changing a threshold became a one-line edit, and debugging became a question of reading one state log per device. pair that with decent tracing so that you can answer “which device ran this failed task, and what state was it in at the time”. I keep that device serial and probe snapshot on every task record, and the tooling comparison in best tools for tracing and debugging AI agents covers where to hang that data.

the second lesson is that hardware choice dominates software cleverness. emulators and shared pools are fine for development, but anything that depends on realistic device behaviour, a real mobile connection, or an app that checks the environment is better on actual handsets, with proper cooling and fixed power. if you do not want to build a rack, renting is a reasonable option. cloudf.one rents real Android phones in Singapore on dedicated hardware, each with a persistent Singapore mobile IP and controlled from the browser. it is Singapore-only, so it only fits if Singapore is where your agent needs to be. whichever route you take, the health layer above still applies, because renting hardware does not make thermal limits and stuck dialogs go away. you can browse the rest of our operator notes on the blog index.

Written by Xavier Fok

references and further reading

Android Debug Bridge (adb) documentation, the primary reference for device states, shell commands and connection handling
dumpsys reference, what the system service dumps contain and how to read them
PowerManager API reference, including the thermal status levels used in the probe above
Appium UiAutomator2 driver, the common UI automation layer on Android
DeviceFarmer/stf, an open source device farm whose provider model is worth studying before you build your own
RFC 6888: Requirements for CGNs, the IETF specification for carrier-grade NAT

disclosure: this article may contain affiliate links. if you buy through them we may earn a commission at no extra cost to you. verdicts are independent of payouts. last reviewed by Xavier Fok on 2026-10-18.