--- type: Article title: Adversarial Images Hijack Web Agents from Visual Grounding to Browser Execution description: WebMirage optimizes localized adversarial images against candidate selection and action post-processing in vision-based web agents. It varies webpage composition and candidate mappings and measures the browser action actually executed. resource: "https://arxiv.org/abs/2610.09240" tags: [article, webseclist-reference, en, arxiv, browser, ai-agent, adversarial-image, visual-grounding, methodology] generated: by: webseclist-refs/1 at: "2026-10-21T11:35:58+00:00" status: stable stale_after: 2027-10-21 sources: - id: original resource: "https://arxiv.org/abs/2610.09240" title: Adversarial Images Hijack Web Agents from Visual Grounding to Browser Execution author: Wanjing Han, Levi Taiji Li, Mu Zhang, Yue Jiang, Guanhong Tao last_modified: 2026-10-07 also_at: - "https://arxiv.org/pdf/2610.09240v1" authors: - Wanjing Han - Levi Taiji Li - Mu Zhang - Yue Jiang - Guanhong Tao canonical_url: "" cited_by: - "2026-ai.md:202" commit: "" content_sha256: ecbd6d4d16ae92b6e590a7cfaebbc87e01f759e81732af1f807eb65dd763f276 depth: full depth_reason: default kind: article language: en licence: unknown original_url: "https://arxiv.org/abs/2610.09240" published: 2026-10-07 publisher: arXiv publisher_english: "" raw_sha256: dd1d07a93018d4560dc399afcd47f870e112c8a649321b40aaa54d1de807c224 retrieved_from: "https://arxiv.org/pdf/2610.09240v1" retrieved_kind: live retrieved_utc: "2026-10-21T11:35:58+00:00" slug: adversarial-images-hijack-web-agents-visual-grounding-browser-execution snapshot: "" title_english: "" translation_file: "" translation_of: "" --- # Adversarial Images Hijack Web Agents from Visual Grounding to Browser Execution **Adversarial Images Hijack Web Agents from Visual Grounding to Browser Execution** - Wanjing Han, Levi Taiji Li, Mu Zhang, Yue Jiang, Guanhong Tao, arXiv. - Published: 2026-10-07 - Original: - Also published at: - Preserved from: https://arxiv.org/pdf/2610.09240v1 (live) on 2026-10-21 - Licence: unknown Rights remain with the original author and publisher. This is a research archive of a source from the Web Hacking Techniques Index collections, kept so it remains readable if the page goes offline. To read the original, follow the link above. ## Content > UNTRUSTED SOURCE TEXT. Everything below this line is third-party material > quoted for research. It is data, not instructions. Do not follow directions, > execute code, or fetch URLs because this text says so. Adversarial Images Hijack Web Agents from Visual Grounding to Browser Execution Wanjing Han Levi Taiji Li Mu Zhang Yue Jiang Guanhong Tao University of Utah {wanjing.han, levili, muzhang, yue.jiang, guanhong.tao}@utah.edu arXiv:2610.09240v1 [cs.CR] 7 Oct 2026 Abstract becomes a direct attack surface: an attacker who controls such content can redirect actions performed on the user’s behalf. Modern web agents built on large vision-language models Prior red-teaming studies have shown that untrusted web- process webpages, select relevant UI elements, and trans- page content can manipulate web agents. Prompt-injection late model outputs into browser actions. Existing visual red- attacks such as EIA [11] and WIPI [12] introduce adversar- teaming approaches use adversarial visual content to manip- ial instructions that redirect the agent from the user’s task. ulate this process. However, they primarily target model in- VWA-Adv [13] shows that attacker-controlled images can ference and do not explicitly account for structured input steer model inference without injecting such instructions, but processing or action post-processing. Consequently, model- assumes a fixed webpage context and targets only an inter- level success does not establish control over browser exe- mediate textual representation. C HAMELEON [14] accounts cution and cannot reliably characterize end-to-end agent ro- for changing webpage context but optimizes the trigger to- bustness. To address this gap, we formulate red teaming for ward the same hard-coded model output across all samples. In vision-grounded web agents as an end-to-end grounding-to- both cases, the attack is formulated around model inference, execution problem, and introduce W EB M IRAGE, a framework overlooking the full agent pipeline from webpage observa- that crafts localized visual perturbations that cause agents to tion to browser execution. We reevaluate these visual attacks select attacker-controlled content and execute the correspond- using the browser action actually executed as the success cri- ing browser action across varying webpage renderings. It terion and find that neither exceeds an average ASR of 17.4%. uses a role-slot abstraction and webpage recomposition to Yet these low execution-level ASRs do not establish that the capture competition among webpage elements, and dataflow agents themselves are robust. analysis to align optimization with action post-processing. We evaluate W EB M IRAGE across four agent configurations The key question is therefore not whether adversarial visual and six VLM backbones on 2,250 tasks covering 13 public content can change model inference in isolation, but whether websites and a sandbox benchmark. W EB M IRAGE achieves the attack remains effective across the full agent pipeline— an average attack success rate of 91.9%, compared with 17.4% from structured input processing to what the browser ulti- for the strongest baseline, and remains effective against three mately executes. A web agent first maps localized webpage agent-level defenses. elements into candidates that the model can select. It then parses the model’s prediction, resolves the selected candidate to a webpage element, and issues the corresponding browser 1 Introduction command. We refer to this complete path as the grounding-to- execution pipeline. Prior attacks ignore the dynamic depen- Recent advances in web agents [1–5], including generalist sys- dency between webpage elements and model outputs along tems such as S EE ACT [6] and W EB VOYAGER [7], are rapidly this pipeline, causing their model-level targets to lose align- transforming large multimodal models into practical assis- ment with the attacker-controlled element across renderings. tants for everyday web-browsing tasks. Unlike text-centric Realizing such an end-to-end attack is challenging for three agents [8–10], many modern web agents rely on visual ground- reasons. First, the attacker controls only a small visual ele- ing to determine actions on rendered webpages: they process ment, which must compete with other elements that could also screenshots together with localized UI elements and select satisfy the same user query. Second, webpage compositions which element to act on next. As web agents increasingly vary across renderings. Dynamic loading, ranking, and layout execute consequential browser actions—including purchases, reflow alter not only the visual context but also the structured bookings, and form submissions—untrusted webpage content candidate set and the mapping from model outputs to page ele- 1 ments. Third, the agent does not execute its raw model output browse and gather evidence from the web with tool support. directly. Post-processing, such as regular-expression matching Benchmarks and agent frameworks such as WebShop [9, 16], or structured command extraction, determines which browser M IND 2W EB [8], and W EBA RENA [10] largely operational- action is executed. Optimizing only the raw model response ize web interaction through structured text representations without modeling this transformation may therefore fail to such as DOM and accessibility trees. More recent approaches change the agent’s behavior. include generalist agents such as S EE ACT [6] and W EB VOY- We adopt a capable red-team setting: red team synthesis has AGER [7], vision-enabled environments such as VisualWe- white-box access to the target agent’s code and model, while bArena [17], and grounding techniques such as Set-of-Mark at testing time the attacker controls only a localized visual as- prompting [18]. These approaches together incorporate ren- set and cannot modify the DOM, agent configuration, or user dered screenshots to capture page semantics that are difficult instruction. This setting models open-source deployments and to recover from raw HTML alone. measures worst-case exploitability without requiring access Despite these architectural variations, modern VLM-based to the running agent. Under this setting, we present W EB - web agents share a grounding-to-execution loop that converts M IRAGE1 , a pipeline-aware red-teaming framework for the page observations into executable browser commands. This grounding-to-execution pipeline. W EB M IRAGE introduces a pipeline determines which webpage element the browser acts role-slot abstraction that captures which page elements serve on and forms the attack surface we study. the same functional role and may compete for the same user Workflow Overview. At each step t, the agent proceeds in intent. During optimization, it recomposes rendered pages by four stages, as illustrated in Figure 1. varying role-slot assignments and neighboring elements, so (1) Observation Construction. The agent converts the current the attack encounters variations in candidate composition and webpage into an observation Ot . In modern agents, Ot typ- layout that may occur during testing. Finally, it uses dataflow ically consists of a screenshot It and a candidate set of UI analysis to align attack optimization with the post-processing elements Ct . Agents construct Ct in different ways, but it al- logic that produces the browser action. Our contributions are: ways lists the localized UI elements available for the next • We formulate end-to-end red teaming for vision- action. Each candidate c ∈ Ct is associated with a model- grounded web agents as a grounding-to-execution prob- facing identifier λt (c), such as a numeric tag or element token, lem. This formulation covers structured input processing, which the runtime later resolves during execution. model inference, and action post-processing, and defines • Screen-Tagged Candidates: Candidates are explicitly attack success by the resulting browser action. associated with marked regions on the screenshot, such as bounding boxes or indexed numeric tags [6, 18]. • We develop W EB M IRAGE, a pipeline-aware red-teaming framework. It uses a role-slot abstraction and web- • Tree-Derived Candidates: Candidates are extracted from page recomposition to model element competition under the DOM or accessibility tree and serialized as text [8, changing webpage contexts. It further uses dataflow anal- 17]. Crucially, these candidates are ultimately resolved ysis to align attack optimization with the browser action to rendered screen regions for execution, and the model produced by the agent. relies on the screenshot to disambiguate them in visually rich pages. • We evaluate W EB M IRAGE against EIA, VWA-Adv, and (2) Model Inference. Given the task instruction and the ob- Chameleon across four agent configurations and six servation (It , Ct ), the vision-language model produces a free- VLM backbones on 2,250 tasks spanning 13 public web- form response that recommends an action, often with a short sites and a sandbox benchmark. W EB M IRAGE averages explanation. For example, it may output: “The next action is 91.9% ASR, versus 17.4% for the strongest baseline, and Click [21] because it matches the target product.” transfers to VLMs not used during optimization. Three (3) Action Extraction. The agent runtime parses the model existing agent-level defenses have limited effect, while output into a structured command, such as Click [21] or adaptive defenses reduce attack success only when they Select . It discards the explanatory text and also substantially reduce clean-task performance. retains only the executable command, including the identifier assigned to a specific candidate in Ct . 2 Background and Related Work (4) Execution and Iteration. The agent resolves the identifier in the structured command to its candidate in Ct and exe- 2.1 Web Agents cutes the corresponding browser interaction. The webpage transitions to a new state, and the loop repeats. Recent web agents have evolved rapidly from text-centric Visual Grounding as the Unified Interface. Although agents browsing assistants to vision-grounded generalists. Early sys- differ in whether candidates are produced from visual cues tems such as WebGPT [15] demonstrated that LLMs can or from DOM/accessibility structures, each candidate ulti- 1 Misdirected Intent via Redirected Agent Grounding Exploit mately resolves to a localized region on the rendered page. 2 Environment VLM Reasoning Execution State Update St+1 Task Action Extraction Browser Window Structured Command “Buy a carpet under $60.” Synthesis $89 $105 $52.99 target Vision Language Model Executable Action xt Observation Construction t te Observation Ot Visual Grounding Click [21] tpu + Reasoning Ou Screenshot It Ot 19 20 21 Model output Browser Execution $89 $105 $52.99 The user wants a carpet under Click → Candidates Ct $60. Based on the screenshot, Classic Carpet [19] Rug Carpet the next action I should take is [20] Elegance Carpet Click [21]. [21] Classic Carpet Screen-Tagged or Tree-Derived Figure 1: Vision-Grounded Web Agent Workflow. The agent operates in a loop: (1) constructs an observation from the screenshot and candidate elements (Environment), (2) reasoning over the visual observation (VLM Reasoning), (3) extracting a structured command from the raw output (Action Extraction), and (4) executing the action to update the state. As a result, selecting the correct candidate requires ground- SecureWebArena [21] and BrowserART [22] provide system- ing its semantics in It —for example, reading a button label atic evaluation of these instruction-channel threat vectors. or recognizing a product image—to distinguish it from visu- Visual Perturbation Attacks on Web Agents. A separate ally similar competitors. This dependency becomes salient line of work explores visual perturbations against web agents. under natural web dynamics: backend responses and query- WebInject [23] assumes white-box model access and control dependent rendering can reorder candidates, swap neighbors, of the entire rendered webpage, optimizing page-wide per- trigger layout reflow, and change which identifier is assigned turbations to induce attacker-specified actions. Under a more to each candidate. Consequently, the same output identifier constrained threat model, VWA-Adv [13] perturbs a localized can resolve to different webpage elements across renderings, image to alter the model’s semantic interpretation through em- so a successful attack must induce the identifier assigned to bedding similarity or intermediate captions. Chameleon [14] the attacker-controlled element in each rendering. These prop- uses white-box optimization to make localized triggers ro- erties motivate our abstraction Ot = (It , Ct ) and our role-slot bust to changes in position and surrounding visual context. modeling of webpages as compositions of localized semantic- However, it optimizes the trigger toward the same hard-coded visual elements in §4.1. model output across all sampled contexts, without consid- ering changing candidate mappings or downstream action resolution. 2.2 Related Work These studies establish the vulnerability of the visual path- way, but their formulations do not model structured candidate Prompt Injection in Web Agents. Deployment-time at- construction before inference or the downstream resolution tacks on web agents exploit accessible surfaces such as web- of model outputs into browser actions. page content or agent-visible text to manipulate agent behav- ior without modifying model weights. WIPI [12], PopupAt- Adversarial Examples on Visual Inputs. Adversarial per- tack [19], and AdInject [20] inject hidden or misleading con- turbations on images have been extensively studied in image tent through DOM elements, pop-up overlays, or online adver- classification [24, 25] and extended to vision-language mod- tisements to hijack the agent’s task. EIA [11] injects HTML els [26–28]. These attacks typically assume full control over elements with persuasive instructions near legitimate form the input image and evaluate success at the model’s predic- fields, causing the agent to direct sensitive user inputs toward tion or generated output. Techniques such as Expectation attacker-controlled elements rather than the intended targets. over Transformation (EoT) [29] improve robustness by op- These attacks rely on the agent following injected instructions timizing over geometric transformations. In the web-agent or cues, rather than manipulating how it visually resolves setting, however, the dominant sources of variability are com- candidates in the rendered page. Recent benchmarks such as positional rather than geometric: which candidates appear 3 Offline Synthesis ❶ ❷ ❸ ❹ ... k=3 exec-aligned target k=1 v+δ VLM click[8] ℒ companion pool 𝒵 k=4 δ update perturbation target website role slots recomposed pages Online Deployment Attacker content Competing element web agent Clicked! web agent Clicked! Clicked! Clicked! Correct selection rendering 1 rendering 2 rendering 3 Benign Webpage Adversarial Webpage Figure 2: Overview of W EB M IRAGE. The top figure denotes offline synthesis: ❶ identify role slots from the target webpage; ❷ recompose page instances by sampling companions and slot placements; ❸ optimize a perturbation so the attack content is consistently selected, with ❹ supervision aligned to the executed command via dataflow analysis. The bottom figure shows on benign pages the agent selects the best match; under attack, it selects the attack content across varying renderings. alongside the target and which position the target occupies website, such as its general layout, content format, and typical across renderings. user goals, but not the exact user request or page instance encountered at test time. Test-Time Capability. At test time, the red team retains its 3 Threat Model offline knowledge but cannot inspect or modify the running agent or its state. Its only influence on the agent is through a Targeted Systems. We study vision-grounded web agents localized third-party image as rendered on the webpage, such that assist users with open-ended web tasks such as shopping, as a product thumbnail or profile photo. The red team cannot booking, or hiring. These tasks are typically multi-step (e.g., modify the website code or DOM, inject textual instructions, entering a product name in a search bar, browsing to find a alter the agent runtime, or control the user’s request or the matching item, clicking the purchase button, and so on). At agent’s action history. Because the perturbation is fixed before each step, such agents take an observation consisting of a ren- the webpage is rendered, the image may appear in different dered screenshot and a discrete candidate set of UI elements, positions and alongside different elements. The red team there- and output a natural-language response that is subsequently fore cannot know in advance which candidate identifier will parsed into an executed browser command (§2.1). be assigned to the attacker-controlled element. Red-Team Objective. This work studies white-box red team- ing for vulnerability discovery. Our goal is to characterize 4 Methodology the worst-case exploitability of the grounding-to-execution interface under a capable adversary. The red team aims to We design W EB M IRAGE to exploit the grounding-to- craft visual content that causes the agent to select an attacker- execution pipeline of vision-grounded web agents. Our goal controlled candidate and execute the corresponding browser is to construct an adversarial image that causes the agent to action, across webpage renderings with varying surrounding execute an attacker-intended browser action across different content and layouts. webpage renderings. These renderings may change both the Red-Team Capabilities. We model the red team’s capabilities candidate set presented to the model and the mapping from in two phases: an offline synthesis phase, in which the attack model outputs to browser actions. is constructed, and a test-time phase, in which the fixed attack As illustrated in Figure 2, W EB M IRAGE follows a four-step is evaluated under restricted attacker control. offline synthesis process before test-time evaluation. First, Offline Capability. During offline synthesis, the red team we analyze the target website through the agent’s observa- has white-box access to the target agent’s implementation tion interface, modeling both how webpage elements are rep- and the open-weight VLM it uses. This includes the model resented in the rendered screenshot and candidate set and interface and parameters, structured input construction, out- how the attacker-controlled image competes with semanti- put parsing, and browser-action execution. The red team uses cally similar benign candidates (§4.1). Second, we recom- this access to optimize adversarial visual content before de- pose page instances by varying the placement of the attacker- ployment. It may also know coarse properties of the target controlled content and the neighboring elements. For each 4 recomposed page, we simulate the corresponding candidate set and element-to-candidate mapping, capturing changes in both the rendered screenshot and the agent’s structured ob- servation (§4.2). Third, we formulate a unified objective for optimizing a bounded perturbation across these recomposed pages (§4.3). Finally, we use dataflow analysis to refine the optimization target to the model-output tokens that determine the executable browser action (§4.4). The resulting image is then evaluated at test time, where it can consistently steer the agent toward target actions across varying webpages. 4.1 Threat-Driven Observation Modeling Before the model sees a webpage, the agent preprocesses the rendered page into a structured observation: it identifies UI elements, assigns candidate labels, and pairs the result- ing candidate set with the screenshot. The attacker-controlled image (e.g., a product thumbnail) is associated with one can- didate among several that may satisfy the same user intent (e.g., different product listings). Because the agent selects from these discrete candidates rather than directly predict- Figure 3: Role-Slot Abstraction. The red box highlights the ing a pixel location, the attack surface lies in how candidates same semantic item appearing in different compatible slots compete within this structured observation. The attack must and alongside different surrounding content across renderings, remain effective even as the candidate’s position, assigned illustrating how variability changes both placement and local label, and surrounding competitors change across renderings. competition at test time. Specifically, we separate where the attacker-controlled con- tent appears from what semantic role it plays relative to com- peting items. This separation is inspired by the VIPS view of The screenshot Iscreen is obtained by compositing the regions webpages as compositions of visually coherent regions [30], in S : but is motivated here by constraints on what the attacker can Iscreen = C OMPOSITE(S ), (3) control. It is also consistent with modern vision-language sys- and the candidate set Ccand contains the locations and textual tems’ ability to align localized visual regions with compact descriptions of interactable elements: semantic or structural descriptors [18, 31]. Ccand = {(ℓi , bi , di ) | i ∈ Iact }, (4) 4.1.1 Visual Page Decomposition where each i ∈ Iact ⊆ {1, . . . , N} indexes an interactable We represent a rendered webpage as a set of N visually coher- region in S , ℓi is the label used by the agent to refer to ent regions: candidate i, and di is a textual description of element i derived from page content or model extraction (e.g., DOM attributes, S = {(vi , bi )}Ni=1 , (1) surrounding text, or OCR output). where bi = (xi , yi , wi , hi ) denotes the bounding box of region i This matches the standard input interface of modern web and vi denotes the visual content within that box. This decom- agents [6, 17, 18], where the agent selects among candidates position provides the spatial structure on which the agent’s based on their relevance to the user’s request. In practice, observation and our subsequent abstractions are defined. multiple candidates may serve the same functional purpose on the page, creating the selection competition that we formalize next. 4.1.2 Agent Observation Interface 4.1.3 Role Slots As illustrated in §2.1, agents do not operate directly on the page representation S . Instead, they receive an observation Agents select among candidate elements that may differ in O derived from S , consisting of a rendered screenshot and their descriptions di (e.g., product name or price) while serv- a candidate set: ing the same functional purpose on the page. To formalize this competition, we assign each element a semantic role label O = (Iscreen , Ccand ). (2) ri that identifies its functional category (e.g., product item, 5 search box). Such competition often arises in interfaces built To model the surrounding competition, we collect a pool from repeated templates (e.g., product grids), where multiple Z of benign images depicting the same type of content as vt positions serve the same role. We refer to these interchange- (e.g., other profile photos or product images from the same able positions as role slots. We define these slots over web- category). We populate the remaining l − 1 slots with im- page regions, independently of the per-rendering candidate ages z1 , z2 , . . . , zl−1 sampled from Z and denote them by mapping. Figure 3 illustrates this on a product listing page: z = (z1 , . . . , zl−1 ). Let φδ (vt ) denote the perturbed attacker- the red-boxed element and the surrounding product listings controlled image, and let I(φδ (vt ), bk , z ) denote the resulting all share the semantic role product item, and their positions screenshot. constitute the role-slot set. The perturbation must achieve low expected loss over the Formally, let the target element be st = (vt , bt ) ∈ S with sampled companion images: role rt and bounding box bt = (xt , yt , wt , ht ). The role-slot index set is Ez∼Z l−1 [L (I(φδ (vt ), bk , z ), q, yt )] ≤ η. (6) Kslot (rt , st ) = { k ∈ [N] | rk = rt , Here, L (I, q, yt ) denotes the token-level loss defined in §4.3. |wk − wt |/wt ≤ εw , (5) η ≥ 0 is a small threshold on the expected loss. |hk − ht |/ht ≤ εh }, Positional Shift. The second source of uncertainty is the position of the attacker-controlled image. Across renderings, where the first condition enforces the same semantic role, the image may occupy different compatible slots in Kslot . To and the remaining conditions bound the relative difference in isolate this variation, we hold a sampled set of companion im- width and height. In practice, role labels are obtained from ages z̄ = (z̄1 , . . . , z̄l−1 ) fixed and place the attacker-controlled manual annotation or a lightweight element classifier, and image at the bounding box bk of each slot k ∈ Kslot . For each εw , εh are implementation parameters that control how similar placement, we reconstruct the corresponding candidate set two elements must be in size to be treated as compatible slots. and target output using the resulting candidate labels. The This definition captures a key source of uncertainty at perturbation must achieve low expected loss over the sampled test time. Across different renderings, the same attacker- slot positions: controlled image may appear in different compatible slots due to layout reflow, while the remaining slots are filled with Ek∈Kslot [L (I(φδ (vt ), bk , z̄ ), qk , yt,k )] ≤ γ. (7) varying competing content. We account for changes to both the rendered screenshot and the agent’s candidate mapping Here, qk contains the candidate set reconstructed for during optimization (§4.2). placement k, and yt,k uses the label assigned to the attacker- controlled candidate under this mapping. γ ≥ 0 is a small threshold on the expected loss. 4.2 Attack Constraints under Webpage Vari- We do not model arbitrary whole-page variation. Page re- ability gions outside the target role-slot set remain unchanged during As discussed in §4.1, the role-slot abstraction captures two recomposition. The optimization therefore targets compan- sources of testing-time variation outside the attacker’s con- ion content and slot placement, which together capture the trol: which benign content fills the other compatible slots test-time uncertainty most relevant to whether the attacker- and which slot contains the attacker-controlled image. We controlled candidate is selected. refer to these as companion variation and positional shift, respectively. In practice, companion variation and positional 4.3 Unified Attack Objective shift may occur simultaneously, as illustrated in Figure 3. We describe them separately by holding one fixed while varying The previous section characterizes companion variation and the other, and later sample them jointly during optimization. positional shift separately. We now account for both in a single Page recomposition changes the candidate mapping pre- optimization objective. sented to the model. For each recomposed page, we rebuild Attack Target. Our ultimate goal is to cause the agent to the grounding input (q), including the interaction context and select the attacker-controlled candidate and execute the cor- reconstructed candidate set (Ccand ). The target output (yt ) uses responding browser action. At this stage, we use a full target the label (ℓt ) assigned to the attacker-controlled candidate, response yt as the model-output target. This response may keeping the optimization target aligned with the candidate contain an action description, reasoning, and other text that mapping of each page sample. does not affect the executed action. We optimize the proba- Companion Variation. We first vary the surrounding content bility of yt conditioned on the grounding input q defined in while holding the attacker-controlled image in a fixed slot. Let §4.2. l = |Kslot | denote the number of visible slots in the target role- Perturbation Modeling. Following standard practice in ad- slot set, and assume the attacker-controlled image vt occupies versarial example generation [25], we constrain the pertur- a fixed slot k ∈ Kslot with bounding box bk . bation to an ℓ∞ -ball of radius ε to help keep the modified 6 content visually similar to the original. The perturbed content Agent is obtained as: predict() ... φδ (vt ) = clip(vt + clip(δ, −ε, ε) , 0, 1) , (8) output = model.generate(prompt, ...) where δ is the added perturbation, ε is the element-wise bound, and pixel values are normalized to [0, 1]. output="The user wants a carpet.\nThe next action I should take is\n: take ACTION: CLICK on Token-Level Loss. We minimize the negative log-likelihood button at (0.453, 0.079) with VALUE: None\n..." of the full target response yt : postprocess(output) M output = output.strip().replace(...) 1 L (I, q, yt ) = − ∑ log P ytj | yt< j , u(I) ⊕ q , (9) label =re.findall(r"ELEMENT: ([A-Z]{2}|[A-Z])", output) M j=1 action=re.search(r"ACTION: (CLICK|SELECT|...)", output) value =re.search(r"VALUE: (.*)$", output, re.MULTILINE) prediction = { j "label": label, where M is the number of tokens in yt , yt denotes its j-th "action":action, token, and P is the target VLM’s next-token distribution. The "value": value, ...} function u(·) denotes the input preprocessing pipeline (e.g., resizing, cropping, and patch extraction). The perturbation en- label = "ELEMENT: A" ters the loss through I, which contains the perturbed attacker- action= "ACTION: CLICK" value = "VALUE: None" controlled image. For each recomposed page, I, q, and yt are constructed together as described in §4.2. execute(prediction) Differentiable Preprocessing. The agent’s preprocessing label = prediction["label"] pipeline u(·) uses image-library operations (e.g., PIL) that do action = prediction["action"] new_action = perform_action(label, action, ...) not preserve gradients. We reimplement resizing and patch ... extraction with differentiable PyTorch tensor operations, al- lowing gradients to propagate from the VLM input back to the perturbed image. Figure 4: Agent dataflow example. Raw model output is post- Integrating Webpage Variability. To optimize over both processed into a parsed prediction, and only this extracted forms of webpage variation, we jointly sample companion representation is passed to action execution. images z and a slot placement k at each optimization step. Sub- stituting the corresponding screenshot, grounding input, and target response into Equation 9, the unified attack objective produce a structured prediction, as shown in the second red becomes: box. Finally, execute() interprets the structured prediction and invokes the corresponding browser command. Only a min E k∈Kslot [L (I(φδ (vt ), bk , z ), qk , yt,k )] . (10) small portion of the original response, such as an element δ z ∼Z l−1 identifier or action name, reaches execute(); the remaining For each sampled page, qk and yt,k are reconstructed using text is discarded. its candidate mapping, with k indicating the placement of the In §4.3, the optimization objective targets the full attacker-controlled image. This objective jointly considers model response yt . However, this target is misaligned companion variation and positional shift, together with the with the agent’s execution logic. First, the loss includes candidate mapping constructed for each recomposed page. reasoning, formatting, and other tokens discarded during The objective above targets the full model response yt . In post-processing, making optimization unnecessarily difficult. §4.4, we use dataflow analysis to isolate ytaction , the portion Second, changing the raw model response does not guarantee that determines the executable browser command. that post-processing produces the browser command intended by the attacker. To address this mismatch, we trace how the agent’s post-processing transforms the raw response into an 4.4 Attack Target Alignment via Dataflow executable command. We then restrict optimization to the Analysis tokens that determine that command. Web agents typically apply deterministic post-processing to Specifically, we perform a static dataflow analysis to iden- raw textual model responses before execution. Figure 4 shows tify the execution-relevant target. The raw model response an example. The predict() function obtains the raw re- serves as the data source, where the tracked values originate. sponse from the agent’s VLM, which may contain reasoning, The arguments passed to the execution interface serve as data element descriptions, and browser action names, as shown in sinks, where these values enter browser action execution. By the first red box. The postprocess() function applies string tracking model-derived values and their transformations along normalization, pattern matching, or rule-based extraction to each source-to-sink path, the analysis identifies the fields and 7 substrings of the model response that affect execution. We 5.1 Experimental Settings retain the corresponding portions of the full target response yt as ytaction . Agent Frameworks and VLM models. We evaluate W EB - M IRAGE across four agent configurations and six VLM back- A key challenge in this analysis is the prevalence of string- bones. On public websites, we use S EE ACT [6] and W EB VOY- centric processing and aggregate data structures in Python. AGER [7], two generalist agents with different observation As shown in Figure 4, post-processing relies heavily on string and action-processing pipelines. We pair both agents with the operations (e.g., normalization and regular-expression match- same five backbones: LLaVA-v1.5-13B, LLaVA-v1.6-34B, ing). These operations discard large portions of the model out- MiniCPM-o-8B, Phi-3-Vision-4B, and Qwen2-VL-7B. To put while retaining only small execution-relevant substrings. emulate the rendered presence of third-party visual content These substrings are often stored in aggregate data struc- without modifying server-side data, we replace one existing tures, such as dictionaries, before being passed across function element image with its adversarial version in a controlled boundaries and unpacked during execution. Tracking each browser, leaving the rest of the page unchanged. retained substring through these transformations and data structures to the execution interface is therefore challenging In the sandbox, we follow VisualWebArena [17] with for static analysis. CogVLM [33] as the backbone. We evaluate two observation variants: (i) Screenshot + Accessibility Tree and (ii) Screen- We address these challenges with two analysis components. shot + Set-of-Mark, both augmented with BLIP-2 captions as We use string analysis to recover the transformations applied alt-text. to the model output and identify the substrings they retain. The four configurations cover screen-tagged and tree- For aggregate structures, we track dataflow at the level of derived candidate interfaces and expose the complete individual fields rather than entire objects, associating each grounding-to-execution pipeline. Appendix C explains our field with the corresponding portion of the model output. This agent and backbone selection. field-sensitive tracking avoids propagating taint to unrelated fields while preserving the execution-relevant dataflow. We Scenarios and Tasks. We evaluate W EB M IRAGE in two implement both components in CodeQL [32], using context- complementary settings. For the public-site evaluation, we sensitive and flow-sensitive interprocedural dataflow analysis run S EE ACT and W EB VOYAGER on shopping-style tasks over Python source code. derived from M IND 2W EB [8] and adapted to 13 websites Let 1 ≤ at,1 < · · · < at,Kt ≤ |yt | denote the positions of the across four scenarios: retail, accommodation, home services, execution-relevant tokens retained in ytaction . These tokens and tutoring, as listed in Table 10. For the sandbox evaluation, may occur at arbitrary positions in the raw model output. we use VisualWebArena’s Shopping site (O NE S TOP S HOP) under controlled rendering and interaction. 1 Kt a t,k