Set-of-mark prompting

A prompting technique that overlays numbered or labelled marks on a UI screenshot so a vision-language agent can refer to and act on specific interface elements by index rather than by fragile pixel coordinates.

grounded in: trends: computer-use agents driving real interfaces via APIs (Coasty)