6 Commits

Author SHA1 Message Date
Dom
ca0b436a61 feat(qw2): hook LoopDetector dans api_stream + extension replay_state
Some checks failed
tests / Lint (ruff + black) (push) Successful in 15s
tests / Tests unitaires (sans GPU) (push) Failing after 17s
tests / Tests sécurité (critique) (push) Has been skipped
replay_state enrichi de _screenshot_history (5 dernières images PIL) et
_action_history (5 dernières signatures action).

report_action_result :
- met à jour les deux anneaux après chaque action
- évalue le LoopDetector (singleton lazy avec _clip_embedder serveur)
- si detected → bascule paused_need_help avec pause_reason="loop_detected"
  et bus event lea:loop_detected (signal + evidence)

Tous les chemins d'erreur (embedder absent, OOM, exception) loggent et
laissent le replay continuer — aucun blocage par la couche détection.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 23:25:04 +02:00
Dom
fc01afa59c fix(qw1): bus event lea:monitor_routed + cablage offset côté executor Agent V1
Cleanup post-review QW1 :
- Émission bus lea:monitor_routed dans /replay/next (idx, source, replay_id, action_id, offset, wh)
  via logger.info "[BUS] lea:monitor_routed ..." (le serveur streaming n'a pas
  de SocketIO local, agent_chat émet déjà lea:* sur 5004 ; ici on logge en INFO
  bien lisible, prêt pour un parser/pont futur)
- Executor Agent V1 (deploy/windows_client) lit action.monitor_resolution.{offset_x, offset_y, idx}
  et applique l'offset aux coords absolues du clic/type/scroll/popup quand idx >= 0
- composite_fallback (idx=-1) : pas d'offset appliqué (backward compat mono-écran)
- Log INFO "QW1 monitor cible idx=N source=X offset=(dx,dy) — appliqué aux coords"
  émis une fois par action quand un offset non nul s'applique

Tests : baseline 95 passed (e2e + phase0_integration + stream_processor + monitor_router + grounding_offset)

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 23:16:06 +02:00
Dom
2a51a844b9 feat(qw2): LoopDetector composite (screen_static + action_repeat + retry)
Module isolé, 3 signaux indépendants :
- screen_static : CLIP similarity > 0.99 sur N captures consécutives
- action_repeat : N actions identiques (type+coords)
- retry_threshold : retried_actions >= seuil

Premier signal positif → LoopVerdict.detected=True (caller responsable de
la bascule en paused_need_help).

Configurable env vars : RPA_LOOP_DETECTOR_ENABLED (kill-switch),
RPA_LOOP_SCREEN_STATIC_N/THRESHOLD, RPA_LOOP_ACTION_REPEAT_N,
RPA_LOOP_RETRY_THRESHOLD.

Tests : 8 cas (chaque signal isolé, kill-switch, embedder absent, exception).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 23:09:43 +02:00
Dom
2d71e2a249 feat(qw1): enrichissement Agent V1 (monitor_index + monitors_geometry) + hook serveur
Some checks failed
tests / Lint (ruff + black) (push) Successful in 16s
tests / Tests unitaires (sans GPU) (push) Failing after 15s
tests / Tests sécurité (critique) (push) Has been skipped
Côté client Agent V1 :
- helpers _get_monitors_geometry() / _get_active_monitor_index() via screeninfo
  (fallback gracieux [] / None si screeninfo absent)
- _enrich_with_monitor_info() ajouté aux payloads dict de capture_dual,
  capture_active_window, et heartbeat_event poussé par main.py
- screeninfo>=0.8 ajouté aux requirements (source + deploy Windows)
- Deploy capturer.py reçoit l'enrichissement de manière additive (pas de
  copie verbatim qui aurait introduit BLUR_SENSITIVE absent côté deploy)

Côté serveur :
- import resolve_target_monitor depuis monitor_router (créé en QW1.1)
- /replay/next : enrichissement action.monitor_resolution avant envoi
  au client (idx, offset_x/y, w, h, source de la décision)
- live_session_manager.add_event : propagation monitor_index +
  monitors_geometry depuis window_capture ET depuis le payload event
  brut (cas heartbeat enrichi sans window/window_title)

Cascade de résolution (cf monitor_router.py) :
1. action.monitor_index (hérité de la session source)
2. session.last_focused_monitor (focus actif vu en dernier heartbeat)
3. composite_fallback (offset 0,0) — backward compat strict

Backward 100% : si geometry vide, fallback composite identique au
comportement actuel mss.monitors[0].

Tests : baseline 89/89 préservée, monitor_router 4/4 OK (total 93/93).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 23:05:44 +02:00
Dom
fae95c5366 feat(qw1): capture par monitor + propagation offsets dans grounding cascade
_capture_screen() accepte un monitor_idx optionnel (None = composite legacy).
Index logique 0..N-1 mappé sur mss.monitors[idx+1] (mss[0] = composite).

Les 3 niveaux de grounding (OCR, UI-TARS, VLM) propagent l'offset retourné
par la capture pour traduire les coordonnées locales monitor en coordonnées
absolues écran (correct pour pyautogui.click).

find_element_on_screen() accepte monitor_idx et le forwarde aux 3 niveaux.

Backward 100% : monitor_idx=None partout → comportement strictement actuel.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 22:55:04 +02:00
Dom
6582a69d31 feat(qw1): MonitorRouter — résolution de l'écran cible pour le replay
Module isolé qui choisit l'écran cible avec stratégie en cascade :
1. action.monitor_index (session source) → cible explicite
2. session.last_focused_monitor → fallback focus actif
3. composite (offset 0,0) → backward compat (comportement actuel)

Backward 100% : actions sans monitor_index → fallback composite identique
au comportement mss.monitors[0] actuel.

Tests : 4 cas (cible OK, fallback focus, fallback composite, index invalide).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-05 22:50:22 +02:00
16 changed files with 913 additions and 30 deletions

View File

@@ -448,6 +448,12 @@ class AgentV1:
window_title = self.vision.get_active_window_title() window_title = self.vision.get_active_window_title()
if window_title: if window_title:
heartbeat_event["active_window_title"] = window_title heartbeat_event["active_window_title"] = window_title
# QW1 — enrichissement multi-écrans (additif, fallback gracieux)
try:
from .vision.capturer import _enrich_with_monitor_info
_enrich_with_monitor_info(heartbeat_event)
except Exception:
pass
self.streamer.push_event(heartbeat_event) self.streamer.push_event(heartbeat_event)
except Exception as e: except Exception as e:
logger.error(f"Heartbeat error: {e}") logger.error(f"Heartbeat error: {e}")

View File

@@ -5,6 +5,7 @@ Pillow>=10.0.0 # Crops et processing image
requests>=2.31.0 # Streaming réseau requests>=2.31.0 # Streaming réseau
python-socketio[client]>=5.10,<6.0 # Bus feedback Léa 'lea:*' (compat Flask-SocketIO 5.3.x serveur) python-socketio[client]>=5.10,<6.0 # Bus feedback Léa 'lea:*' (compat Flask-SocketIO 5.3.x serveur)
psutil>=5.9.0 # Monitoring CPU/RAM psutil>=5.9.0 # Monitoring CPU/RAM
screeninfo>=0.8 # QW1 — détection des monitors physiques + offsets
pystray>=0.19.5 # Icône Tray UI pystray>=0.19.5 # Icône Tray UI
plyer>=2.1.0 # Notifications toast natives (remplace PyQt5) plyer>=2.1.0 # Notifications toast natives (remplace PyQt5)
pywebview>=5.0 # Fenêtre de chat Léa intégrée (Edge WebView2 sur Windows) pywebview>=5.0 # Fenêtre de chat Léa intégrée (Edge WebView2 sur Windows)

View File

@@ -15,7 +15,7 @@ import time
import logging import logging
import hashlib import hashlib
import platform import platform
from typing import Any, Dict, Optional from typing import Any, Dict, List, Optional
from PIL import Image, ImageFilter, ImageStat from PIL import Image, ImageFilter, ImageStat
import mss import mss
from ..config import TARGETED_CROP_SIZE, SCREENSHOT_QUALITY, BLUR_SENSITIVE from ..config import TARGETED_CROP_SIZE, SCREENSHOT_QUALITY, BLUR_SENSITIVE
@@ -26,6 +26,66 @@ logger = logging.getLogger(__name__)
# OS courant (détecté une seule fois) # OS courant (détecté une seule fois)
_SYSTEM = platform.system() _SYSTEM = platform.system()
# QW1 — détection multi-écrans (fallback gracieux si screeninfo absent)
try:
from screeninfo import get_monitors as _screeninfo_get_monitors
_SCREENINFO_AVAILABLE = True
except ImportError:
_SCREENINFO_AVAILABLE = False
def _get_monitors_geometry() -> List[Dict[str, Any]]:
"""Retourne la liste des monitors physiques avec leurs offsets.
Returns:
List[dict] : [{idx, x, y, w, h, primary}, ...]. Vide si screeninfo
indisponible (le serveur tombera sur fallback composite).
"""
if not _SCREENINFO_AVAILABLE:
return []
try:
monitors = _screeninfo_get_monitors()
return [
{
"idx": i,
"x": int(m.x),
"y": int(m.y),
"w": int(m.width),
"h": int(m.height),
"primary": bool(getattr(m, "is_primary", False)),
}
for i, m in enumerate(monitors)
]
except Exception:
return []
def _get_active_monitor_index() -> Optional[int]:
"""Retourne l'index logique du monitor où se trouve le curseur (focus actif).
Returns:
int ou None si indéterminable.
"""
if not _SCREENINFO_AVAILABLE:
return None
try:
import pyautogui # import paresseux : évite la dépendance dure
cx, cy = pyautogui.position()
for i, m in enumerate(_screeninfo_get_monitors()):
if m.x <= cx < m.x + m.width and m.y <= cy < m.y + m.height:
return i
except Exception:
return None
return None
def _enrich_with_monitor_info(payload: dict) -> dict:
"""Ajoute monitor_index et monitors_geometry au payload (in-place + return)."""
if isinstance(payload, dict):
payload["monitor_index"] = _get_active_monitor_index()
payload["monitors_geometry"] = _get_monitors_geometry()
return payload
class VisionCapturer: class VisionCapturer:
def __init__(self, session_dir: str): def __init__(self, session_dir: str):
self.session_dir = session_dir self.session_dir = session_dir
@@ -121,6 +181,9 @@ class VisionCapturer:
if window_info: if window_info:
result["window_capture"] = window_info result["window_capture"] = window_info
# QW1 — enrichissement multi-écrans (additif, fallback gracieux)
_enrich_with_monitor_info(result)
return result return result
except Exception as e: except Exception as e:
logger.error(f"Erreur Dual Capture: {e}") logger.error(f"Erreur Dual Capture: {e}")
@@ -223,6 +286,9 @@ class VisionCapturer:
"click_inside_window": click_inside, "click_inside_window": click_inside,
} }
# QW1 — enrichissement multi-écrans (additif)
_enrich_with_monitor_info(result)
logger.debug( logger.debug(
f"Fenêtre capturée : {title} ({win_w}x{win_h}) — " f"Fenêtre capturée : {title} ({win_w}x{win_h}) — "
f"clic relatif ({click_rel_x}, {click_rel_y})" f"clic relatif ({click_rel_x}, {click_rel_y})"

View File

@@ -512,6 +512,21 @@ class ActionExecutorV1:
x_pct = action.get("x_pct", 0.0) x_pct = action.get("x_pct", 0.0)
y_pct = action.get("y_pct", 0.0) y_pct = action.get("y_pct", 0.0)
# QW1 — Si le serveur a résolu un monitor cible (idx >= 0),
# appliquer son offset aux coords absolues. Pour idx == -1
# (composite_fallback), aucun offset (backward compat).
# Le calcul des coords reste percent * (width/height) du monitor[1]
# côté client (x_pct est exprimé sur l'écran physique principal).
mon_res = action.get("monitor_resolution") or {}
mon_idx = mon_res.get("idx", -1)
mon_offset_x = mon_res.get("offset_x", 0) if mon_idx >= 0 else 0
mon_offset_y = mon_res.get("offset_y", 0) if mon_idx >= 0 else 0
if mon_idx >= 0 and (mon_offset_x or mon_offset_y):
logger.info(
f"[REPLAY] QW1 monitor cible idx={mon_idx} source={mon_res.get('source')} "
f"offset=({mon_offset_x},{mon_offset_y}) — appliqué aux coords"
)
# ── Diagnostic résolution ── # ── Diagnostic résolution ──
logger.info( logger.info(
f"[REPLAY] Action {action_id} ({action_type}) — " f"[REPLAY] Action {action_id} ({action_type}) — "
@@ -578,8 +593,8 @@ class ActionExecutorV1:
print(f" [OBSERVER] Popup détectée : '{popup_label}' — fermeture") print(f" [OBSERVER] Popup détectée : '{popup_label}' — fermeture")
logger.info(f"Observer : popup '{popup_label}' détectée avant résolution") logger.info(f"Observer : popup '{popup_label}' détectée avant résolution")
if popup_coords: if popup_coords:
real_x = int(popup_coords["x_pct"] * width) real_x = int(popup_coords["x_pct"] * width) + mon_offset_x
real_y = int(popup_coords["y_pct"] * height) real_y = int(popup_coords["y_pct"] * height) + mon_offset_y
self._click((real_x, real_y), "left") self._click((real_x, real_y), "left")
time.sleep(1.0) time.sleep(1.0)
print(f" [OBSERVER] Popup fermée — reprise du flow normal") print(f" [OBSERVER] Popup fermée — reprise du flow normal")
@@ -718,8 +733,8 @@ class ActionExecutorV1:
self.notifier.replay_target_not_found(target_desc) self.notifier.replay_target_not_found(target_desc)
return result return result
real_x = int(x_pct * width) real_x = int(x_pct * width) + mon_offset_x
real_y = int(y_pct * height) real_y = int(y_pct * height) + mon_offset_y
button = action.get("button", "left") button = action.get("button", "left")
mode = "VISUAL" if result.get("visual_resolved") else "COORD" mode = "VISUAL" if result.get("visual_resolved") else "COORD"
print( print(
@@ -781,8 +796,8 @@ class ActionExecutorV1:
print(f" [TYPE] raw_keys disponibles ({len(raw_keys)} events) — replay exact") print(f" [TYPE] raw_keys disponibles ({len(raw_keys)} events) — replay exact")
# Cliquer sur le champ avant de taper (si coordonnees disponibles) # Cliquer sur le champ avant de taper (si coordonnees disponibles)
if x_pct > 0 and y_pct > 0: if x_pct > 0 and y_pct > 0:
real_x = int(x_pct * width) real_x = int(x_pct * width) + mon_offset_x
real_y = int(y_pct * height) real_y = int(y_pct * height) + mon_offset_y
print(f" [TYPE] Clic prealable sur ({real_x}, {real_y})") print(f" [TYPE] Clic prealable sur ({real_x}, {real_y})")
self._click((real_x, real_y), "left") self._click((real_x, real_y), "left")
time.sleep(0.3) time.sleep(0.3)
@@ -808,8 +823,8 @@ class ActionExecutorV1:
logger.info(f"Replay key_combo : {keys} (raw_keys={'oui' if raw_keys else 'non'})") logger.info(f"Replay key_combo : {keys} (raw_keys={'oui' if raw_keys else 'non'})")
elif action_type == "scroll": elif action_type == "scroll":
real_x = int(x_pct * width) if x_pct > 0 else int(0.5 * width) real_x = (int(x_pct * width) if x_pct > 0 else int(0.5 * width)) + mon_offset_x
real_y = int(y_pct * height) if y_pct > 0 else int(0.5 * height) real_y = (int(y_pct * height) if y_pct > 0 else int(0.5 * height)) + mon_offset_y
delta = action.get("delta", -3) delta = action.get("delta", -3)
print(f" [SCROLL] delta={delta} a ({real_x}, {real_y})") print(f" [SCROLL] delta={delta} a ({real_x}, {real_y})")
self.mouse.position = (real_x, real_y) self.mouse.position = (real_x, real_y)
@@ -1386,6 +1401,16 @@ Example: x_pct=0.50, y_pct=0.30"""
data = resp.json() data = resp.json()
action = data.get("action") action = data.get("action")
if action is None: if action is None:
# pause_for_human : afficher le message de décision à l'utilisateur
if data.get("replay_paused") and data.get("pause_message"):
msg = data["pause_message"]
print(f"[PAUSE] {msg}")
logger.info(f"Replay en pause — message : {msg}")
self.notifier.notify(
title="Léa — Validation requise",
message=msg[:250],
timeout=30,
)
return False return False
except (requests.exceptions.ConnectionError, requests.exceptions.Timeout) as e: except (requests.exceptions.ConnectionError, requests.exceptions.Timeout) as e:

View File

@@ -8,12 +8,73 @@ import os
import time import time
import logging import logging
import hashlib import hashlib
from typing import Any, Dict, List, Optional
from PIL import Image, ImageFilter, ImageStat from PIL import Image, ImageFilter, ImageStat
import mss import mss
from ..config import TARGETED_CROP_SIZE, SCREENSHOT_QUALITY from ..config import TARGETED_CROP_SIZE, SCREENSHOT_QUALITY
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
# QW1 — détection multi-écrans (fallback gracieux si screeninfo absent)
try:
from screeninfo import get_monitors as _screeninfo_get_monitors
_SCREENINFO_AVAILABLE = True
except ImportError:
_SCREENINFO_AVAILABLE = False
def _get_monitors_geometry() -> List[Dict[str, Any]]:
"""Retourne la liste des monitors physiques avec leurs offsets.
Returns:
List[dict] : [{idx, x, y, w, h, primary}, ...]. Vide si screeninfo
indisponible (le serveur tombera sur fallback composite).
"""
if not _SCREENINFO_AVAILABLE:
return []
try:
monitors = _screeninfo_get_monitors()
return [
{
"idx": i,
"x": int(m.x),
"y": int(m.y),
"w": int(m.width),
"h": int(m.height),
"primary": bool(getattr(m, "is_primary", False)),
}
for i, m in enumerate(monitors)
]
except Exception:
return []
def _get_active_monitor_index() -> Optional[int]:
"""Retourne l'index logique du monitor où se trouve le curseur (focus actif).
Returns:
int ou None si indéterminable.
"""
if not _SCREENINFO_AVAILABLE:
return None
try:
import pyautogui # import paresseux : évite la dépendance dure
cx, cy = pyautogui.position()
for i, m in enumerate(_screeninfo_get_monitors()):
if m.x <= cx < m.x + m.width and m.y <= cy < m.y + m.height:
return i
except Exception:
return None
return None
def _enrich_with_monitor_info(payload: dict) -> dict:
"""Ajoute monitor_index et monitors_geometry au payload (in-place + return)."""
if isinstance(payload, dict):
payload["monitor_index"] = _get_active_monitor_index()
payload["monitors_geometry"] = _get_monitors_geometry()
return payload
class VisionCapturer: class VisionCapturer:
def __init__(self, session_dir: str): def __init__(self, session_dir: str):
self.session_dir = session_dir self.session_dir = session_dir
@@ -72,7 +133,12 @@ class VisionCapturer:
# Mise à jour du hash pour le prochain heartbeat # Mise à jour du hash pour le prochain heartbeat
self.last_img_hash = self._compute_quick_hash(img) self.last_img_hash = self._compute_quick_hash(img)
return {"full": full_path, "crop": crop_path} result = {"full": full_path, "crop": crop_path}
# QW1 — enrichissement multi-écrans (additif, fallback gracieux)
_enrich_with_monitor_info(result)
return result
except Exception as e: except Exception as e:
logger.error(f"Erreur Dual Capture: {e}") logger.error(f"Erreur Dual Capture: {e}")
return {} return {}

View File

@@ -5,6 +5,7 @@ Pillow>=10.0.0 # Crops et processing image
requests>=2.31.0 # Streaming réseau requests>=2.31.0 # Streaming réseau
python-socketio[client]>=5.10,<6.0 # Bus feedback Léa 'lea:*' (compat Flask-SocketIO 5.3.x serveur) python-socketio[client]>=5.10,<6.0 # Bus feedback Léa 'lea:*' (compat Flask-SocketIO 5.3.x serveur)
psutil>=5.9.0 # Monitoring CPU/RAM psutil>=5.9.0 # Monitoring CPU/RAM
screeninfo>=0.8 # QW1 — détection des monitors physiques + offsets
pystray>=0.19.5 # Icône Tray UI pystray>=0.19.5 # Icône Tray UI
plyer>=2.1.0 # Notifications toast natives (remplace PyQt5) plyer>=2.1.0 # Notifications toast natives (remplace PyQt5)

View File

@@ -33,6 +33,8 @@ from .audit_trail import AuditTrail, AuditEntry
from .agent_registry import AgentRegistry, AgentAlreadyEnrolledError from .agent_registry import AgentRegistry, AgentAlreadyEnrolledError
from .stream_processor import StreamProcessor, build_replay_from_raw_events, enrich_click_from_screenshot from .stream_processor import StreamProcessor, build_replay_from_raw_events, enrich_click_from_screenshot
from .worker_stream import StreamWorker from .worker_stream import StreamWorker
from .monitor_router import resolve_target_monitor # QW1 — résolution écran cible
from .loop_detector import LoopDetector # QW2 — détection de boucle pendant replay
from .execution_plan_runner import ( from .execution_plan_runner import (
execution_plan_to_actions, execution_plan_to_actions,
inject_plan_into_queue, inject_plan_into_queue,
@@ -222,6 +224,7 @@ from .replay_engine import (
_resolve_runtime_vars, _resolve_runtime_vars,
_SERVER_SIDE_ACTION_TYPES, _SERVER_SIDE_ACTION_TYPES,
_handle_extract_text_action, _handle_extract_text_action,
_handle_extract_table_action,
_handle_t2a_decision_action, _handle_t2a_decision_action,
_expand_compound_steps, _expand_compound_steps,
_pre_check_screen_state as _pre_check_screen_state_impl, _pre_check_screen_state as _pre_check_screen_state_impl,
@@ -359,6 +362,18 @@ REPLAY_LOCK_FILE = _DATA_DIR / "_replay_active.lock"
processor = StreamProcessor(data_dir=str(LIVE_SESSIONS_DIR)) processor = StreamProcessor(data_dir=str(LIVE_SESSIONS_DIR))
worker = StreamWorker(live_dir=str(LIVE_SESSIONS_DIR), processor=processor) worker = StreamWorker(live_dir=str(LIVE_SESSIONS_DIR), processor=processor)
# QW2 — LoopDetector singleton lazy (utilise le CLIP embedder du processor)
_loop_detector: Optional["LoopDetector"] = None
def _get_loop_detector() -> "LoopDetector":
"""Singleton lazy — crée le LoopDetector avec le CLIP embedder du processor."""
global _loop_detector
if _loop_detector is None:
embedder = getattr(processor, "_clip_embedder", None)
_loop_detector = LoopDetector(clip_embedder=embedder)
return _loop_detector
# Registre des postes Lea enroles (table enrolled_agents dans rpa_data.db) # Registre des postes Lea enroles (table enrolled_agents dans rpa_data.db)
# Emplacement configurable via RPA_AGENTS_DB_PATH pour les tests. # Emplacement configurable via RPA_AGENTS_DB_PATH pour les tests.
_AGENTS_DB_PATH = os.environ.get( _AGENTS_DB_PATH = os.environ.get(
@@ -511,6 +526,7 @@ class ReplayRequest(BaseModel):
session_id: str session_id: str
machine_id: Optional[str] = None # Machine cible pour le replay (multi-machine) machine_id: Optional[str] = None # Machine cible pour le replay (multi-machine)
params: Optional[Dict[str, Any]] = None params: Optional[Dict[str, Any]] = None
variables: Optional[Dict[str, Any]] = None # Variables runtime initiales (templating {{var}})
class RawReplayRequest(BaseModel): class RawReplayRequest(BaseModel):
@@ -765,6 +781,21 @@ async def startup():
_cleanup_thread = threading.Thread(target=_cleanup_loop, daemon=True, name="replay_cleanup") _cleanup_thread = threading.Thread(target=_cleanup_loop, daemon=True, name="replay_cleanup")
_cleanup_thread.start() _cleanup_thread.start()
# Préchargement EasyOCR en arrière-plan : sans ça, le 1er extract_text /
# extract_table déclenche un cold start de ~3-5s qui bloque l'event loop
# FastAPI (constaté 2026-05-05 : streaming server inaccessible 2 min).
# Le thread tourne pendant que le boot continue ; le 1er appel OCR sera rapide.
def _preload_easyocr():
try:
t0 = time.time()
from core.llm.ocr_extractor import _get_reader
_get_reader()
logger.info("[OCR] EasyOCR préchargé (fr+en, CPU) en %.1fs", time.time() - t0)
except Exception as e:
logger.warning("[OCR] Échec préchargement EasyOCR : %s", e)
threading.Thread(target=_preload_easyocr, daemon=True, name="preload_easyocr").start()
logger.info( logger.info(
"API Streaming démarrée — StreamProcessor, Worker et Cleanup prêts. " "API Streaming démarrée — StreamProcessor, Worker et Cleanup prêts. "
"VLM Worker dans un process séparé (run_worker.py)." "VLM Worker dans un process séparé (run_worker.py)."
@@ -1962,6 +1993,11 @@ async def start_replay(request: ReplayRequest):
machine_id=resolved_machine_id, machine_id=resolved_machine_id,
actions=actions, actions=actions,
) )
# Pré-injection des variables runtime (templating {{var}} sur by_text,
# text, target_spec.* etc.). Permet à l'orchestrateur d'appeler ce
# workflow avec p.ex. variables={"patient_id": "25003284"} pour boucler.
if request.variables:
_replay_states[replay_id]["variables"].update(request.variables)
# Enregistrer le mapping machine -> session pour le replay ciblé # Enregistrer le mapping machine -> session pour le replay ciblé
if resolved_machine_id and resolved_machine_id != "default": if resolved_machine_id and resolved_machine_id != "default":
_machine_replay_target[resolved_machine_id] = session_id _machine_replay_target[resolved_machine_id] = session_id
@@ -2914,6 +2950,12 @@ async def get_next_action(session_id: str, machine_id: str = "default"):
_handle_extract_text_action, _handle_extract_text_action,
action, owning_replay, session_id, _last_heartbeat, action, owning_replay, session_id, _last_heartbeat,
) )
elif type_ == "extract_table":
await loop.run_in_executor(
None,
_handle_extract_table_action,
action, owning_replay, session_id, _last_heartbeat,
)
elif type_ == "t2a_decision": elif type_ == "t2a_decision":
await loop.run_in_executor( await loop.run_in_executor(
None, None,
@@ -3117,6 +3159,51 @@ async def get_next_action(session_id: str, machine_id: str = "default"):
f"{_precheck_sim}" f"{_precheck_sim}"
) )
# QW1 — Résoudre l'écran cible et joindre l'info à l'action
# Cascade : action.monitor_index → session.last_focused_monitor → composite_fallback
try:
session_qw1 = processor.session_manager.get_session(session_id)
last_window_info_qw1 = (
session_qw1.last_window_info if session_qw1 is not None else {}
) or {}
session_state_qw1 = {
"monitors_geometry": last_window_info_qw1.get("monitors_geometry", []),
"last_focused_monitor": last_window_info_qw1.get("monitor_index"),
}
target = resolve_target_monitor(action, session_state_qw1)
action["monitor_resolution"] = {
"idx": target.idx,
"offset_x": target.offset_x,
"offset_y": target.offset_y,
"w": target.w,
"h": target.h,
"source": target.source,
}
# QW1 — Émission bus lea:monitor_routed (no-op si bus indisponible)
# Le serveur streaming n'a pas de SocketIO local : on logge en INFO
# bien lisible. Un consommateur (agent_chat / dashboard) peut tailer
# `journalctl -u rpa-streaming | grep '\[BUS\] lea:monitor_routed'`.
try:
_replay_id_bus = (
owning_replay.get("replay_id") if owning_replay else None
)
logger.info(
"[BUS] lea:monitor_routed replay=%s action=%s idx=%d source=%s "
"offset=(%d,%d) wh=(%d,%d)",
_replay_id_bus,
action.get("action_id"),
target.idx,
target.source,
target.offset_x,
target.offset_y,
target.w,
target.h,
)
except Exception as _e_bus:
logger.debug("emit lea:monitor_routed échec (non bloquant): %s", _e_bus)
except Exception as e:
logger.debug("QW1 monitor_resolution skip (%s)", e)
response: Dict[str, Any] = { response: Dict[str, Any] = {
"action": action, "action": action,
"session_id": session_id, "session_id": session_id,
@@ -3855,6 +3942,82 @@ async def report_action_result(report: ReplayResultReport):
f"— worker VLM autorisé à reprendre" f"— worker VLM autorisé à reprendre"
) )
# ===================================================================
# QW2 — LoopDetector : alimentation des anneaux + évaluation
# ===================================================================
# On n'évalue que si le replay est encore "running" — inutile de
# pauser quelque chose de déjà completed/error/paused.
if replay_state["status"] == "running":
# Snapshot image (PIL) dans l'anneau
try:
from PIL import Image
ss_raw = screenshot_after or replay_state.get("last_screenshot")
img = None
if isinstance(ss_raw, str) and ss_raw:
if os.path.isfile(ss_raw):
img = Image.open(ss_raw).copy() # détache du file handle
else:
# Possible base64 — décoder
try:
import base64
import io as _io
img_bytes = base64.b64decode(ss_raw, validate=False)
img = Image.open(_io.BytesIO(img_bytes)).copy()
except Exception:
img = None
if img is not None:
replay_state.setdefault("_screenshot_history", []).append(img)
replay_state["_screenshot_history"] = replay_state["_screenshot_history"][-5:]
except Exception as e:
logger.debug("LoopDetector: snapshot historique échoué: %s", e)
# Snapshot signature de l'action courante
try:
_act_pos = report.actual_position or {}
action_sig = {
"type": (original_action or {}).get("type")
or replay_state.get("_last_action_type", ""),
"x_pct": _act_pos.get("x_pct") if isinstance(_act_pos, dict)
else (original_action or {}).get("x_pct"),
"y_pct": _act_pos.get("y_pct") if isinstance(_act_pos, dict)
else (original_action or {}).get("y_pct"),
}
replay_state.setdefault("_action_history", []).append(action_sig)
replay_state["_action_history"] = replay_state["_action_history"][-5:]
except Exception as e:
logger.debug("LoopDetector: snapshot action_sig échoué: %s", e)
# Évaluation (silencieux si rien)
try:
verdict = _get_loop_detector().evaluate(
replay_state,
screenshots=replay_state.get("_screenshot_history", []),
actions=replay_state.get("_action_history", []),
)
if verdict.detected:
replay_state["status"] = "paused_need_help"
replay_state["pause_reason"] = "loop_detected"
replay_state["pause_message"] = (
f"Léa semble bloquée — {verdict.signal} "
f"(détail: {verdict.evidence})"
)
logger.warning(
"LoopDetector: replay %s mis en pause — signal=%s evidence=%s",
replay_state["replay_id"], verdict.signal, verdict.evidence,
)
# Bus event d'observabilité (logger pattern QW1)
try:
logger.info(
"[BUS] lea:loop_detected replay=%s signal=%s evidence=%s",
replay_state["replay_id"],
verdict.signal,
verdict.evidence,
)
except Exception as _e_bus:
logger.debug("emit lea:loop_detected échec: %s", _e_bus)
except Exception as e:
logger.warning("LoopDetector: évaluation échouée (non bloquant): %s", e)
return { return {
"status": "recorded", "status": "recorded",
"action_id": action_id, "action_id": action_id,

View File

@@ -256,6 +256,20 @@ class LiveSessionManager:
session.last_window_info["title"] = wc_title session.last_window_info["title"] = wc_title
if wc_app: if wc_app:
session.last_window_info["app_name"] = wc_app session.last_window_info["app_name"] = wc_app
# QW1 — propager monitor_index et monitors_geometry depuis window_capture
if "monitor_index" in window_capture:
session.last_window_info["monitor_index"] = window_capture["monitor_index"]
if "monitors_geometry" in window_capture:
session.last_window_info["monitors_geometry"] = window_capture["monitors_geometry"]
# QW1 — propager monitor_index/monitors_geometry du payload event
# (cas heartbeat enrichi sans window/window_title). Toujours
# rafraîchir le focus actif (change souvent) et la géométrie
# (l'utilisateur peut brancher/débrancher un écran).
if "monitor_index" in event_data:
session.last_window_info["monitor_index"] = event_data["monitor_index"]
if "monitors_geometry" in event_data and event_data["monitors_geometry"]:
session.last_window_info["monitors_geometry"] = event_data["monitors_geometry"]
# Accumuler les titres/apps pour le nommage automatique # Accumuler les titres/apps pour le nommage automatique
title = session.last_window_info.get("title", "").strip() title = session.last_window_info.get("title", "").strip()

View File

@@ -0,0 +1,154 @@
# agent_v0/server_v1/loop_detector.py
"""LoopDetector composite — détection de stagnation de Léa pendant un replay (QW2).
Trois signaux indépendants :
- screen_static : N captures consécutives avec CLIP similarity > seuil
- action_repeat : N actions consécutives identiques (type + coords)
- retry_threshold : nombre de retries cumulés >= seuil
Un seul signal positif → verdict.detected=True. Le serveur bascule alors le
replay en paused_need_help avec pause_reason explicite.
Désactivable via env var RPA_LOOP_DETECTOR_ENABLED=0.
"""
import logging
import os
from dataclasses import dataclass, field
from typing import Any, Dict, List, Optional
logger = logging.getLogger(__name__)
@dataclass
class LoopVerdict:
detected: bool = False
reason: str = ""
signal: str = "" # "screen_static" | "action_repeat" | "retry_threshold" | ""
evidence: Dict[str, Any] = field(default_factory=dict)
def _env_int(name: str, default: int) -> int:
try:
return int(os.environ.get(name, default))
except (TypeError, ValueError):
return default
def _env_float(name: str, default: float) -> float:
try:
return float(os.environ.get(name, default))
except (TypeError, ValueError):
return default
def _env_bool_enabled(name: str) -> bool:
val = os.environ.get(name, "1").strip().lower()
return val not in ("0", "false", "no", "off", "")
def _cosine_similarity(a, b) -> float:
"""Similarité cosine entre deux vecteurs (listes ou np.array). Robuste vecteur nul."""
import numpy as np
av = np.asarray(a, dtype=np.float32).flatten()
bv = np.asarray(b, dtype=np.float32).flatten()
na, nb = float(np.linalg.norm(av)), float(np.linalg.norm(bv))
if na < 1e-8 or nb < 1e-8:
return 0.0
return float(np.dot(av, bv) / (na * nb))
class LoopDetector:
def __init__(self, clip_embedder=None):
self.clip_embedder = clip_embedder
def evaluate(
self,
state: Dict[str, Any],
screenshots: List[Any],
actions: List[Dict[str, Any]],
) -> LoopVerdict:
"""Évalue les 3 signaux. Retourne le premier déclenché.
Args:
state: replay_state (utilisé pour retried_actions)
screenshots: anneau d'embeddings CLIP (les N derniers)
actions: anneau des N dernières actions exécutées
"""
if not _env_bool_enabled("RPA_LOOP_DETECTOR_ENABLED"):
return LoopVerdict(detected=False)
# Signal A : screen_static
verdict = self._check_screen_static(screenshots)
if verdict.detected:
return verdict
# Signal B : action_repeat
verdict = self._check_action_repeat(actions)
if verdict.detected:
return verdict
# Signal C : retry_threshold
verdict = self._check_retry_threshold(state)
if verdict.detected:
return verdict
return LoopVerdict(detected=False)
def _check_screen_static(self, screenshots: List[Any]) -> LoopVerdict:
n_required = _env_int("RPA_LOOP_SCREEN_STATIC_N", 4)
threshold = _env_float("RPA_LOOP_SCREEN_STATIC_THRESHOLD", 0.99)
if self.clip_embedder is None or len(screenshots) < n_required:
return LoopVerdict()
try:
recent = screenshots[-n_required:]
# Embed chaque capture via le CLIP embedder (peut lever)
embeddings = [self.clip_embedder.embed_image(img) for img in recent]
sims = [_cosine_similarity(embeddings[i], embeddings[i + 1])
for i in range(len(embeddings) - 1)]
min_sim = min(sims)
if min_sim > threshold:
return LoopVerdict(
detected=True,
reason="loop_detected",
signal="screen_static",
evidence={"min_similarity": round(min_sim, 4),
"n_captures": n_required,
"threshold": threshold},
)
except Exception as e:
logger.warning("LoopDetector signal_A erreur (%s) — signal inerte ce tick", e)
return LoopVerdict()
def _check_action_repeat(self, actions: List[Dict[str, Any]]) -> LoopVerdict:
n_required = _env_int("RPA_LOOP_ACTION_REPEAT_N", 3)
if len(actions) < n_required:
return LoopVerdict()
recent = actions[-n_required:]
def _signature(a: Dict[str, Any]) -> tuple:
return (a.get("type"), a.get("x_pct"), a.get("y_pct"))
sigs = [_signature(a) for a in recent]
if all(s == sigs[0] for s in sigs):
return LoopVerdict(
detected=True,
reason="loop_detected",
signal="action_repeat",
evidence={"signature": sigs[0], "count": n_required},
)
return LoopVerdict()
def _check_retry_threshold(self, state: Dict[str, Any]) -> LoopVerdict:
threshold = _env_int("RPA_LOOP_RETRY_THRESHOLD", 3)
retried = int(state.get("retried_actions", 0))
if retried >= threshold:
return LoopVerdict(
detected=True,
reason="loop_detected",
signal="retry_threshold",
evidence={"retried_actions": retried, "threshold": threshold},
)
return LoopVerdict()

View File

@@ -0,0 +1,88 @@
# agent_v0/server_v1/monitor_router.py
"""MonitorRouter — résolution de l'écran cible pour le replay (QW1).
Stratégie en cascade :
1. action.monitor_index (hérité de la session source) → cible cet écran
2. session.last_focused_monitor (focus actif vu en dernier heartbeat) → fallback
3. composite (offset 0, 0) → backward compat
Émet sur le bus lea:* l'event monitor_routed avec la source de la décision.
"""
from dataclasses import dataclass
from typing import Any, Dict, List, Optional
@dataclass
class MonitorTarget:
"""Représente l'écran cible résolu pour une action de replay."""
idx: int
offset_x: int
offset_y: int
w: int
h: int
source: str # "action" | "focus" | "composite_fallback"
_COMPOSITE_FALLBACK = MonitorTarget(
idx=-1,
offset_x=0,
offset_y=0,
w=0,
h=0,
source="composite_fallback",
)
def _find_monitor(geometry: List[Dict[str, Any]], idx: int) -> Optional[Dict[str, Any]]:
"""Retourne le monitor d'index donné, ou None si absent."""
for m in geometry:
if m.get("idx") == idx:
return m
return None
def _to_target(monitor: Dict[str, Any], source: str) -> MonitorTarget:
return MonitorTarget(
idx=int(monitor["idx"]),
offset_x=int(monitor.get("x", 0)),
offset_y=int(monitor.get("y", 0)),
w=int(monitor.get("w", 0)),
h=int(monitor.get("h", 0)),
source=source,
)
def resolve_target_monitor(
action: Dict[str, Any],
session_state: Dict[str, Any],
) -> MonitorTarget:
"""Résout l'écran cible d'une action de replay.
Args:
action: Dict de l'action (peut contenir `monitor_index`).
session_state: État de la session (doit contenir `monitors_geometry`
et `last_focused_monitor`).
Returns:
MonitorTarget avec l'offset à appliquer aux coordonnées de grounding.
"""
geometry: List[Dict[str, Any]] = session_state.get("monitors_geometry") or []
# 1. Cible explicite via action
explicit_idx = action.get("monitor_index")
if explicit_idx is not None and geometry:
m = _find_monitor(geometry, int(explicit_idx))
if m is not None:
return _to_target(m, source="action")
# Index invalide → on tombe sur le fallback focus
# 2. Fallback focus actif
focused_idx = session_state.get("last_focused_monitor")
if focused_idx is not None and geometry:
m = _find_monitor(geometry, int(focused_idx))
if m is not None:
return _to_target(m, source="focus")
# 3. Fallback composite (backward compat — comportement actuel mss.monitors[0])
return _COMPOSITE_FALLBACK

View File

@@ -1381,6 +1381,9 @@ def _create_replay_state(
# t2a_decision, etc.). Résolues via templating {{var}} ou {{var.field}} # t2a_decision, etc.). Résolues via templating {{var}} ou {{var.field}}
# dans les paramètres des actions suivantes. # dans les paramètres des actions suivantes.
"variables": {}, "variables": {},
# QW2 — Anneaux d'historique pour LoopDetector (5 derniers max)
"_screenshot_history": [], # images PIL des N derniers heartbeats (LoopDetector embed à chaque tick)
"_action_history": [], # N dernières actions exécutées (signature)
} }

View File

@@ -22,6 +22,18 @@ try:
except ImportError: except ImportError:
PYAUTOGUI_AVAILABLE = False PYAUTOGUI_AVAILABLE = False
try:
import mss
MSS_AVAILABLE = True
except ImportError:
MSS_AVAILABLE = False
try:
from PIL import Image as PILImage
PIL_AVAILABLE = True
except ImportError:
PIL_AVAILABLE = False
def safe_type_text(text: str): def safe_type_text(text: str):
"""Saisie de texte compatible VM/Citrix et claviers AZERTY/QWERTY. """Saisie de texte compatible VM/Citrix et claviers AZERTY/QWERTY.
@@ -157,11 +169,13 @@ def handle_detected_pattern(pattern: Dict[str, Any]) -> bool:
screenshot = sct.grab(monitor) screenshot = sct.grab(monitor)
screen = Image.frombytes('RGB', screenshot.size, screenshot.bgra, 'raw', 'BGRX') screen = Image.frombytes('RGB', screenshot.size, screenshot.bgra, 'raw', 'BGRX')
# EasyOCR (rapide, bonne qualité GUI) avec fallback docTR # EasyOCR (rapide, bonne qualité GUI) avec fallback docTR.
# gpu=True : harmonisé avec dialog_handler.py et title_verifier.py.
# Coût VRAM ~0.5 GB, sous le budget RTX 5070 (cf. deploy/VRAM_BUDGET.md).
words = [] words = []
try: try:
import easyocr import easyocr
_reader = easyocr.Reader(['fr', 'en'], gpu=False, verbose=False) _reader = easyocr.Reader(['fr', 'en'], gpu=True, verbose=False)
results = _reader.readtext(np.array(screen)) results = _reader.readtext(np.array(screen))
for (bbox_pts, text, conf) in results: for (bbox_pts, text, conf) in results:
if not text or len(text.strip()) < 1: if not text or len(text.strip()) < 1:
@@ -312,6 +326,7 @@ def find_element_on_screen(
target_description: str = "", target_description: str = "",
anchor_image_base64: Optional[str] = None, anchor_image_base64: Optional[str] = None,
anchor_bbox: Optional[Dict] = None, anchor_bbox: Optional[Dict] = None,
monitor_idx: Optional[int] = None,
) -> Optional[Dict[str, Any]]: ) -> Optional[Dict[str, Any]]:
""" """
Cherche un élément sur l'écran en utilisant 3 méthodes en cascade. Cherche un élément sur l'écran en utilisant 3 méthodes en cascade.
@@ -325,6 +340,7 @@ def find_element_on_screen(
target_description: Description plus longue (ex: "le dossier Demo sur le bureau") target_description: Description plus longue (ex: "le dossier Demo sur le bureau")
anchor_image_base64: Image de référence de l'ancre (pour CLIP matching, réservé futur) anchor_image_base64: Image de référence de l'ancre (pour CLIP matching, réservé futur)
anchor_bbox: Position originale de l'ancre (pour désambiguïser les matchs multiples) anchor_bbox: Position originale de l'ancre (pour désambiguïser les matchs multiples)
monitor_idx: Index logique 0..N-1 du monitor à scruter. None = composite legacy.
Returns: Returns:
{'x': int, 'y': int, 'method': str, 'confidence': float} ou None {'x': int, 'y': int, 'method': str, 'confidence': float} ou None
@@ -347,6 +363,13 @@ def find_element_on_screen(
logger.debug("find_element_on_screen: ni target_text ni target_description fournis") logger.debug("find_element_on_screen: ni target_text ni target_description fournis")
return None return None
# Propager monitor_idx au niveau OCR via anchor_bbox (sans muter l'argument original)
if monitor_idx is not None and anchor_bbox is not None:
anchor_bbox = dict(anchor_bbox) # copie pour ne pas muter l'argument
anchor_bbox["monitor_idx"] = monitor_idx
elif monitor_idx is not None:
anchor_bbox = {"monitor_idx": monitor_idx}
search_label = target_description or target_text search_label = target_description or target_text
logger.info(f"[Grounding] Recherche élément: '{search_label}' (cascade 3 niveaux)") logger.info(f"[Grounding] Recherche élément: '{search_label}' (cascade 3 niveaux)")
@@ -356,12 +379,12 @@ def find_element_on_screen(
return result return result
# ─── Niveau 2 — UI-TARS grounding (~3s) ─── # ─── Niveau 2 — UI-TARS grounding (~3s) ───
result = _grounding_ui_tars(target_text, target_description) result = _grounding_ui_tars(target_text, target_description, monitor_idx=monitor_idx)
if result: if result:
return result return result
# ─── Niveau 3 — VLM reasoning (~10s) ─── # ─── Niveau 3 — VLM reasoning (~10s) ───
result = _grounding_vlm(target_text, target_description) result = _grounding_vlm(target_text, target_description, monitor_idx=monitor_idx)
if result: if result:
return result return result
@@ -411,20 +434,43 @@ def _describe_anchor_image(anchor_image_base64: str) -> Optional[str]:
return None return None
def _capture_screen(): def _capture_screen(monitor_idx=None):
"""Capture l'écran principal et retourne (PIL.Image, width, height).""" """Capture l'écran et retourne (PIL.Image, width, height, offset_x, offset_y).
try:
import mss
from PIL import Image as PILImage
Args:
monitor_idx: Index logique 0..N-1 du monitor à capturer (cf. screeninfo).
Si None : capture composite (mss.monitors[0]) — comportement legacy.
Returns:
(image, w, h, offset_x, offset_y). offset = (0,0) en mode composite.
"""
try:
with mss.mss() as sct: with mss.mss() as sct:
monitor = sct.monitors[0] if monitor_idx is None:
# Comportement actuel : composite tous écrans
monitor = sct.monitors[0]
offset_x, offset_y = 0, 0
else:
# mss skip monitors[0] (composite). Index logique 0 → mss.monitors[1].
mss_idx = int(monitor_idx) + 1
if mss_idx >= len(sct.monitors):
logger.warning(
"mss.monitors[%d] hors limites (n=%d) — fallback composite",
mss_idx, len(sct.monitors),
)
monitor = sct.monitors[0]
offset_x, offset_y = 0, 0
else:
monitor = sct.monitors[mss_idx]
offset_x = int(monitor.get("left", 0))
offset_y = int(monitor.get("top", 0))
screenshot = sct.grab(monitor) screenshot = sct.grab(monitor)
screen = PILImage.frombytes('RGB', screenshot.size, screenshot.bgra, 'raw', 'BGRX') screen = PILImage.frombytes('RGB', screenshot.size, screenshot.bgra, 'raw', 'BGRX')
return screen, monitor['width'], monitor['height'] return screen, monitor['width'], monitor['height'], offset_x, offset_y
except Exception as e: except Exception as e:
logger.debug(f"Capture écran échouée: {e}") logger.debug(f"Capture écran échouée: {e}")
return None, 0, 0 return None, 0, 0, 0, 0
def _grounding_ocr(target_text: str, anchor_bbox: Optional[Dict] = None) -> Optional[Dict[str, Any]]: def _grounding_ocr(target_text: str, anchor_bbox: Optional[Dict] = None) -> Optional[Dict[str, Any]]:
@@ -439,7 +485,8 @@ def _grounding_ocr(target_text: str, anchor_bbox: Optional[Dict] = None) -> Opti
return None return None
try: try:
screen, screen_w, screen_h = _capture_screen() monitor_idx_param = anchor_bbox.get("monitor_idx") if anchor_bbox else None
screen, screen_w, screen_h, ox, oy = _capture_screen(monitor_idx=monitor_idx_param)
if screen is None: if screen is None:
return None return None
@@ -503,14 +550,14 @@ def _grounding_ocr(target_text: str, anchor_bbox: Optional[Dict] = None) -> Opti
sel = " ← CHOISI" if m is best else "" sel = " ← CHOISI" if m is best else ""
logger.info(f" [OCR] Candidat: '{m['text']}' à ({m['x']}, {m['y']}) [{m['type']}]{sel}") logger.info(f" [OCR] Candidat: '{m['text']}' à ({m['x']}, {m['y']}) [{m['type']}]{sel}")
return {'x': best['x'], 'y': best['y'], 'method': 'ocr', 'confidence': best['conf']} return {'x': best['x'] + ox, 'y': best['y'] + oy, 'method': 'ocr', 'confidence': best['conf']}
except Exception as e: except Exception as e:
logger.debug(f"[Grounding/OCR] Erreur: {e}") logger.debug(f"[Grounding/OCR] Erreur: {e}")
return None return None
def _grounding_ui_tars(target_text: str, target_description: str = "") -> Optional[Dict[str, Any]]: def _grounding_ui_tars(target_text: str, target_description: str = "", monitor_idx=None) -> Optional[Dict[str, Any]]:
"""Niveau 2 — UI-TARS grounding visuel (~3s).""" """Niveau 2 — UI-TARS grounding visuel (~3s)."""
try: try:
import requests import requests
@@ -519,7 +566,7 @@ def _grounding_ui_tars(target_text: str, target_description: str = "") -> Option
import re import re
import os import os
screen, screen_w, screen_h = _capture_screen() screen, screen_w, screen_h, ox, oy = _capture_screen(monitor_idx=monitor_idx)
if screen is None: if screen is None:
return None return None
@@ -564,7 +611,7 @@ def _grounding_ui_tars(target_text: str, target_description: str = "") -> Option
# Valider que les coordonnées sont dans l'écran # Valider que les coordonnées sont dans l'écran
if 0 <= x <= screen_w and 0 <= y <= screen_h: if 0 <= x <= screen_w and 0 <= y <= screen_h:
logger.info(f"[Grounding/UI-TARS] Grounding → ({x}, {y})") logger.info(f"[Grounding/UI-TARS] Grounding → ({x}, {y})")
return {'x': x, 'y': y, 'method': 'ui_tars', 'confidence': 0.85} return {'x': x + ox, 'y': y + oy, 'method': 'ui_tars', 'confidence': 0.85}
else: else:
logger.warning(f"[Grounding/UI-TARS] Coordonnées hors écran: ({x}, {y}) pour {screen_w}x{screen_h}") logger.warning(f"[Grounding/UI-TARS] Coordonnées hors écran: ({x}, {y}) pour {screen_w}x{screen_h}")
return None return None
@@ -624,7 +671,7 @@ def _parse_ui_tars_coordinates(text: str, screen_w: int, screen_h: int) -> Optio
return None return None
def _grounding_vlm(target_text: str, target_description: str = "") -> Optional[Dict[str, Any]]: def _grounding_vlm(target_text: str, target_description: str = "", monitor_idx=None) -> Optional[Dict[str, Any]]:
"""Niveau 3 — VLM reasoning + confirmation OCR (~10s).""" """Niveau 3 — VLM reasoning + confirmation OCR (~10s)."""
try: try:
search_label = target_description or target_text search_label = target_description or target_text
@@ -646,7 +693,7 @@ def _grounding_vlm(target_text: str, target_description: str = "") -> Optional[D
logger.info(f"[Grounding/VLM] VLM suggère de cliquer sur: '{vlm_target}'") logger.info(f"[Grounding/VLM] VLM suggère de cliquer sur: '{vlm_target}'")
# Confirmation par OCR : chercher le target VLM sur l'écran # Confirmation par OCR : chercher le target VLM sur l'écran
screen, screen_w, screen_h = _capture_screen() screen, screen_w, screen_h, ox, oy = _capture_screen(monitor_idx=monitor_idx)
if screen is None: if screen is None:
return None return None
@@ -668,7 +715,7 @@ def _grounding_vlm(target_text: str, target_description: str = "") -> Optional[D
x = int((x1 + x2) / 2) x = int((x1 + x2) / 2)
y = int((y1 + y2) / 2) y = int((y1 + y2) / 2)
logger.info(f"[Grounding/VLM] Confirmé par OCR: '{word['text']}' à ({x}, {y})") logger.info(f"[Grounding/VLM] Confirmé par OCR: '{word['text']}' à ({x}, {y})")
return {'x': x, 'y': y, 'method': 'vlm', 'confidence': 0.75} return {'x': x + ox, 'y': y + oy, 'method': 'vlm', 'confidence': 0.75}
logger.debug(f"[Grounding/VLM] Target VLM '{vlm_target}' non trouvé par OCR") logger.debug(f"[Grounding/VLM] Target VLM '{vlm_target}' non trouvé par OCR")
return None return None

View File

@@ -0,0 +1,41 @@
# tests/integration/test_grounding_offset.py
"""Tests intégration pour la propagation d'offset multi-écrans (QW1)."""
import pytest
from unittest.mock import patch, MagicMock
from core.execution import input_handler
@pytest.fixture
def mock_screen():
"""Mock une capture mss : retourne un PIL Image factice + offsets."""
from PIL import Image
img = Image.new("RGB", (1920, 1080), color="white")
return img
def test_capture_screen_default_returns_composite_when_no_idx(mock_screen):
"""_capture_screen() sans monitor_idx → composite, offset (0, 0)."""
with patch("core.execution.input_handler.mss") as mock_mss:
ctx = mock_mss.mss.return_value.__enter__.return_value
ctx.monitors = [{"left": 0, "top": 0, "width": 3840, "height": 1080}]
ctx.grab.return_value = MagicMock(size=(3840, 1080), bgra=b"\x00" * (3840 * 1080 * 4))
with patch("core.execution.input_handler.PILImage.frombytes", return_value=mock_screen):
screen, w, h, ox, oy = input_handler._capture_screen()
assert (w, h, ox, oy) == (3840, 1080, 0, 0)
def test_capture_screen_targets_specific_monitor_with_offset(mock_screen):
"""_capture_screen(monitor_idx=1) → cible monitors[2] (mss skip [0]), offset = monitor.left."""
with patch("core.execution.input_handler.mss") as mock_mss:
ctx = mock_mss.mss.return_value.__enter__.return_value
# mss layout : [0]=composite, [1]=primary, [2]=secondary
ctx.monitors = [
{"left": 0, "top": 0, "width": 3840, "height": 1080},
{"left": 0, "top": 0, "width": 1920, "height": 1080},
{"left": 1920, "top": 0, "width": 1920, "height": 1080},
]
ctx.grab.return_value = MagicMock(size=(1920, 1080), bgra=b"\x00" * (1920 * 1080 * 4))
with patch("core.execution.input_handler.PILImage.frombytes", return_value=mock_screen):
screen, w, h, ox, oy = input_handler._capture_screen(monitor_idx=1)
assert (w, h, ox, oy) == (1920, 1080, 1920, 0)

View File

@@ -0,0 +1,61 @@
# tests/integration/test_loop_detector_replay.py
"""Tests intégration : un replay simulé qui boucle bascule en paused_need_help."""
import pytest
from unittest.mock import MagicMock
from agent_v0.server_v1.loop_detector import LoopDetector
def test_replay_state_transitions_to_paused_on_screen_static():
"""Cas : 4 screenshots identiques → replay passe à paused_need_help."""
embedder = MagicMock()
embedder.embed_image.return_value = [1.0, 0.0, 0.0] # constant
detector = LoopDetector(clip_embedder=embedder)
state = {
"replay_id": "r_test",
"status": "running",
"retried_actions": 0,
"_screenshot_history": ["img1", "img2", "img3", "img4"], # 4 images factices
"_action_history": [
{"type": "click", "x_pct": 0.1, "y_pct": 0.1},
{"type": "type", "x_pct": 0.2, "y_pct": 0.2},
],
}
verdict = detector.evaluate(state, state["_screenshot_history"], state["_action_history"])
# Simuler ce que ferait api_stream après verdict
if verdict.detected:
state["status"] = "paused_need_help"
state["pause_reason"] = verdict.reason
state["pause_message"] = f"signal={verdict.signal}"
assert state["status"] == "paused_need_help"
assert state["pause_reason"] == "loop_detected"
assert "screen_static" in state["pause_message"]
def test_replay_state_transitions_on_action_repeat():
"""Cas : 3 actions identiques → paused_need_help signal action_repeat."""
detector = LoopDetector(clip_embedder=None)
actions = [{"type": "click", "x_pct": 0.5, "y_pct": 0.5}] * 3
state = {"replay_id": "r2", "status": "running", "retried_actions": 0,
"_screenshot_history": [], "_action_history": actions}
verdict = detector.evaluate(state, [], actions)
assert verdict.detected and verdict.signal == "action_repeat"
def test_kill_switch_keeps_replay_running(monkeypatch):
"""Avec RPA_LOOP_DETECTOR_ENABLED=0 le replay continue même en boucle."""
monkeypatch.setenv("RPA_LOOP_DETECTOR_ENABLED", "0")
embedder = MagicMock()
embedder.embed_image.return_value = [1.0, 0.0, 0.0]
detector = LoopDetector(clip_embedder=embedder)
state = {"retried_actions": 10,
"_screenshot_history": ["img1"] * 10,
"_action_history": [{"type": "click", "x_pct": 0.5, "y_pct": 0.5}] * 10}
verdict = detector.evaluate(state, state["_screenshot_history"], state["_action_history"])
assert verdict.detected is False

View File

@@ -0,0 +1,96 @@
# tests/unit/test_loop_detector.py
"""Tests unitaires pour LoopDetector composite (QW2)."""
import os
import pytest
from unittest.mock import MagicMock
from agent_v0.server_v1.loop_detector import LoopDetector, LoopVerdict
@pytest.fixture
def detector():
"""LoopDetector avec embedder mocké (signal A toujours dispo)."""
embedder = MagicMock()
# Par défaut : 4 embeddings tous identiques → similarity 1.0
embedder.embed_image.return_value = [1.0, 0.0, 0.0]
return LoopDetector(clip_embedder=embedder)
def _state(retried=0, n_screenshots=0, n_actions=0):
return {
"retried_actions": retried,
"_screenshot_history": [[1.0, 0.0, 0.0]] * n_screenshots,
"_action_history": [{"type": "click", "x_pct": 0.5, "y_pct": 0.5}] * n_actions,
}
def test_screen_static_triggers_when_n_identical_embeddings(detector):
"""Signal A : 4 captures identiques (similarity > 0.99) → detected."""
state = _state(n_screenshots=4)
verdict = detector.evaluate(state, screenshots=state["_screenshot_history"], actions=[])
assert verdict.detected is True
assert verdict.signal == "screen_static"
def test_screen_static_skipped_when_history_too_short(detector):
"""Signal A : moins de N captures → pas de détection."""
state = _state(n_screenshots=2)
verdict = detector.evaluate(state, screenshots=state["_screenshot_history"], actions=[])
# Si seul A pourrait déclencher mais skip, et B/C pas remplis : detected=False
assert verdict.detected is False
def test_action_repeat_triggers_when_n_identical_actions(detector):
"""Signal B : 3 actions consécutives identiques → detected."""
state = _state(n_actions=3)
verdict = detector.evaluate(state, screenshots=[], actions=state["_action_history"])
assert verdict.detected is True
assert verdict.signal == "action_repeat"
def test_action_repeat_skipped_when_actions_differ(detector):
"""Signal B : actions différentes → pas de détection."""
actions = [
{"type": "click", "x_pct": 0.1, "y_pct": 0.1},
{"type": "click", "x_pct": 0.2, "y_pct": 0.2},
{"type": "click", "x_pct": 0.3, "y_pct": 0.3},
]
verdict = detector.evaluate(_state(), screenshots=[], actions=actions)
assert verdict.detected is False
def test_retry_threshold_triggers_at_3(detector):
"""Signal C : retried_actions >= 3 → detected."""
state = _state(retried=3)
verdict = detector.evaluate(state, screenshots=[], actions=[])
assert verdict.detected is True
assert verdict.signal == "retry_threshold"
def test_kill_switch_disables_all_signals(monkeypatch, detector):
"""Si RPA_LOOP_DETECTOR_ENABLED=0 → toujours detected=False."""
monkeypatch.setenv("RPA_LOOP_DETECTOR_ENABLED", "0")
state = _state(retried=10, n_screenshots=10, n_actions=10)
verdict = detector.evaluate(state, screenshots=state["_screenshot_history"],
actions=state["_action_history"])
assert verdict.detected is False
def test_embedder_unavailable_skips_signal_A_continues_others():
"""Si CLIP embedder None → signal A skip, B et C continuent."""
detector = LoopDetector(clip_embedder=None)
# Trigger signal C
state = _state(retried=3)
verdict = detector.evaluate(state, screenshots=[], actions=[])
assert verdict.detected is True
assert verdict.signal == "retry_threshold"
def test_embedder_exception_does_not_crash(detector):
"""Si embed_image lève une exception → log + verdict detected=False."""
detector.clip_embedder.embed_image.side_effect = RuntimeError("CUDA OOM")
state = _state(n_screenshots=4)
# Ne doit PAS lever : signal A devient inerte
verdict = detector.evaluate(state, screenshots=state["_screenshot_history"], actions=[])
# Signal A inerte, B/C pas remplis → detected False
assert verdict.detected is False

View File

@@ -0,0 +1,51 @@
# tests/unit/test_monitor_router.py
"""Tests unitaires pour MonitorRouter (QW1)."""
import pytest
from agent_v0.server_v1.monitor_router import resolve_target_monitor, MonitorTarget
# Geometry de référence pour les 3 tests : 2 écrans côte à côte
TWO_MONITORS = [
{"idx": 0, "x": 0, "y": 0, "w": 1920, "h": 1080, "primary": True},
{"idx": 1, "x": 1920, "y": 0, "w": 1920, "h": 1080, "primary": False},
]
def test_resolve_uses_action_monitor_index_when_present():
"""Si action.monitor_index présent et valide → cible cet écran."""
action = {"monitor_index": 1}
session_state = {"monitors_geometry": TWO_MONITORS, "last_focused_monitor": 0}
result = resolve_target_monitor(action, session_state)
assert result.idx == 1
assert result.offset_x == 1920
assert result.offset_y == 0
assert result.source == "action"
def test_resolve_falls_back_to_focused_monitor_when_action_missing():
"""Si action.monitor_index absent → fallback focus actif."""
action = {} # pas de monitor_index
session_state = {"monitors_geometry": TWO_MONITORS, "last_focused_monitor": 1}
result = resolve_target_monitor(action, session_state)
assert result.idx == 1
assert result.source == "focus"
def test_resolve_falls_back_to_composite_when_geometry_empty():
"""Si geometry vide (vieux Agent V1) → fallback composite (idx=-1, offset=0)."""
action = {}
session_state = {"monitors_geometry": [], "last_focused_monitor": None}
result = resolve_target_monitor(action, session_state)
assert result.source == "composite_fallback"
assert result.offset_x == 0
assert result.offset_y == 0
def test_resolve_falls_back_when_action_index_out_of_range():
"""Si action.monitor_index hors limites (écran débranché) → fallback focus."""
action = {"monitor_index": 5} # n'existe pas
session_state = {"monitors_geometry": TWO_MONITORS, "last_focused_monitor": 0}
result = resolve_target_monitor(action, session_state)
assert result.idx == 0
assert result.source == "focus"