feat(knowledge): câblage UIPatternLibrary dans executor + stream processor
Some checks failed
security-audit / Bandit (scan statique) (push) Successful in 12s
security-audit / pip-audit (CVE dépendances) (push) Successful in 12s
security-audit / Scan secrets (grep) (push) Successful in 9s
tests / Lint (ruff + black) (push) Successful in 15s
tests / Tests unitaires (sans GPU) (push) Failing after 14s
tests / Tests sécurité (critique) (push) Has been skipped

VWB Executor :
- _check_screen_for_patterns() : capture écran + OCR + pattern matching
- _handle_detected_pattern() : clic automatique sur dialogues connus
- Vérifie entre chaque étape en mode intelligent/debug
- Si un dialogue bloque (OK, Save, Cancel), Léa le gère seule

Stream Processor :
- Enrichit les ScreenState avec ui_pattern/ui_pattern_action/ui_pattern_target
- Les patterns détectés sont loggés et stockés dans les résultats
- Permet au GraphBuilder de savoir quels écrans sont des dialogues

Phase 2 du plan "connaissance native de l'environnement".

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Dom
2026-04-19 10:54:19 +02:00
parent af4eae28b9
commit 23a06a744c
2 changed files with 100 additions and 0 deletions

View File

@@ -2042,6 +2042,24 @@ class StreamProcessor:
self._screen_states[session_id] = []
self._screen_states[session_id].append(screen_state)
# Enrichir avec les patterns UI connus
try:
from core.knowledge.ui_patterns import UIPatternLibrary
detected_text = getattr(screen_state.perception, "detected_text", [])
if detected_text:
ocr_text = " ".join(str(t) for t in detected_text) if isinstance(detected_text, list) else str(detected_text)
lib = UIPatternLibrary()
pattern = lib.find_pattern(ocr_text)
if pattern:
result["ui_pattern"] = pattern["pattern"]
result["ui_pattern_action"] = pattern["action"]
result["ui_pattern_target"] = pattern["target"]
logger.info(f"Pattern UI détecté: {pattern['pattern']}{pattern['target']}")
except ImportError:
pass
except Exception as e:
logger.debug(f"Pattern check: {e}")
logger.info(
f"Screenshot analysé: {shot_id} | "
f"{result['ui_elements_count']} UI elements, "