feat(knowledge): câblage UIPatternLibrary dans executor + stream processor
Some checks failed
security-audit / Bandit (scan statique) (push) Successful in 12s
security-audit / pip-audit (CVE dépendances) (push) Successful in 12s
security-audit / Scan secrets (grep) (push) Successful in 9s
tests / Lint (ruff + black) (push) Successful in 15s
tests / Tests unitaires (sans GPU) (push) Failing after 14s
tests / Tests sécurité (critique) (push) Has been skipped
Some checks failed
security-audit / Bandit (scan statique) (push) Successful in 12s
security-audit / pip-audit (CVE dépendances) (push) Successful in 12s
security-audit / Scan secrets (grep) (push) Successful in 9s
tests / Lint (ruff + black) (push) Successful in 15s
tests / Tests unitaires (sans GPU) (push) Failing after 14s
tests / Tests sécurité (critique) (push) Has been skipped
VWB Executor : - _check_screen_for_patterns() : capture écran + OCR + pattern matching - _handle_detected_pattern() : clic automatique sur dialogues connus - Vérifie entre chaque étape en mode intelligent/debug - Si un dialogue bloque (OK, Save, Cancel), Léa le gère seule Stream Processor : - Enrichit les ScreenState avec ui_pattern/ui_pattern_action/ui_pattern_target - Les patterns détectés sont loggés et stockés dans les résultats - Permet au GraphBuilder de savoir quels écrans sont des dialogues Phase 2 du plan "connaissance native de l'environnement". Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -2042,6 +2042,24 @@ class StreamProcessor:
|
||||
self._screen_states[session_id] = []
|
||||
self._screen_states[session_id].append(screen_state)
|
||||
|
||||
# Enrichir avec les patterns UI connus
|
||||
try:
|
||||
from core.knowledge.ui_patterns import UIPatternLibrary
|
||||
detected_text = getattr(screen_state.perception, "detected_text", [])
|
||||
if detected_text:
|
||||
ocr_text = " ".join(str(t) for t in detected_text) if isinstance(detected_text, list) else str(detected_text)
|
||||
lib = UIPatternLibrary()
|
||||
pattern = lib.find_pattern(ocr_text)
|
||||
if pattern:
|
||||
result["ui_pattern"] = pattern["pattern"]
|
||||
result["ui_pattern_action"] = pattern["action"]
|
||||
result["ui_pattern_target"] = pattern["target"]
|
||||
logger.info(f"Pattern UI détecté: {pattern['pattern']} → {pattern['target']}")
|
||||
except ImportError:
|
||||
pass
|
||||
except Exception as e:
|
||||
logger.debug(f"Pattern check: {e}")
|
||||
|
||||
logger.info(
|
||||
f"Screenshot analysé: {shot_id} | "
|
||||
f"{result['ui_elements_count']} UI elements, "
|
||||
|
||||
Reference in New Issue
Block a user