Dom/rpa_vision_v3
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: vérification croisée CLIP+OCR + description ancre avant exécution
Some checks failed
security-audit / Bandit (scan statique) (push) Successful in 12s
Details
security-audit / pip-audit (CVE dépendances) (push) Successful in 10s
Details
security-audit / Scan secrets (grep) (push) Successful in 9s
Details
tests / Lint (ruff + black) (push) Successful in 14s
Details
tests / Tests unitaires (sans GPU) (push) Failing after 13s
Details
tests / Tests sécurité (critique) (push) Has been skipped
Details

Browse Source 
Quand CLIP dit "trouvé", on vérifie par OCR que le texte à cette
position correspond au target. Si CLIP clique sur "Ce PC" au lieu
de "CR_patient_demo", l'OCR le rejette → fallback sur la cascade.

Description VLM de l'ancre AVANT le CLIP quand le label est un
type d'action (double_click_anchor → "text file icon CR_patient").
Le target_text enrichi sert à la vérification croisée ET au grounding.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Dom
2026-04-21 11:10:01 +02:00
parent c50adab3a1
commit 7355d315a3
2 changed files with 46 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files

42
visual_workflow_builder/backend/services/intelligent_executor.py
View File

@@ -724,15 +724,39 @@ def find_and_click(
# clip_result.found est conditionné par les seuils dans find_anchor_in_screen
if clip_result.found:
print(f"✅ [Vision] UI-DETR-1+CLIP réussi! Confiance: {clip_result.confidence:.2f}")
return {
'found': True,
'confidence': clip_result.confidence,
'coordinates': clip_result.center,
'bbox': clip_result.bbox,
'method': 'clip',
'search_time_ms': (_time.time() - start_time) * 1000
}
# Vérification croisée OCR : le texte à cette position correspond-il ?
clip_validated = True
if target_text and target_text not in ('click_anchor', 'double_click_anchor',
'right_click_anchor', 'hover_anchor', 'focus_anchor'):
try:
from services.ocr_service import ocr_extract_words
words = ocr_extract_words(screen_image)
cx, cy = clip_result.center['x'], clip_result.center['y']
nearby_texts = []
for w in words:
wx = (w['bbox'][0] + w['bbox'][2]) / 2
wy = (w['bbox'][1] + w['bbox'][3]) / 2
dist = ((wx - cx)**2 + (wy - cy)**2) ** 0.5
if dist < 100:
nearby_texts.append(w['text'])
nearby_str = ' '.join(nearby_texts).lower()
target_lower = target_text.lower()
if target_lower not in nearby_str and not any(t.lower() in target_lower for t in nearby_texts if len(t) > 2):
print(f"⛔ [Vision] CLIP rejeté par OCR: texte proche='{nearby_str}' ne contient pas '{target_text}'")
clip_validated = False
except Exception as ocr_err:
print(f"⚠️ [Vision] Vérification OCR échouée: {ocr_err}")
if clip_validated:
print(f"✅ [Vision] UI-DETR-1+CLIP réussi! Confiance: {clip_result.confidence:.2f}")
return {
'found': True,
'confidence': clip_result.confidence,
'coordinates': clip_result.center,
'bbox': clip_result.bbox,
'method': 'clip',
'search_time_ms': (_time.time() - start_time) * 1000
}
else:
print(f"⚠️ [Vision] UI-DETR-1+CLIP: rejeté (confiance: {clip_result.confidence:.2f})")
except Exception as clip_err: