feat: replay visuel VLM-first, worker séparé, package Léa, AZERTY, sécurité HTTPS
Pipeline replay visuel : - VLM-first : l'agent appelle Ollama directement pour trouver les éléments - Template matching en fallback (seuil strict 0.90) - Stop immédiat si élément non trouvé (pas de clic blind) - Replay depuis session brute (/replay-session) sans attendre le VLM - Vérification post-action (screenshot hash avant/après) - Gestion des popups (Enter/Escape/Tab+Enter) Worker VLM séparé : - run_worker.py : process distinct du serveur HTTP - Communication par fichiers (_worker_queue.txt + _replay_active.lock) - Le serveur HTTP ne fait plus jamais de VLM → toujours réactif - Service systemd rpa-worker.service Capture clavier : - raw_keys (vk + press/release) pour replay exact indépendant du layout - Fix AZERTY : ToUnicodeEx + AltGr detection - Enter capturé comme \n, Tab comme \t - Filtrage modificateurs seuls (Ctrl/Alt/Shift parasites) - Fusion text_input consécutifs, dédup key_combo Sécurité & Internet : - HTTPS Let's Encrypt (lea.labs + vwb.labs.laurinebazin.design) - Token API fixe dans .env.local - HTTP Basic Auth sur VWB - Security headers (HSTS, CSP, nosniff) - CORS domaines publics, plus de wildcard Infrastructure : - DPI awareness (SetProcessDpiAwareness) Python + Rust - Métadonnées système (dpi_scale, window_bounds, monitors, os_theme) - Template matching multi-scale [0.5, 2.0] - Résolution dynamique (plus de hardcode 1920x1080) - VLM prefill fix (47x speedup, 3.5s au lieu de 180s) Modules : - core/auth/ : credential vault (Fernet AES), TOTP (RFC 6238), auth handler - core/federation/ : LearningPack export/import anonymisé, FAISS global - deploy/ : package Léa (config.txt, Lea.bat, install.bat, LISEZMOI.txt) UX : - Filtrage OS (VWB + Chat montrent que les workflows de l'OS courant) - Bibliothèque persistante (cache local + SQLite) - Clustering hybride (titre fenêtre + DBSCAN) - EdgeConstraints + PostConditions peuplés - GraphBuilder compound actions (toutes les frappes) Agent Rust : - Token Bearer auth (network.rs) - sysinfo.rs (DPI, résolution, window bounds via Win32 API) - config.txt lu automatiquement - Support Chrome/Brave/Firefox (pas que Edge) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -0,0 +1,95 @@
|
||||
{
|
||||
"workflow_id": "wf_372eabf626f8_1773752892",
|
||||
"workflow_name": "test_windows",
|
||||
"description": "",
|
||||
"tags": [],
|
||||
"steps": [
|
||||
{
|
||||
"order": 0,
|
||||
"action_type": "click_anchor",
|
||||
"label": "click_anchor",
|
||||
"parameters": {
|
||||
"visual_anchor": {
|
||||
"anchor_id": "anchor_e54628b75227_1773829452",
|
||||
"bounding_box": {
|
||||
"x": 560,
|
||||
"y": 1536,
|
||||
"width": 230,
|
||||
"height": 59
|
||||
}
|
||||
}
|
||||
},
|
||||
"has_anchor": true
|
||||
},
|
||||
{
|
||||
"order": 0,
|
||||
"action_type": "type_text",
|
||||
"label": "type_text",
|
||||
"parameters": {
|
||||
"clear_before": true,
|
||||
"text": "bloc note"
|
||||
},
|
||||
"has_anchor": false
|
||||
},
|
||||
{
|
||||
"order": 0,
|
||||
"action_type": "click_anchor",
|
||||
"label": "click_anchor",
|
||||
"parameters": {
|
||||
"visual_anchor": {
|
||||
"anchor_id": "anchor_dd32b09238ca_1773833606",
|
||||
"bounding_box": {
|
||||
"x": 670,
|
||||
"y": 310,
|
||||
"width": 51,
|
||||
"height": 51
|
||||
}
|
||||
}
|
||||
},
|
||||
"has_anchor": true
|
||||
},
|
||||
{
|
||||
"order": 0,
|
||||
"action_type": "wait_for_anchor",
|
||||
"label": "wait_for_anchor",
|
||||
"parameters": {
|
||||
"visual_anchor": {
|
||||
"anchor_id": "anchor_2d51b88900d7_1773833805",
|
||||
"bounding_box": {
|
||||
"x": 287,
|
||||
"y": 290,
|
||||
"width": 70,
|
||||
"height": 92
|
||||
}
|
||||
}
|
||||
},
|
||||
"has_anchor": true
|
||||
},
|
||||
{
|
||||
"order": 0,
|
||||
"action_type": "type_text",
|
||||
"label": "type_text",
|
||||
"parameters": {
|
||||
"text": "hello"
|
||||
},
|
||||
"has_anchor": false
|
||||
}
|
||||
],
|
||||
"exported_at": "2026-03-18T11:38:45.271209",
|
||||
"metadata": {
|
||||
"step_count": 5,
|
||||
"action_types": [
|
||||
"wait_for_anchor",
|
||||
"type_text",
|
||||
"click_anchor"
|
||||
],
|
||||
"has_anchors": true,
|
||||
"warnings": [
|
||||
"Étape 1 (click_anchor): pas de label personnalisé",
|
||||
"Étape 2 (type_text): pas de label personnalisé",
|
||||
"Étape 3 (click_anchor): pas de label personnalisé",
|
||||
"Étape 4 (wait_for_anchor): pas de label personnalisé",
|
||||
"Étape 5 (type_text): pas de label personnalisé"
|
||||
]
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user