Initial commit
This commit is contained in:
Dom
962
docs/reference/ARCHITECTURE_ENRICHISSEMENTS.md
Normal file
962
docs/reference/ARCHITECTURE_ENRICHISSEMENTS.md
Normal file
@@ -0,0 +1,962 @@
|
||||
# Enrichissements Architecture - Concepts Avancés
|
||||
|
||||
**Date** : 22 novembre 2024
|
||||
**Version** : 1.1
|
||||
**Statut** : 🔧 Raffinements et Angles Morts
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Objectif
|
||||
|
||||
Ce document enrichit [ARCHITECTURE_VISION_COMPLETE.md](./ARCHITECTURE_VISION_COMPLETE.md) avec des concepts avancés qui transforment l'architecture d'une vision théorique en **contrat stable** prêt pour l'implémentation.
|
||||
|
||||
### Concepts Ajoutés
|
||||
|
||||
1. **Grammaire du temps** - Découpage en épisodes de workflow
|
||||
2. **Identités stables** - Tracking d'éléments dans le temps
|
||||
3. **Versioning d'espaces** - Gestion des migrations de modèles
|
||||
4. **Variables métier** - Paramétrage des workflows
|
||||
5. **Gestion d'erreurs** - Nœuds et chemins d'erreur explicites
|
||||
6. **Matrice Mode × Maturité** - Clarification des comportements
|
||||
7. **Layout vs Contenu** - Robustesse aux variations
|
||||
8. **Actionnabilité** - Scores de cliquabilité
|
||||
|
||||
---
|
||||
|
||||
## 📊 Couche 0 : RawSession - Grammaire du Temps
|
||||
|
||||
### Problème Identifié
|
||||
|
||||
Une RawSession est actuellement un "long film" sans chapitres. Pour apprendre des workflows, nous avons besoin de **découper en épisodes**.
|
||||
|
||||
### Solution : Workflow Episodes
|
||||
|
||||
**Ajout au format RawSession** :
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": "rawsession_v1",
|
||||
"session_id": "sess_2025-11-21T10-15-00_formateurX",
|
||||
|
||||
"segments": [
|
||||
{
|
||||
"segment_id": "seg_001",
|
||||
"start_event_index": 0,
|
||||
"end_event_index": 32,
|
||||
"start_time": "2025-11-21T10:15:00.523Z",
|
||||
"end_time": "2025-11-21T10:18:45.123Z",
|
||||
"label": "workflow_candidate",
|
||||
"workflow_id": "WF_validation_facture",
|
||||
"confidence": 0.87,
|
||||
"trigger": "window_change"
|
||||
},
|
||||
{
|
||||
"segment_id": "seg_002",
|
||||
"start_event_index": 33,
|
||||
"end_event_index": 67,
|
||||
"start_time": "2025-11-21T10:19:00.000Z",
|
||||
"end_time": "2025-11-21T10:22:30.456Z",
|
||||
"label": "workflow_candidate",
|
||||
"workflow_id": "WF_validation_facture",
|
||||
"confidence": 0.92,
|
||||
"trigger": "long_pause"
|
||||
}
|
||||
],
|
||||
|
||||
"events": [ /* ... */ ]
|
||||
}
|
||||
```
|
||||
|
||||
**Critères de découpage** :
|
||||
- Grande pause (>30s sans action)
|
||||
- Changement d'application
|
||||
- Retour au bureau
|
||||
- Détection de pattern complet
|
||||
|
||||
**Bénéfices** :
|
||||
- ✅ Comptage propre des "runs" d'un workflow
|
||||
- ✅ Comparaison d'épisodes entre eux
|
||||
- ✅ Détection de patterns en séquence
|
||||
- ✅ Métriques par épisode
|
||||
|
||||
### Marquage du Bruit
|
||||
|
||||
**Problème** : Sessions réelles contiennent du bruit (hésitations, clics ratés, alt-tab inutiles).
|
||||
|
||||
**Solution** : Classification d'événements
|
||||
|
||||
```json
|
||||
{
|
||||
"events": [
|
||||
{
|
||||
"t": 0.523,
|
||||
"type": "mouse_click",
|
||||
"button": "left",
|
||||
"pos": [1620, 920],
|
||||
"noise_flag": false,
|
||||
"semantic_role": "validation",
|
||||
"confidence": 0.94
|
||||
},
|
||||
{
|
||||
"t": 1.200,
|
||||
"type": "mouse_move",
|
||||
"pos": [800, 600],
|
||||
"noise_flag": true,
|
||||
"semantic_role": null,
|
||||
"confidence": 0.12
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Rôles sémantiques** :
|
||||
- `navigation` - Déplacement entre écrans
|
||||
- `validation` - Confirmation d'action
|
||||
- `search` - Recherche d'information
|
||||
- `input` - Saisie de données
|
||||
- `correction` - Annulation/correction
|
||||
- `null` - Bruit ou indéterminé
|
||||
|
||||
---
|
||||
|
||||
## 📊 Couche 1 : ScreenState - Structure vs Contenu
|
||||
|
||||
### Lien Explicite avec RawSession
|
||||
|
||||
**Problème** : Traçabilité entre ScreenState et événements source.
|
||||
|
||||
**Solution** : Référence explicite
|
||||
|
||||
```json
|
||||
{
|
||||
"screen_state_id": "screen_2025-11-21T10-15-32.123Z",
|
||||
|
||||
"source": {
|
||||
"session_id": "sess_2025-11-21T10-15-00_formateurX",
|
||||
"screenshot_id": "shot_0002",
|
||||
"segment_id": "seg_001",
|
||||
"event_ids": [120, 121],
|
||||
"trigger_event": {
|
||||
"event_id": 121,
|
||||
"type": "mouse_click",
|
||||
"role": "validate_button"
|
||||
}
|
||||
},
|
||||
|
||||
"window": { /* ... */ },
|
||||
"raw": { /* ... */ },
|
||||
"perception": { /* ... */ }
|
||||
}
|
||||
```
|
||||
|
||||
**Bénéfices** :
|
||||
- ✅ Traçabilité complète
|
||||
- ✅ Debugging facilité
|
||||
- ✅ Analyse de causalité
|
||||
|
||||
### Layout Signature
|
||||
|
||||
**Problème** : Robustesse aux variations de contenu (noms de patients, montants différents).
|
||||
|
||||
**Solution** : Séparation Layout vs Contenu
|
||||
|
||||
```json
|
||||
{
|
||||
"screen_state_id": "screen_2025-11-21T10-15-32.123Z",
|
||||
|
||||
"layout_signature": {
|
||||
"hash": "layout_abc123def456",
|
||||
"method": "structure_ui_v1",
|
||||
"components": {
|
||||
"ui_topology": "12_elements_3_buttons_1_table",
|
||||
"spatial_layout": "grid_3x4",
|
||||
"element_types": ["button", "button", "button", "table", "input"]
|
||||
},
|
||||
"stability_score": 0.96
|
||||
},
|
||||
|
||||
"content_signature": {
|
||||
"hash": "content_xyz789abc012",
|
||||
"variable_fields": ["patient_name", "invoice_amount", "date"],
|
||||
"static_fields": ["window_title", "button_labels"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Utilisation** :
|
||||
- **Matching de nodes** : Utiliser `layout_signature.hash`
|
||||
- **Regroupement** : Écrans de même structure
|
||||
- **Variations** : Gérer changements de contenu
|
||||
|
||||
### Vue Debug Human-Friendly
|
||||
|
||||
**Solution** : Résumé généré par VLM
|
||||
|
||||
```json
|
||||
{
|
||||
"debug": {
|
||||
"summary": "Liste de 34 factures avec colonnes Patient, Date, Montant, Statut. 1 bouton 'Valider' en bas à droite, 1 champ de recherche en haut.",
|
||||
"key_elements": [
|
||||
"34 lignes de factures",
|
||||
"12 factures 'À valider'",
|
||||
"22 factures 'Validée'",
|
||||
"Bouton 'Valider' actif"
|
||||
],
|
||||
"anomalies": [],
|
||||
"generated_by": "qwen_vl",
|
||||
"generation_time_ms": 234
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 Couche 2 : UIElement - Identité Stable
|
||||
|
||||
### Identité Stable dans le Temps
|
||||
|
||||
**Problème** : Suivre un élément à travers différents ScreenStates.
|
||||
|
||||
**Solution** : Dual ID (local + stable)
|
||||
|
||||
```json
|
||||
{
|
||||
"element_id": "el_btn_valider_001",
|
||||
"stable_id": "btn_valider_facture_v1",
|
||||
"stability_confidence": 0.87,
|
||||
|
||||
"stability_tracking": {
|
||||
"first_seen": "2025-11-21T10:15:00Z",
|
||||
"last_seen": "2025-11-21T10:30:00Z",
|
||||
"appearance_count": 15,
|
||||
"position_variance": 12.5,
|
||||
"label_variants": ["Valider", "Valider la facture"]
|
||||
},
|
||||
|
||||
"type": "button",
|
||||
"role": "validate_invoice",
|
||||
"bbox": [1500, 900, 1600, 940]
|
||||
}
|
||||
```
|
||||
|
||||
**Calcul de `stable_id`** :
|
||||
```python
|
||||
stable_id = hash(
|
||||
app_name +
|
||||
window_title_normalized +
|
||||
role +
|
||||
label_normalized +
|
||||
bbox_approximate # arrondi à 50px
|
||||
)
|
||||
```
|
||||
|
||||
**Bénéfices** :
|
||||
- ✅ Tracking d'éléments dans workflows complexes
|
||||
- ✅ Détection de disparition d'éléments
|
||||
- ✅ Support des boucles et répétitions
|
||||
|
||||
### Score d'Actionnabilité
|
||||
|
||||
**Problème** : Choisir le bon élément parmi plusieurs candidats.
|
||||
|
||||
**Solution** : Scores de cliquabilité
|
||||
|
||||
```json
|
||||
{
|
||||
"element_id": "el_btn_valider_001",
|
||||
"type": "button",
|
||||
"role": "validate_invoice",
|
||||
|
||||
"action_affordance": {
|
||||
"clickable_score": 0.95,
|
||||
"primary_action_score": 0.88,
|
||||
"dangerous_action_score": 0.02,
|
||||
|
||||
"factors": {
|
||||
"visual_prominence": 0.92,
|
||||
"position_importance": 0.85,
|
||||
"label_clarity": 0.96,
|
||||
"color_significance": 0.89
|
||||
}
|
||||
},
|
||||
|
||||
"interaction_history": {
|
||||
"click_count": 45,
|
||||
"success_rate": 0.98,
|
||||
"avg_dwell_time_ms": 1200
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Utilisation dans Edge** :
|
||||
```json
|
||||
{
|
||||
"action": {
|
||||
"target": {
|
||||
"role": "validate_button",
|
||||
"min_affordance_score": 0.80
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 Couche 3 : State Embedding - Versioning d'Espaces
|
||||
|
||||
### Identité de l'Espace d'Embedding
|
||||
|
||||
**Problème** : Migration de modèles invalide les seuils de similarité.
|
||||
|
||||
**Solution** : Versioning explicite
|
||||
|
||||
```json
|
||||
{
|
||||
"state_embedding": {
|
||||
"embedding_id": "state_emb_2025-11-21T10-15-32.123Z",
|
||||
"vector_id": "data/embeddings/states/state_2025-11-21T10-15-32.123Z.npy",
|
||||
|
||||
"space": {
|
||||
"id": "clip_ViT-B-32_openclip_v1",
|
||||
"version": "2025-11-01",
|
||||
"model_name": "ViT-B-32",
|
||||
"provider": "openclip",
|
||||
"dimensions": 512,
|
||||
"metric": "cosine",
|
||||
"normalization": "l2"
|
||||
},
|
||||
|
||||
"fusion_method": "weighted",
|
||||
"components": { /* ... */ }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Contrat** :
|
||||
- Un WorkflowGraph référence un `embedding_space.id`
|
||||
- Un index FAISS est attaché à ce même `space.id`
|
||||
- Cohabitation de plusieurs espaces possible
|
||||
|
||||
**Migration de modèle** :
|
||||
```json
|
||||
{
|
||||
"workflow_id": "WF_validation_facture",
|
||||
|
||||
"embedding_spaces": {
|
||||
"current": "clip_ViT-B-32_openclip_v1",
|
||||
"legacy": ["clip_ViT-B-16_openclip_v0"],
|
||||
"migration_status": "in_progress",
|
||||
"migration_progress": 0.65
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Multi-Têtes d'Embeddings
|
||||
|
||||
**Solution** : Embeddings primaire + auxiliaire
|
||||
|
||||
```json
|
||||
{
|
||||
"state_embedding": {
|
||||
"primary": {
|
||||
"space": {
|
||||
"id": "clip_ViT-B-32_fast",
|
||||
"purpose": "runtime_matching"
|
||||
},
|
||||
"vector_id": "data/embeddings/states/state_..._primary.npy",
|
||||
"dimensions": 512,
|
||||
"computation_time_ms": 45
|
||||
},
|
||||
|
||||
"auxiliary": {
|
||||
"space": {
|
||||
"id": "clip_ViT-L-14_rich",
|
||||
"purpose": "offline_analysis"
|
||||
},
|
||||
"vector_id": "data/embeddings/states/state_..._aux.npy",
|
||||
"dimensions": 768,
|
||||
"computation_time_ms": 180
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Usages** :
|
||||
- **Primary** : Matching en temps réel, détection de nodes
|
||||
- **Auxiliary** : Recherche sémantique, visualisation, clustering
|
||||
|
||||
---
|
||||
|
||||
|
||||
## 📊 Couche 4 : Workflow Graph - Variables & Erreurs
|
||||
|
||||
### Variables de Workflow (Slots)
|
||||
|
||||
**Problème** : Workflows réels nécessitent des paramètres (patient_name, invoice_id, etc.).
|
||||
|
||||
**Solution** : Bloc variables explicite
|
||||
|
||||
```json
|
||||
{
|
||||
"workflow_id": "WF_validation_facture",
|
||||
"name": "Validation facture consultation",
|
||||
|
||||
"variables": {
|
||||
"invoice_id": {
|
||||
"type": "string",
|
||||
"source": {
|
||||
"method": "ui_element_extraction",
|
||||
"element_role": "invoice_row",
|
||||
"extraction_pattern": "FAC-\\d{4}-\\d{5}"
|
||||
},
|
||||
"example_values": ["FAC-2025-00123", "FAC-2025-00124"],
|
||||
"required": true,
|
||||
"validation": {
|
||||
"regex": "^FAC-\\d{4}-\\d{5}$"
|
||||
}
|
||||
},
|
||||
|
||||
"patient_name": {
|
||||
"type": "string",
|
||||
"source": {
|
||||
"method": "ui_element_extraction",
|
||||
"element_role": "patient_name_field"
|
||||
},
|
||||
"example_values": ["DUPONT Jean", "MARTIN Marie"],
|
||||
"required": true
|
||||
},
|
||||
|
||||
"amount": {
|
||||
"type": "float",
|
||||
"source": {
|
||||
"method": "ui_element_extraction",
|
||||
"element_role": "invoice_amount",
|
||||
"extraction_pattern": "\\d+[,.]\\d{2}"
|
||||
},
|
||||
"example_values": [120.00, 250.50],
|
||||
"required": true,
|
||||
"validation": {
|
||||
"min": 0.0,
|
||||
"max": 10000.0
|
||||
}
|
||||
},
|
||||
|
||||
"validation_date": {
|
||||
"type": "datetime",
|
||||
"source": {
|
||||
"method": "system_time"
|
||||
},
|
||||
"required": false
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Utilisation dans Edges** :
|
||||
|
||||
```json
|
||||
{
|
||||
"edge_id": "E2_valider_si_montant_ok",
|
||||
"from_node": "N2_detail_facture",
|
||||
"to_node": "N3_controle_facture",
|
||||
|
||||
"condition": {
|
||||
"type": "expression",
|
||||
"expression": "variables.amount < 1000.0",
|
||||
"on_true": "N3_controle_facture",
|
||||
"on_false": "N_manual_review"
|
||||
},
|
||||
|
||||
"action": {
|
||||
"type": "mouse_click",
|
||||
"target": {
|
||||
"role": "validate_button"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Boucles avec variables** :
|
||||
|
||||
```json
|
||||
{
|
||||
"edge_id": "E_loop_factures",
|
||||
"from_node": "N1_liste_factures",
|
||||
"to_node": "N2_detail_facture",
|
||||
|
||||
"loop": {
|
||||
"type": "for_each",
|
||||
"iterator": "invoice_row",
|
||||
"filter": "status == 'À valider'",
|
||||
"max_iterations": 50,
|
||||
"variable_binding": {
|
||||
"invoice_id": "invoice_row.id",
|
||||
"patient_name": "invoice_row.patient"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Nœuds d'Erreur Explicites
|
||||
|
||||
**Problème** : Graphe ne représente pas les chemins d'erreur.
|
||||
|
||||
**Solution** : Nodes d'erreur + Edges de fallback
|
||||
|
||||
```json
|
||||
{
|
||||
"nodes": [
|
||||
{
|
||||
"node_id": "N1_liste_factures",
|
||||
"label": "Liste des factures",
|
||||
"is_error_node": false,
|
||||
"screen_template": { /* ... */ }
|
||||
},
|
||||
|
||||
{
|
||||
"node_id": "N_error_generic",
|
||||
"label": "Erreur générique",
|
||||
"is_error_node": true,
|
||||
"error_type": "generic",
|
||||
"screen_template": {
|
||||
"required_text_any": [
|
||||
"Erreur",
|
||||
"Impossible de",
|
||||
"Échec",
|
||||
"Error",
|
||||
"Failed"
|
||||
],
|
||||
"embedding_prototype": {
|
||||
"vector_id": "data/embeddings/workflows/errors/generic.npy",
|
||||
"min_cosine_similarity": 0.75
|
||||
}
|
||||
},
|
||||
"recovery_actions": [
|
||||
{
|
||||
"action": "click_ok_button",
|
||||
"return_to_node": "N1_liste_factures"
|
||||
},
|
||||
{
|
||||
"action": "close_dialog",
|
||||
"return_to_node": "N1_liste_factures"
|
||||
}
|
||||
]
|
||||
},
|
||||
|
||||
{
|
||||
"node_id": "N_error_session_expired",
|
||||
"label": "Session expirée",
|
||||
"is_error_node": true,
|
||||
"error_type": "authentication",
|
||||
"screen_template": {
|
||||
"required_text_any": [
|
||||
"Session expirée",
|
||||
"Reconnexion nécessaire",
|
||||
"Veuillez vous reconnecter"
|
||||
]
|
||||
},
|
||||
"recovery_actions": [
|
||||
{
|
||||
"action": "trigger_login_workflow",
|
||||
"workflow_id": "WF_login"
|
||||
}
|
||||
]
|
||||
},
|
||||
|
||||
{
|
||||
"node_id": "N_error_validation_failed",
|
||||
"label": "Échec de validation",
|
||||
"is_error_node": true,
|
||||
"error_type": "business_logic",
|
||||
"screen_template": {
|
||||
"required_text_any": [
|
||||
"Validation impossible",
|
||||
"Données manquantes",
|
||||
"Montant incorrect"
|
||||
]
|
||||
},
|
||||
"recovery_actions": [
|
||||
{
|
||||
"action": "return_to_previous",
|
||||
"return_to_node": "N2_detail_facture"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
|
||||
"edges": [
|
||||
{
|
||||
"edge_id": "E_error_fallback_any",
|
||||
"from_node": "ANY",
|
||||
"to_node": "N_error_generic",
|
||||
"priority": -1,
|
||||
"condition": {
|
||||
"type": "screen_similarity",
|
||||
"target_node": "N_error_generic",
|
||||
"min_similarity": 0.75
|
||||
},
|
||||
"trigger": "unexpected_screen"
|
||||
},
|
||||
|
||||
{
|
||||
"edge_id": "E_recovery_from_error",
|
||||
"from_node": "N_error_generic",
|
||||
"to_node": "N1_liste_factures",
|
||||
"action": {
|
||||
"type": "compound",
|
||||
"steps": [
|
||||
{
|
||||
"type": "mouse_click",
|
||||
"target": {"role": "ok_button"}
|
||||
},
|
||||
{
|
||||
"type": "wait",
|
||||
"duration_ms": 1000
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Types d'erreurs** :
|
||||
- `generic` - Erreur générique
|
||||
- `authentication` - Session expirée, login requis
|
||||
- `business_logic` - Validation métier échouée
|
||||
- `network` - Problème de connexion
|
||||
- `timeout` - Délai dépassé
|
||||
- `ui_not_found` - Élément UI introuvable
|
||||
|
||||
**Statistiques d'erreurs** :
|
||||
|
||||
```json
|
||||
{
|
||||
"stats": {
|
||||
"total_executions": 150,
|
||||
"success_count": 142,
|
||||
"error_count": 8,
|
||||
|
||||
"errors_by_type": {
|
||||
"generic": 3,
|
||||
"authentication": 2,
|
||||
"business_logic": 2,
|
||||
"timeout": 1
|
||||
},
|
||||
|
||||
"recovery_success_rate": 0.875,
|
||||
|
||||
"error_transitions": [
|
||||
{
|
||||
"from_node": "N2_detail_facture",
|
||||
"to_node": "N_error_validation_failed",
|
||||
"count": 2
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🔄 Matrice Mode × Maturité
|
||||
|
||||
### Clarification des Comportements
|
||||
|
||||
**Problème** : Confusion entre mode global et maturité du workflow.
|
||||
|
||||
**Solution** : Matrice explicite
|
||||
|
||||
| Mode Global | Learning State | Comportement |
|
||||
|-------------|----------------|--------------|
|
||||
| **Shadow** | OBSERVATION | Observer uniquement, jamais d'exécution |
|
||||
| **Shadow** | COACHING | Observer uniquement, jamais d'exécution |
|
||||
| **Shadow** | AUTO_CANDIDATE | Observer uniquement, jamais d'exécution |
|
||||
| **Shadow** | AUTO_CONFIRMÉ | Observer uniquement, jamais d'exécution |
|
||||
| **Assist** | OBSERVATION | Suggestion faible "J'apprends ce workflow" |
|
||||
| **Assist** | COACHING | Suggestions fortes, utilisateur exécute |
|
||||
| **Assist** | AUTO_CANDIDATE | Suggestions + proposition d'auto-exécution |
|
||||
| **Assist** | AUTO_CONFIRMÉ | Suggestions + auto-exécution disponible |
|
||||
| **Auto** | OBSERVATION | Pas d'auto-exécution (maturité insuffisante) |
|
||||
| **Auto** | COACHING | Pas d'auto-exécution (maturité insuffisante) |
|
||||
| **Auto** | AUTO_CANDIDATE | Propose auto-exécution avec confirmation |
|
||||
| **Auto** | AUTO_CONFIRMÉ | Exécution directe + rétrogradation si anomalie |
|
||||
|
||||
**Implémentation** :
|
||||
|
||||
```json
|
||||
{
|
||||
"workflow_id": "WF_validation_facture",
|
||||
|
||||
"learning": {
|
||||
"state": "AUTO_CANDIDATE",
|
||||
"progression": {
|
||||
"current_phase": "AUTO_CANDIDATE",
|
||||
"progress_percent": 75.0,
|
||||
"next_phase": "AUTO_CONFIRMÉ"
|
||||
}
|
||||
},
|
||||
|
||||
"execution_policy": {
|
||||
"shadow_mode": {
|
||||
"allowed_actions": ["observe", "log"],
|
||||
"user_interaction": "none"
|
||||
},
|
||||
|
||||
"assist_mode": {
|
||||
"OBSERVATION": {
|
||||
"allowed_actions": ["observe", "suggest_weak"],
|
||||
"user_interaction": "passive_notification"
|
||||
},
|
||||
"COACHING": {
|
||||
"allowed_actions": ["observe", "suggest_strong", "highlight_ui"],
|
||||
"user_interaction": "active_guidance"
|
||||
},
|
||||
"AUTO_CANDIDATE": {
|
||||
"allowed_actions": ["observe", "suggest_strong", "propose_auto"],
|
||||
"user_interaction": "confirmation_dialog"
|
||||
},
|
||||
"AUTO_CONFIRMÉ": {
|
||||
"allowed_actions": ["observe", "suggest_strong", "auto_execute"],
|
||||
"user_interaction": "notification_only"
|
||||
}
|
||||
},
|
||||
|
||||
"auto_mode": {
|
||||
"OBSERVATION": {
|
||||
"allowed_actions": ["observe"],
|
||||
"user_interaction": "notification",
|
||||
"reason": "insufficient_maturity"
|
||||
},
|
||||
"COACHING": {
|
||||
"allowed_actions": ["observe"],
|
||||
"user_interaction": "notification",
|
||||
"reason": "insufficient_maturity"
|
||||
},
|
||||
"AUTO_CANDIDATE": {
|
||||
"allowed_actions": ["observe", "auto_execute_supervised"],
|
||||
"user_interaction": "step_by_step_confirmation",
|
||||
"pause_on_anomaly": true
|
||||
},
|
||||
"AUTO_CONFIRMÉ": {
|
||||
"allowed_actions": ["observe", "auto_execute"],
|
||||
"user_interaction": "notification_only",
|
||||
"pause_on_anomaly": true,
|
||||
"rollback_on_error": true
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🌐 Multi-App & Multi-Contextes
|
||||
|
||||
### Workflows Multi-Applications
|
||||
|
||||
**Clarification** : Un workflow peut traverser plusieurs applications.
|
||||
|
||||
**Exemple** :
|
||||
|
||||
```json
|
||||
{
|
||||
"workflow_id": "WF_export_rapport_complet",
|
||||
"name": "Export rapport vers Excel",
|
||||
"description": "Extraire données du HIS, les traiter dans Excel, envoyer par email",
|
||||
|
||||
"applications": [
|
||||
"logiciel_his.exe",
|
||||
"EXCEL.EXE",
|
||||
"outlook.exe"
|
||||
],
|
||||
|
||||
"nodes": [
|
||||
{
|
||||
"node_id": "N1_his_rapport",
|
||||
"label": "Écran rapport HIS",
|
||||
"screen_template": {
|
||||
"window": {
|
||||
"app_name_any_of": ["logiciel_his.exe"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"node_id": "N2_excel_import",
|
||||
"label": "Excel - Import données",
|
||||
"screen_template": {
|
||||
"window": {
|
||||
"app_name_any_of": ["EXCEL.EXE"]
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"node_id": "N3_outlook_compose",
|
||||
"label": "Outlook - Nouveau message",
|
||||
"screen_template": {
|
||||
"window": {
|
||||
"app_name_any_of": ["outlook.exe"]
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
|
||||
"edges": [
|
||||
{
|
||||
"edge_id": "E1_export_to_excel",
|
||||
"from_node": "N1_his_rapport",
|
||||
"to_node": "N2_excel_import",
|
||||
"action": {
|
||||
"type": "compound",
|
||||
"steps": [
|
||||
{"type": "key_press", "keys": ["CTRL", "E"]},
|
||||
{"type": "wait", "duration_ms": 2000},
|
||||
{"type": "window_switch", "target_app": "EXCEL.EXE"}
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Contraintes** :
|
||||
- `window.app_name` + `window_title` font partie des contraintes de Node
|
||||
- Transitions peuvent inclure `window_switch` actions
|
||||
- Variables peuvent être passées entre applications (clipboard, fichiers)
|
||||
|
||||
---
|
||||
|
||||
## 📚 Dataset & Offline Training
|
||||
|
||||
### Stratégie de Dataset
|
||||
|
||||
**Clarification** : Export vers datasets supervisés pour fine-tuning.
|
||||
|
||||
```json
|
||||
{
|
||||
"dataset_export": {
|
||||
"dataset_id": "ds_ui_detection_2025_11",
|
||||
"version": "1.0",
|
||||
"created_at": "2025-11-22T00:00:00Z",
|
||||
|
||||
"source_workflows": [
|
||||
"WF_validation_facture",
|
||||
"WF_creation_dossier",
|
||||
"WF_export_rapport"
|
||||
],
|
||||
|
||||
"export_criteria": {
|
||||
"min_learning_state": "COACHING",
|
||||
"min_confidence": 0.85,
|
||||
"user_validated": true
|
||||
},
|
||||
|
||||
"samples": [
|
||||
{
|
||||
"sample_id": "sample_001",
|
||||
"screen_state_id": "screen_2025-11-21T10-15-32.123Z",
|
||||
"screenshot_path": "datasets/ds_ui_detection_2025_11/images/sample_001.png",
|
||||
"annotations": {
|
||||
"ui_elements": [ /* ... */ ],
|
||||
"layout_signature": "layout_abc123",
|
||||
"workflow_context": "WF_validation_facture",
|
||||
"node_id": "N2_detail_facture"
|
||||
},
|
||||
"validation_status": "user_confirmed"
|
||||
}
|
||||
],
|
||||
|
||||
"statistics": {
|
||||
"total_samples": 1250,
|
||||
"unique_layouts": 45,
|
||||
"unique_workflows": 12,
|
||||
"annotation_quality": 0.94
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Pipeline d'export** :
|
||||
1. Filtrer ScreenStates validés (COACHING/AUTO_CANDIDATE)
|
||||
2. Exporter images + annotations JSON
|
||||
3. Versionner le dataset
|
||||
4. Fine-tuner modèles (UIElement detection, rôle classification)
|
||||
5. Évaluer sur test set
|
||||
6. Déployer nouveau modèle avec nouveau `space.id`
|
||||
|
||||
---
|
||||
|
||||
## ✅ Checklist de Figeage
|
||||
|
||||
### À Figer Immédiatement
|
||||
|
||||
**Couche 0 - RawSession** :
|
||||
- [x] Schéma actuel documenté
|
||||
- [x] Segments (workflow episodes) ajoutés
|
||||
- [x] Marquage du bruit (noise_flag, semantic_role)
|
||||
- [x] Métadonnées de performance
|
||||
|
||||
**Couche 1 - ScreenState** :
|
||||
- [x] Lien explicite source (screenshot_id, event_ids)
|
||||
- [x] Layout signature (hash, method)
|
||||
- [x] Vue debug (summary généré par VLM)
|
||||
- [x] Zones d'intérêt (ROI)
|
||||
|
||||
**Couche 2 - UIElement** :
|
||||
- [x] Identité stable (stable_id, stability_confidence)
|
||||
- [x] Score d'actionnabilité (clickable_score, primary_action_score)
|
||||
- [x] Historique d'interaction
|
||||
- [x] États d'éléments
|
||||
|
||||
**Couche 3 - State Embedding** :
|
||||
- [x] Bloc space (id, version, metric) obligatoire
|
||||
- [x] Multi-têtes (primary, auxiliary)
|
||||
- [x] Métriques de qualité
|
||||
- [x] Versioning explicite
|
||||
|
||||
**Couche 4 - Workflow Graph** :
|
||||
- [x] Bloc variables (slots métier)
|
||||
- [x] Nœuds d'erreur (is_error_node, recovery_actions)
|
||||
- [x] Conditions et boucles
|
||||
- [x] Support multi-app
|
||||
|
||||
**Transverse** :
|
||||
- [x] Matrice Mode × Maturité documentée
|
||||
- [x] Stratégie dataset/training clarifiée
|
||||
- [x] Versioning de modèles géré
|
||||
|
||||
---
|
||||
|
||||
## 🎯 Conclusion
|
||||
|
||||
Ces enrichissements transforment l'architecture d'une **vision théorique** en **contrat stable** prêt pour l'implémentation.
|
||||
|
||||
### Concepts Clés Ajoutés
|
||||
|
||||
1. ✅ **Grammaire du temps** - Épisodes de workflow
|
||||
2. ✅ **Identités stables** - Tracking d'éléments
|
||||
3. ✅ **Versioning** - Migration de modèles
|
||||
4. ✅ **Variables métier** - Paramétrage
|
||||
5. ✅ **Gestion d'erreurs** - Chemins explicites
|
||||
6. ✅ **Matrice comportementale** - Mode × Maturité
|
||||
7. ✅ **Robustesse** - Layout vs Contenu
|
||||
8. ✅ **Actionnabilité** - Scores de cliquabilité
|
||||
|
||||
### Impact
|
||||
|
||||
**Avant** : Architecture élégante mais incomplète
|
||||
**Après** : Contrat stable pour implémentation production
|
||||
|
||||
**Prêt pour** :
|
||||
- 🔧 Implémentation des structures de données
|
||||
- 📊 Définition des schémas JSON formels
|
||||
- 🧪 Tests avec données réelles
|
||||
- 🚀 Déploiement progressif
|
||||
|
||||
---
|
||||
|
||||
**Document créé le** : 22 novembre 2024
|
||||
**Basé sur** : Analyse approfondie et retours d'expérience
|
||||
**Version** : 1.1
|
||||
**Statut** : ✅ Enrichissements Intégrés
|
||||
|
||||
Reference in New Issue
Block a user