v1.0 - Version stable: multi-PC, détection UI-DETR-1, 3 modes exécution

- Frontend v4 accessible sur réseau local (192.168.1.40) - Ports ouverts: 3002 (frontend), 5001 (backend), 5004 (dashboard) - Ollama GPU fonctionnel - Self-healing interactif - Dashboard confiance Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
2026-01-29 11:23:51 +01:00
parent 21bfa3b337
commit a27b74cf22
1595 changed files with 412691 additions and 400 deletions
--- a/visual_workflow_builder/docs/API_INTEGRATION.md
+++ b/visual_workflow_builder/docs/API_INTEGRATION.md
@@ -0,0 +1,289 @@
+# Guide d'Intégration API - Visual Workflow Builder
+
+## Vue d'ensemble
+
+Ce guide détaille l'intégration avec les APIs backend du Visual Workflow Builder pour le système de sélection visuelle 100% vision-based.
+
+## 🔌 Endpoints Principaux
+
+### Visual Targets API (`/api/visual/`)
+
+#### Créer une Cible Visuelle
+```http
+POST /api/visual/targets
+Content-Type: application/json
+
+{
+  "position": {"x": 100, "y": 200}
+}
+```
+
+**Réponse :**
+```json
+{
+  "signature": "visual_abc123_def456",
+  "screenshot": "base64_image_data",
+  "bounding_box": {"x": 95, "y": 195, "width": 50, "height": 30},
+  "confidence": 1.0,
+  "metadata": {
+    "element_type": "button",
+    "visual_description": "Bouton bleu rectangulaire",
+    "text_content": "Valider"
+  },
+  "created_at": "2024-01-07T10:30:00Z"
+}
+```
+
+#### Valider une Cible
+```http
+POST /api/visual/targets/{signature}/validate
+```
+
+**Réponse :**
+```json
+{
+  "is_valid": true,
+  "confidence": 0.95,
+  "current_position": {"x": 95, "y": 195, "width": 50, "height": 30},
+  "suggestions": [],
+  "issues": []
+}
+```
+
+### Element Detection API (`/api/detection/`)
+
+#### Détecter les Éléments
+```http
+POST /api/detection/elements
+Content-Type: application/json
+
+{
+  "screenshot": "base64_image_data",
+  "confidence_threshold": 0.7
+}
+```
+
+**Réponse :**
+```json
+{
+  "elements": [
+    {
+      "id": "element_1",
+      "type": "button",
+      "bounds": {"x": 100, "y": 200, "width": 80, "height": 30},
+      "confidence": 0.95,
+      "text": "Cliquer ici"
+    }
+  ],
+  "processing_time": 1.23,
+  "total_elements": 15
+}
+```
+
+## 🛠 Services Frontend
+
+### VisualTargetService
+
+```typescript
+import { visualTargetService } from '../services/VisualTargetService';
+
+// Créer une cible
+const target = await visualTargetService.captureAndSelectElement({x: 100, y: 200});
+
+// Valider une cible
+const validation = await visualTargetService.validateTarget(signature);
+
+// Trouver des éléments similaires
+const similar = await visualTargetService.findSimilarElements(signature);
+```
+
+### ScreenCaptureService
+
+```typescript
+import { screenCaptureService } from '../services/ScreenCaptureService';
+
+// Capture d'écran
+const result = await screenCaptureService.captureScreen({
+  mode: 'fullscreen',
+  quality: 0.8,
+  timeout: 10000
+});
+
+// Capture de région
+const regionResult = await screenCaptureService.captureRegion({
+  x: 0, y: 0, width: 800, height: 600
+});
+```
+
+## 🔧 Configuration
+
+### Variables d'Environnement
+
+```bash
+# Backend
+VISUAL_TARGET_CACHE_SIZE=100
+DETECTION_CONFIDENCE_THRESHOLD=0.7
+SCREEN_CAPTURE_TIMEOUT=10000
+
+# Frontend
+REACT_APP_API_BASE_URL=http://localhost:5002
+REACT_APP_ENABLE_DEBUG=true
+```
+
+### Initialisation Backend
+
+```python
+# app.py
+from core.capture.screen_capturer import ScreenCapturer
+from core.detection.ui_detector import UIDetector
+from core.embedding.fusion_engine import FusionEngine
+
+# Initialiser les composants
+screen_capturer = ScreenCapturer()
+ui_detector = UIDetector()
+fusion_engine = FusionEngine()
+
+# Initialiser les services
+init_visual_target_manager(screen_capturer, ui_detector, fusion_engine)
+init_element_detection(ui_detector, screen_capturer)
+```
+
+## 📊 Gestion d'Erreurs
+
+### Codes d'Erreur Communs
+
+| Code | Description | Solution |
+|------|-------------|----------|
+| 400 | Données invalides | Vérifier le format des paramètres |
+| 404 | Cible non trouvée | Vérifier la signature |
+| 500 | Erreur serveur | Vérifier les logs backend |
+| 408 | Timeout | Augmenter le timeout |
+
+### Gestion Frontend
+
+```typescript
+try {
+  const target = await visualTargetService.validateTarget(signature);
+} catch (error) {
+  if (error.message.includes('timeout')) {
+    // Retry avec timeout plus long
+    await visualTargetService.validateTarget(signature, { timeout: 15000 });
+  } else if (error.message.includes('not found')) {
+    // Recréer la cible
+    console.warn('Cible expirée, recréation nécessaire');
+  }
+}
+```
+
+## 🚀 Optimisations
+
+### Cache et Performance
+
+```typescript
+// Configuration du cache
+const cache = new ImageCache(50, 30); // 50MB, 30min
+
+// Préchargement
+await visualTargetService.preloadImages([
+  { signature: 'target1', data: 'base64...' },
+  { signature: 'target2', data: 'base64...' }
+]);
+
+// Statistiques
+const stats = cache.getStats();
+console.log(`Hit rate: ${stats.hitRate}%`);
+```
+
+### Batch Operations
+
+```typescript
+// Validation en lot
+const signatures = ['sig1', 'sig2', 'sig3'];
+const validations = await Promise.allSettled(
+  signatures.map(sig => visualTargetService.validateTarget(sig))
+);
+```
+
+## 🔒 Sécurité
+
+### Authentification
+
+```typescript
+// Headers requis
+const headers = {
+  'Authorization': `Bearer ${token}`,
+  'Content-Type': 'application/json'
+};
+```
+
+### Validation des Données
+
+```python
+# Backend validation
+from marshmallow import Schema, fields, validate
+
+class VisualTargetSchema(Schema):
+    position = fields.Dict(required=True)
+    confidence_threshold = fields.Float(validate=validate.Range(0, 1))
+```
+
+## 📈 Monitoring
+
+### Métriques Backend
+
+```python
+# Logs structurés
+logger.info(f"Visual target created: {signature}", extra={
+    'confidence': confidence,
+    'processing_time': processing_time,
+    'element_type': element_type
+})
+```
+
+### Métriques Frontend
+
+```typescript
+// Performance tracking
+const { metrics } = usePerformanceOptimization();
+
+// Envoi des métriques
+analytics.track('visual_target_created', {
+  confidence: target.confidence,
+  processing_time: metrics.renderTime,
+  element_type: target.metadata.element_type
+});
+```
+
+## 🧪 Tests d'Intégration
+
+### Tests API
+
+```python
+def test_create_visual_target():
+    response = client.post('/api/visual/targets', json={
+        'position': {'x': 100, 'y': 200}
+    })
+    assert response.status_code == 201
+    assert 'signature' in response.json()
+```
+
+### Tests Frontend
+
+```typescript
+describe('VisualTargetService Integration', () => {
+  it('should create and validate target', async () => {
+    const target = await visualTargetService.captureAndSelectElement({x: 100, y: 200});
+    expect(target.signature).toBeTruthy();
+    
+    const validation = await visualTargetService.validateTarget(target.signature);
+    expect(validation.is_valid).toBe(true);
+  });
+});
+```
+
+## 📚 Ressources
+
+- [Guide de sélection visuelle](./VISUAL_SELECTION_GUIDE.md)
+- [Guide de dépannage](./TROUBLESHOOTING.md)
+- [Tests de propriétés](../frontend/src/__tests__/properties/)
+- [Documentation API complète](./api-docs/)