rpa_vision_v3/visual_workflow_builder/docs/API_INTEGRATION.md

Guide d'Intégration API - Visual Workflow Builder

Vue d'ensemble

Ce guide détaille l'intégration avec les APIs backend du Visual Workflow Builder pour le système de sélection visuelle 100% vision-based.

🔌 Endpoints Principaux

Visual Targets API (/api/visual/)

Créer une Cible Visuelle

POST /api/visual/targets
Content-Type: application/json

{
  "position": {"x": 100, "y": 200}
}

Réponse :

{
  "signature": "visual_abc123_def456",
  "screenshot": "base64_image_data",
  "bounding_box": {"x": 95, "y": 195, "width": 50, "height": 30},
  "confidence": 1.0,
  "metadata": {
    "element_type": "button",
    "visual_description": "Bouton bleu rectangulaire",
    "text_content": "Valider"
  },
  "created_at": "2024-01-07T10:30:00Z"
}

Valider une Cible

POST /api/visual/targets/{signature}/validate

Réponse :

{
  "is_valid": true,
  "confidence": 0.95,
  "current_position": {"x": 95, "y": 195, "width": 50, "height": 30},
  "suggestions": [],
  "issues": []
}

Element Detection API (/api/detection/)

Détecter les Éléments

POST /api/detection/elements
Content-Type: application/json

{
  "screenshot": "base64_image_data",
  "confidence_threshold": 0.7
}

Réponse :

{
  "elements": [
    {
      "id": "element_1",
      "type": "button",
      "bounds": {"x": 100, "y": 200, "width": 80, "height": 30},
      "confidence": 0.95,
      "text": "Cliquer ici"
    }
  ],
  "processing_time": 1.23,
  "total_elements": 15
}

🛠 Services Frontend

VisualTargetService

import { visualTargetService } from '../services/VisualTargetService';

// Créer une cible
const target = await visualTargetService.captureAndSelectElement({x: 100, y: 200});

// Valider une cible
const validation = await visualTargetService.validateTarget(signature);

// Trouver des éléments similaires
const similar = await visualTargetService.findSimilarElements(signature);

ScreenCaptureService

import { screenCaptureService } from '../services/ScreenCaptureService';

// Capture d'écran
const result = await screenCaptureService.captureScreen({
  mode: 'fullscreen',
  quality: 0.8,
  timeout: 10000
});

// Capture de région
const regionResult = await screenCaptureService.captureRegion({
  x: 0, y: 0, width: 800, height: 600
});

🔧 Configuration

Variables d'Environnement

# Backend
VISUAL_TARGET_CACHE_SIZE=100
DETECTION_CONFIDENCE_THRESHOLD=0.7
SCREEN_CAPTURE_TIMEOUT=10000

# Frontend
REACT_APP_API_BASE_URL=http://localhost:5002
REACT_APP_ENABLE_DEBUG=true

Initialisation Backend

# app.py
from core.capture.screen_capturer import ScreenCapturer
from core.detection.ui_detector import UIDetector
from core.embedding.fusion_engine import FusionEngine

# Initialiser les composants
screen_capturer = ScreenCapturer()
ui_detector = UIDetector()
fusion_engine = FusionEngine()

# Initialiser les services
init_visual_target_manager(screen_capturer, ui_detector, fusion_engine)
init_element_detection(ui_detector, screen_capturer)

📊 Gestion d'Erreurs

Codes d'Erreur Communs

Code	Description	Solution
400	Données invalides	Vérifier le format des paramètres
404	Cible non trouvée	Vérifier la signature
500	Erreur serveur	Vérifier les logs backend
408	Timeout	Augmenter le timeout

Gestion Frontend

try {
  const target = await visualTargetService.validateTarget(signature);
} catch (error) {
  if (error.message.includes('timeout')) {
    // Retry avec timeout plus long
    await visualTargetService.validateTarget(signature, { timeout: 15000 });
  } else if (error.message.includes('not found')) {
    // Recréer la cible
    console.warn('Cible expirée, recréation nécessaire');
  }
}

🚀 Optimisations

Cache et Performance

// Configuration du cache
const cache = new ImageCache(50, 30); // 50MB, 30min

// Préchargement
await visualTargetService.preloadImages([
  { signature: 'target1', data: 'base64...' },
  { signature: 'target2', data: 'base64...' }
]);

// Statistiques
const stats = cache.getStats();
console.log(`Hit rate: ${stats.hitRate}%`);

Batch Operations

// Validation en lot
const signatures = ['sig1', 'sig2', 'sig3'];
const validations = await Promise.allSettled(
  signatures.map(sig => visualTargetService.validateTarget(sig))
);

🔒 Sécurité

Authentification

// Headers requis
const headers = {
  'Authorization': `Bearer ${token}`,
  'Content-Type': 'application/json'
};

Validation des Données

# Backend validation
from marshmallow import Schema, fields, validate

class VisualTargetSchema(Schema):
    position = fields.Dict(required=True)
    confidence_threshold = fields.Float(validate=validate.Range(0, 1))

📈 Monitoring

Métriques Backend

# Logs structurés
logger.info(f"Visual target created: {signature}", extra={
    'confidence': confidence,
    'processing_time': processing_time,
    'element_type': element_type
})

Métriques Frontend

// Performance tracking
const { metrics } = usePerformanceOptimization();

// Envoi des métriques
analytics.track('visual_target_created', {
  confidence: target.confidence,
  processing_time: metrics.renderTime,
  element_type: target.metadata.element_type
});

🧪 Tests d'Intégration

Tests API

def test_create_visual_target():
    response = client.post('/api/visual/targets', json={
        'position': {'x': 100, 'y': 200}
    })
    assert response.status_code == 201
    assert 'signature' in response.json()

Tests Frontend

describe('VisualTargetService Integration', () => {
  it('should create and validate target', async () => {
    const target = await visualTargetService.captureAndSelectElement({x: 100, y: 200});
    expect(target.signature).toBeTruthy();
    
    const validation = await visualTargetService.validateTarget(target.signature);
    expect(validation.is_valid).toBe(true);
  });
});

📚 Ressources

• Guide de sélection visuelle
• Guide de dépannage
• Tests de propriétés
• Documentation API complète