feat: replay E2E fonctionnel — 25/25 actions, 0 retries, SomEngine via serveur

Validé sur PC Windows (DESKTOP-58D5CAC, 2560x1600) :
- 8 clics résolus visuellement (1 anchor_template, 1 som_text_match, 6 som_vlm)
- Score moyen 0.75, temps moyen 1.6s
- Texte tapé correctement (bonjour, test word, date, email)
- 0 retries, 2 actions non vérifiées (OK)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This commit is contained in:
Dom
2026-03-31 14:04:41 +02:00
parent 5e0b53cfd1
commit a7de6a488b
79542 changed files with 6091757 additions and 1 deletions

View File

@@ -0,0 +1,369 @@
---
inclusion: always
---
# RPA Vision V3 Design System Rules
This document provides comprehensive design system guidelines for integrating Figma designs with the RPA Vision V3 codebase using the Figma MCP integration.
## Project Architecture Overview
RPA Vision V3 is a multi-component system with distinct UI frameworks:
- **Visual Workflow Builder**: React + TypeScript + Material-UI frontend
- **Web Dashboard**: Flask + HTML/CSS/JavaScript with custom styling
- **Desktop GUI**: PyQt5 Python application
- **Agent V0**: Cross-platform capture tool with PyQt5 UI
- **Server API**: Flask-based REST API with minimal UI
## Design Token Definitions
### Color System
**Primary Colors:**
- Primary Blue: `#1976d2` (Material-UI primary)
- Secondary Pink: `#dc004e` (Material-UI secondary)
- Success Green: `#22c55e`
- Warning Orange: `#f59e0b`
- Error Red: `#ef4444`
**Background Colors:**
- Dark Background: `#0f172a` (Dashboard primary)
- Card Background: `#1e293b` (Dashboard cards)
- Border Color: `#334155`
- Text Primary: `#e2e8f0`
- Text Secondary: `#94a3b8`
- Text Muted: `#64748b`
**PyQt5 Colors:**
- Background: `#1e1e1e` (Dark theme)
- Text: `#d4d4d4`
- Success: `#4CAF50`
- Accent: `#2196f3`
### Typography Scale
**Web (Material-UI + Custom):**
- Headers: `'Segoe UI', Tahoma, Geneva, Verdana, sans-serif`
- Body: Material-UI default typography
- Monospace: `monospace` (for logs and code)
**PyQt5:**
- Default system fonts with size variations
- Monospace for technical displays
### Spacing System
**Web Spacing (px):**
- xs: 4px
- sm: 8px
- md: 12px
- lg: 16px
- xl: 20px
- xxl: 24px
- xxxl: 30px
**Component Spacing:**
- Card padding: 20px
- Button padding: 10px 20px
- Input padding: 12px
- Grid gaps: 15px-20px
## Component Library Structure
### React Components Location
```
visual_workflow_builder/frontend/src/components/
├── Canvas/ # Main workflow canvas
├── Palette/ # Tool palette
├── PropertiesPanel/ # Node properties editor
├── TemplateSelector/ # Workflow templates
├── AnalyticsDashboard/ # Analytics display
├── DocumentationTab/ # Interactive documentation
├── ExecutionPanel/ # Workflow execution controls
├── KeyboardShortcuts/ # Keyboard shortcuts help
└── [Other components]/
```
### Component Architecture
- **Material-UI based**: Primary component library
- **Custom CSS modules**: Component-specific styling in `.css` files
- **Hooks pattern**: Custom hooks for state management (`useWorkflow`, `useSelection`, etc.)
- **TypeScript interfaces**: Defined in `src/types/`
### PyQt5 Components Location
```
gui/
├── main_window.py # Main application window
├── components/ # Reusable UI components
└── orchestrator.py # Business logic integration
```
## Styling Approach
### React/TypeScript (Visual Workflow Builder)
**Primary Method**: Material-UI with custom CSS modules
```typescript
// Component structure
import { Box, Button, Typography } from '@mui/material';
import './ComponentName.css';
// Theme usage
const theme = createTheme({
palette: {
primary: { main: '#1976d2' },
secondary: { main: '#dc004e' }
}
});
```
**CSS Modules Pattern**:
```css
/* Component.css */
.component-container {
background: #1e293b;
border-radius: 12px;
padding: 20px;
border: 1px solid #334155;
}
.component-title {
color: #e2e8f0;
margin-bottom: 15px;
}
```
### Flask Dashboard (Web Dashboard)
**Method**: Inline CSS with utility classes
```html
<!-- Embedded styles in templates -->
<style>
.card {
background: #1e293b;
border-radius: 12px;
padding: 20px;
border: 1px solid #334155;
}
.btn-primary {
background: #3b82f6;
color: white;
padding: 10px 20px;
border-radius: 8px;
}
</style>
```
### PyQt5 (Desktop GUI)
**Method**: Inline StyleSheet strings
```python
# PyQt5 styling
widget.setStyleSheet("""
QWidget {
background: #1e1e1e;
color: #d4d4d4;
font-family: monospace;
}
QPushButton {
background: #4CAF50;
color: white;
padding: 10px;
font-size: 14px;
border-radius: 4px;
}
""")
```
## Asset Management
### Icons
- **Material-UI Icons**: `@mui/icons-material` for React components
- **Emoji Icons**: Used extensively in dashboard (🚀, 📊, ⚡, etc.)
- **Unicode Symbols**: For PyQt5 components (▶, ⏸, ⏹)
### Images and Screenshots
```
data/
├── screenshots/ # Runtime captured screenshots
├── training/ # Training data assets
└── [other data]/
```
### Asset Optimization
- **No CDN**: Local asset serving
- **Webpack bundling**: For React frontend
- **Flask static serving**: For dashboard assets
## Responsive Design Implementation
### React Components
```typescript
// Material-UI responsive breakpoints
const useStyles = makeStyles((theme) => ({
container: {
[theme.breakpoints.down('md')]: {
padding: theme.spacing(1),
},
[theme.breakpoints.up('lg')]: {
padding: theme.spacing(3),
},
},
}));
```
### Dashboard CSS
```css
/* Responsive grid */
.grid {
display: grid;
grid-template-columns: repeat(auto-fit, minmax(300px, 1fr));
gap: 20px;
}
@media (max-width: 768px) {
.grid { grid-template-columns: 1fr; }
.react-flow__minimap { display: none; }
}
```
## Figma Integration Guidelines
### Design-to-Code Mapping
**When using Figma MCP tools:**
1. **Replace Tailwind with Material-UI**:
```typescript
// Figma output (Tailwind)
<div className="bg-blue-500 text-white p-4 rounded-lg">
// Convert to Material-UI
<Box sx={{
bgcolor: 'primary.main',
color: 'white',
p: 2,
borderRadius: 2
}}>
```
2. **Use Project Color Tokens**:
```typescript
// Instead of Figma colors
color: '#1976d2' // Use theme.palette.primary.main
// Dashboard colors
background: '#1e293b' // Use established card background
```
3. **Maintain Component Consistency**:
```typescript
// Reuse existing components
import { Button } from '@mui/material';
// Instead of creating new button styles
```
4. **Respect Layout Patterns**:
```typescript
// Follow established grid patterns
<div className="grid grid-2"> // Dashboard pattern
<Grid container spacing={2}> // Material-UI pattern
```
### Code Connect Mapping
**Component Mapping Strategy**:
- Map Figma components to existing Material-UI components
- Create custom components only when necessary
- Maintain consistency with existing component APIs
**File Structure for Code Connect**:
```
visual_workflow_builder/frontend/src/components/
├── Button/index.tsx # Map to Material-UI Button
├── Card/index.tsx # Map to Material-UI Card
├── Input/index.tsx # Map to Material-UI TextField
└── [Custom components]/ # Project-specific components
```
## Build System Integration
### React Frontend
```json
// package.json build commands
{
"scripts": {
"start": "webpack serve --mode development",
"build": "webpack --mode production",
"type-check": "tsc --noEmit"
}
}
```
### Python Backend
```python
# Flask app structure
app/
├── static/ # Static assets
├── templates/ # Jinja2 templates
└── api/ # API endpoints
```
### Development Workflow
1. **Design in Figma**: Create/update designs
2. **Extract with MCP**: Use Figma tools to generate code
3. **Adapt to Project**: Convert to project patterns
4. **Integrate Components**: Add to existing component library
5. **Test Integration**: Ensure consistency with existing UI
## Framework-Specific Patterns
### React + Material-UI
- Use theme provider for consistent styling
- Leverage Material-UI's responsive system
- Custom CSS modules for component-specific styles
- TypeScript interfaces for props and state
### Flask Dashboard
- Inline styles for rapid development
- Utility CSS classes for common patterns
- WebSocket integration for real-time updates
- Chart.js for data visualization
### PyQt5 Desktop
- StyleSheet strings for theming
- Signal/slot pattern for interactions
- QThread for background operations
- System tray integration
## Quality Assurance
### Visual Consistency Checklist
- [ ] Colors match established palette
- [ ] Typography follows project scale
- [ ] Spacing uses project tokens
- [ ] Components reuse existing patterns
- [ ] Responsive behavior works across breakpoints
- [ ] Dark theme compatibility maintained
### Code Quality Standards
- [ ] TypeScript types defined
- [ ] Component props documented
- [ ] Accessibility attributes included
- [ ] Performance optimizations applied
- [ ] Error boundaries implemented
- [ ] Testing coverage maintained
## Migration Strategy
When integrating new Figma designs:
1. **Analyze Design**: Identify reusable vs. custom components
2. **Map to Existing**: Use existing components where possible
3. **Extend Carefully**: Create new components only when necessary
4. **Maintain Consistency**: Follow established patterns
5. **Update Documentation**: Keep design system current
6. **Test Thoroughly**: Ensure integration doesn't break existing functionality
This design system ensures consistent, maintainable integration between Figma designs and the RPA Vision V3 codebase while respecting the multi-framework architecture and existing component patterns.

35
.kiro/steering/product.md Normal file
View File

@@ -0,0 +1,35 @@
# Product Overview
RPA Vision V3 is a 100% vision-based workflow automation system that learns from user interactions and automates repetitive tasks through semantic understanding of user interfaces.
## Core Concept
Unlike traditional RPA systems that rely on fixed coordinates, RPA Vision V3 uses:
- **Semantic UI understanding** through computer vision and VLM models
- **Multi-modal embeddings** combining screenshots, text, and UI elements
- **Progressive learning** from observation to autonomous execution
- **Robust matching** that adapts to UI changes
## Key Features
- **Agent V0**: Cross-platform capture tool for recording user sessions
- **Hybrid Detection**: Combines OpenCV, CLIP embeddings, and VLM models
- **Visual Workflow Builder**: Web-based interface for creating and editing workflows
- **Self-Healing**: Automatic adaptation when UI elements change
- **Analytics System**: Performance monitoring and insights
- **Multi-modal Fusion**: Combines visual, textual, and spatial information
## Architecture Layers
1. **RawSession (Layer 0)**: Raw event capture (clicks, keystrokes, screenshots)
2. **ScreenState (Layer 1)**: Multi-modal analysis of screen content
3. **UIElement Detection (Layer 2)**: Semantic detection of interface elements
4. **State Embedding (Layer 3)**: Vector representation for similarity matching
5. **Workflow Graph (Layer 4)**: Executable workflow representation
## Learning Progression
- **OBSERVATION**: 5+ executions to learn patterns
- **COACHING**: 10+ assisted executions with >90% success
- **AUTO_CANDIDATE**: 20+ executions with >95% success rate
- **AUTO_CONFIRMED**: User-validated autonomous execution

120
.kiro/steering/structure.md Normal file
View File

@@ -0,0 +1,120 @@
# Project Structure
## Root Directory Organization
```
rpa_vision_v3/
├── core/ # Core system modules (5-layer architecture)
├── agent_v0/ # Cross-platform capture agent
├── server/ # API server and processing pipeline
├── web_dashboard/ # Flask web dashboard (port 5001)
├── visual_workflow_builder/ # React/TypeScript workflow editor
├── command_interface/ # Simple web command interface
├── gui/ # PyQt5 desktop application
├── data/ # Runtime data storage
├── tests/ # Comprehensive test suite
├── examples/ # Usage examples and demos
├── docs/ # Documentation and specifications
└── training_data/ # ML training datasets
```
## Core Architecture (`core/`)
Organized by the 5-layer architecture:
```
core/
├── models/ # Data structures for all layers (0-4)
├── capture/ # Layer 0: Raw event capture
├── detection/ # Layer 2: UI element detection
├── embedding/ # Layer 3: Multi-modal embeddings & FAISS
├── graph/ # Layer 4: Workflow graph construction
├── execution/ # Action execution and robustness
├── matching/ # Similarity matching algorithms
├── learning/ # Continuous learning and feedback
├── healing/ # Self-healing strategies
├── workflow/ # Workflow composition and management
├── analytics/ # Performance monitoring and insights
├── monitoring/ # Admin monitoring and automation
├── persistence/ # Data storage and retrieval
├── pipeline/ # Processing pipelines
├── training/ # ML model training
├── variants/ # UI variant handling
└── gpu/ # GPU resource management
```
## Data Organization (`data/`)
```
data/
├── sessions/ # RawSession JSON files
├── screen_states/ # ScreenState processed data
├── embeddings/ # Vector embeddings (.npy files)
├── faiss_index/ # FAISS similarity indices
├── workflows/ # Workflow graph definitions
├── screenshots/ # Captured screenshots
├── training/ # Training data and uploads
├── analytics/ # Analytics data storage
├── healing/ # Self-healing learning data
└── diagnostics/ # System diagnostic data
```
## Testing Structure (`tests/`)
```
tests/
├── unit/ # Unit tests for individual components
├── integration/ # Integration tests across components
├── property/ # Property-based tests (Hypothesis)
├── performance/ # Performance benchmarks
└── fixtures/ # Test data and fixtures
```
## Component Applications
- **`agent_v0/`**: Standalone capture agent with encryption and cross-platform support
- **`server/`**: Production API server with processing pipeline and queue management
- **`web_dashboard/`**: Flask-based monitoring dashboard
- **`visual_workflow_builder/`**: Full-stack React/TypeScript workflow editor
- **`gui/`**: PyQt5 desktop application for local use
## Naming Conventions
### Files
- **Python modules**: `snake_case.py`
- **Classes**: `PascalCase`
- **Functions/variables**: `snake_case`
- **Constants**: `UPPER_SNAKE_CASE`
### Directories
- **Core modules**: `snake_case`
- **Data directories**: `snake_case`
- **Component apps**: `snake_case`
### Data Files
- **Sessions**: `rawsession_YYYY-MM-DDTHH-MM-SS_userID.json`
- **Screenshots**: `shot_NNNN.png` (within session directories)
- **Embeddings**: `embedding_sessionID_stateID.npy`
- **Workflows**: `workflow_name_version.json`
## Configuration Management
- **Central config**: `core/config.py` with dataclass-based configuration
- **Environment files**: `.env` (local), `.env.example` (template)
- **Component configs**: Each component has its own config handling
- **Production overrides**: Environment variables take precedence
## Import Patterns
```python
# Core imports
from core.models import RawSession, ScreenState, UIElement
from core.detection import create_detector
from core.embedding import FusionEngine
from core.graph import GraphBuilder, NodeMatcher
# Component imports
from agent_v0.storage_encrypted import EncryptedStorage
from server.processing_pipeline import ProcessingPipeline
from web_dashboard.app import create_app
```

74
.kiro/steering/tech.md Normal file
View File

@@ -0,0 +1,74 @@
# Technology Stack
## Core Technologies
- **Python 3.8+**: Primary development language
- **PyTorch**: Deep learning framework for embeddings and models
- **FAISS**: Vector similarity search and indexing
- **OpenCV**: Computer vision and image processing
- **Flask**: Web API framework for server components
- **React + TypeScript**: Frontend for Visual Workflow Builder
## Key Libraries
- **open_clip_torch**: CLIP embeddings for visual-semantic understanding
- **transformers**: Hugging Face models (OWL-ViT for object detection)
- **Pillow**: Image processing and manipulation
- **PyQt5**: Desktop GUI framework
- **scikit-learn**: Machine learning utilities
- **pytest**: Testing framework with property-based testing (Hypothesis)
## External Dependencies
- **Ollama**: Local VLM inference server (qwen3-vl:8b model)
- **NVIDIA GPU**: Optional but recommended for performance
- **System capture libraries**: mss, pygetwindow, pyautogui
## Build System
### Environment Setup
```bash
# Create virtual environment
python3 -m venv venv_v3
source venv_v3/bin/activate # Linux/macOS
# or venv_v3\Scripts\activate # Windows
# Install dependencies
pip install -r requirements.txt
```
### Common Commands
```bash
# Complete setup and launch
./run.sh # Launch GUI (default)
./run.sh --server # API server only (port 8000)
./run.sh --dashboard # Web dashboard (port 5001)
./run.sh --all # All services
./run.sh --agent # Capture agent
./run.sh --check # Environment check only
# Testing
./test_quick.sh # Quick system test
pytest tests/ # Full test suite
pytest tests/unit/ # Unit tests only
pytest tests/integration/ # Integration tests
pytest tests/property/ # Property-based tests
# Development
./install_dependencies.sh # Manual dependency install
./status.sh # System status check
```
### Configuration
- **Environment variables**: Use `.env` file (see `.env.example`)
- **Central config**: `core/config.py` with dataclass-based configuration
- **Production**: Set `ENVIRONMENT=production` for security validation
### Project Structure
- **Modular architecture**: Each layer in separate `core/` subdirectories
- **Test-driven**: Comprehensive test coverage with property-based testing
- **Multi-component**: GUI, server, dashboard, agent as separate runnable components
- **Cross-platform**: Linux, macOS, Windows support with platform-specific handling