Dom/rpa_vision_v3
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: chat unifié, GestureCatalog, Copilot, Léa UI, extraction données, vérification replay

Browse Source 
Refonte majeure du système Agent Chat et ajout de nombreux modules :

- Chat unifié : suppression du dual Workflows/Agent Libre, tout passe par /api/chat
  avec résolution en 3 niveaux (workflow → geste → "montre-moi")
- GestureCatalog : 38 raccourcis clavier universels Windows avec matching sémantique,
  substitution automatique dans les replays, et endpoint /api/gestures
- Mode Copilot : exécution pas-à-pas des workflows avec validation humaine via WebSocket
  (approve/skip/abort) avant chaque action
- Léa UI (agent_v0/lea_ui/) : interface PyQt5 pour Windows avec overlay transparent
  pour feedback visuel pendant le replay
- Data Extraction (core/extraction/) : moteur d'extraction visuelle de données
  (OCR + VLM → SQLite), avec schémas YAML et export CSV/Excel
- ReplayVerifier (agent_v0/server_v1/) : vérification post-action par comparaison
  de screenshots, avec logique de retry (max 3)
- IntentParser durci : meilleur fallback regex, type GREETING, patterns améliorés
- Dashboard : nouvelles pages gestures, streaming, extractions
- Tests : 63 tests GestureCatalog, 47 tests extraction, corrections tests existants
- Dépréciation : /api/agent/plan et /api/agent/execute retournent HTTP 410,
  suppression du code hardcodé _plan_to_replay_actions

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Dom
2026-03-15 10:02:09 +01:00
parent 74a1cb4e03
commit cf495dd82f
93 changed files with 12463 additions and 1080 deletions
Download Patch File Download Diff File Expand all files Collapse all files

185
run.sh
View File

@@ -38,7 +38,8 @@ show_help() {
echo -e " ${YELLOW}--monitoring${NC} 📈 Interface de monitoring (port 5003)"
echo -e " ${CYAN}--workflow${NC} 🔧 Visual Workflow Builder (port 3002)"
echo -e " ${GREEN}--agent${NC} 📹 Agent V0 (capture tool)"
echo -e " ${BLUE}--chat${NC} 💬 Agent Chat (port 5002)"
echo -e " ${BLUE}--chat${NC} 💬 Agent Chat (port 5004)"
echo -e " ${GREEN}--stream${NC} 📡 Streaming Server (port 5005)"
echo ""
echo -e "${BOLD}🧪 Tests & Validation:${NC}"
echo -e " ${GREEN}--test${NC} 🧪 Tests complets"
@@ -62,7 +63,8 @@ show_help() {
echo -e "${BOLD}🌐 URLs d'accès:${NC}"
echo -e " API Server: ${BLUE}http://localhost:8000${NC}"
echo -e " Dashboard: ${PURPLE}http://localhost:5001${NC}"
echo -e " Agent Chat: ${BLUE}http://localhost:5002${NC}"
echo -e " Agent Chat: ${BLUE}http://localhost:5004${NC}"
echo -e " Streaming API: ${GREEN}http://localhost:5005${NC}"
echo -e " Monitoring: ${YELLOW}http://localhost:5003${NC}"
echo -e " Workflow Builder: ${CYAN}http://localhost:3002${NC}"
echo ""
@@ -78,6 +80,7 @@ LAUNCH_MONITORING=false
LAUNCH_WORKFLOW=false
LAUNCH_AGENT=false
LAUNCH_COMMAND=false
LAUNCH_STREAM=false
for arg in "$@"; do
case $arg in
@@ -120,6 +123,10 @@ for arg in "$@"; do
MODE="chat"
LAUNCH_COMMAND=true
;;
--stream)
MODE="stream"
LAUNCH_STREAM=true
;;
--test)
MODE="test"
;;
@@ -344,13 +351,10 @@ cleanup() {
[ ! -z "$WORKFLOW_PID" ] && kill $WORKFLOW_PID 2>/dev/null || true
[ ! -z "$COMMAND_PID" ] && kill $COMMAND_PID 2>/dev/null || true
# Kill any remaining processes on our ports
pkill -f "port 8000" 2>/dev/null || true
pkill -f "port 5001" 2>/dev/null || true
pkill -f "port 5002" 2>/dev/null || true
pkill -f "port 5003" 2>/dev/null || true
pkill -f "port 3002" 2>/dev/null || true
pkill -f "vite.*3002" 2>/dev/null || true
# Kill any remaining processes on our ports (fuser est fiable, pkill -f "port" ne matche pas)
for p in 8000 5001 5002 5003 5004 5005 3002; do
fuser -k "${p}/tcp" 2>/dev/null || true
done
deactivate 2>/dev/null || true
echo -e "${GREEN}✓${NC} Cleanup complete"
@@ -395,72 +399,6 @@ case $MODE in
echo ""
echo "Access: http://localhost:5003"
echo ""
# Create simple monitoring interface
cat > monitoring_server.py << 'EOF'
from flask import Flask, render_template_string
import psutil
import json
from datetime import datetime
app = Flask(__name__)
@app.route('/')
def monitoring():
return render_template_string('''
<!DOCTYPE html>
<html>
<head>
<title>🎼 RPA Vision V3 - Monitoring</title>
<meta http-equiv="refresh" content="5">
<style>
body { font-family: Arial; margin: 20px; background: #f5f5f5; }
.container { max-width: 1200px; margin: 0 auto; }
.card { background: white; padding: 20px; margin: 10px 0; border-radius: 8px; box-shadow: 0 2px 4px rgba(0,0,0,0.1); }
.status-ok { color: #28a745; }
.status-warning { color: #ffc107; }
.status-error { color: #dc3545; }
.metric { display: inline-block; margin: 10px 20px 10px 0; }
.services { display: grid; grid-template-columns: repeat(auto-fit, minmax(300px, 1fr)); gap: 15px; }
</style>
</head>
<body>
<div class="container">
<h1>🎼 RPA Vision V3 - Monitoring Dashboard</h1>
<div class="card">
<h2>📊 System Metrics</h2>
<div class="metric">CPU: {{ cpu }}%</div>
<div class="metric">Memory: {{ memory }}%</div>
<div class="metric">Disk: {{ disk }}%</div>
<div class="metric">Uptime: {{ uptime }}</div>
</div>
<div class="card">
<h2>🌐 Services Status</h2>
<div class="services">
<div>API Server (8000): <span id="api-status">Checking...</span></div>
<div>Dashboard (5001): <span id="dashboard-status">Checking...</span></div>
<div>Command (5002): <span id="command-status">Checking...</span></div>
<div>Workflow (3000): <span id="workflow-status">Checking...</span></div>
</div>
</div>
<div class="card">
<h2>📈 RPA Vision V3 Status</h2>
<p>✅ Fiche #1 & #2 Corrections Applied</p>
<p>🎯 BBOX Precision: ~95% (improved from ~60%)</p>
<p>🔧 All contrats de données unified</p>
</div>
</div>
</body>
</html>
''',
cpu=psutil.cpu_percent(),
memory=psutil.virtual_memory().percent,
disk=psutil.disk_usage('/').percent,
uptime=str(datetime.now() - datetime.fromtimestamp(psutil.boot_time())).split('.')[0]
)
if __name__ == '__main__':
app.run(host='0.0.0.0', port=5003, debug=False)
EOF
$VENV_DIR/bin/python3 monitoring_server.py
;;
@@ -468,7 +406,7 @@ EOF
echo ""
echo -e "${CYAN}🔧 Launching Visual Workflow Builder v4...${NC}"
echo ""
echo "Access: http://localhost:3002 (frontend) / http://localhost:5001 (backend)"
echo "Access: http://localhost:3002 (frontend) / http://localhost:5002 (backend)"
echo ""
cd visual_workflow_builder
./run_v4.sh
@@ -495,13 +433,22 @@ EOF
chat)
echo ""
echo -e "${BLUE}💬 Launching Agent Chat on port 5002...${NC}"
echo -e "${BLUE}💬 Launching Agent Chat on port 5004...${NC}"
echo ""
echo "Access: http://localhost:5002"
echo "Access: http://localhost:5004"
echo ""
$VENV_DIR/bin/python3 agent_chat/app.py
;;
stream)
echo ""
echo -e "${GREEN}📡 Launching Streaming Server on port 5005...${NC}"
echo ""
echo "Access: http://localhost:5005"
echo ""
$VENV_DIR/bin/python3 -m agent_v0.server_v1.api_stream
;;
full)
echo ""
echo -e "${GREEN}${BOLD}🎯 Launching FULL ECOSYSTEM...${NC}"
@@ -514,72 +461,6 @@ EOF
DASHBOARD_PID=$(start_service "Dashboard" "$VENV_DIR/bin/python3 web_dashboard/app.py" "5001" "dashboard.log")
# Start Monitoring
cat > monitoring_server.py << 'EOF'
from flask import Flask, render_template_string
import psutil
import json
from datetime import datetime
app = Flask(__name__)
@app.route('/')
def monitoring():
return render_template_string('''
<!DOCTYPE html>
<html>
<head>
<title>🎼 RPA Vision V3 - Monitoring</title>
<meta http-equiv="refresh" content="5">
<style>
body { font-family: Arial; margin: 20px; background: #f5f5f5; }
.container { max-width: 1200px; margin: 0 auto; }
.card { background: white; padding: 20px; margin: 10px 0; border-radius: 8px; box-shadow: 0 2px 4px rgba(0,0,0,0.1); }
.status-ok { color: #28a745; }
.status-warning { color: #ffc107; }
.status-error { color: #dc3545; }
.metric { display: inline-block; margin: 10px 20px 10px 0; }
.services { display: grid; grid-template-columns: repeat(auto-fit, minmax(300px, 1fr)); gap: 15px; }
</style>
</head>
<body>
<div class="container">
<h1>🎼 RPA Vision V3 - Monitoring Dashboard</h1>
<div class="card">
<h2>📊 System Metrics</h2>
<div class="metric">CPU: {{ cpu }}%</div>
<div class="metric">Memory: {{ memory }}%</div>
<div class="metric">Disk: {{ disk }}%</div>
<div class="metric">Uptime: {{ uptime }}</div>
</div>
<div class="card">
<h2>🌐 Services Status</h2>
<div class="services">
<div>API Server (8000): <span class="status-ok">✅ Running</span></div>
<div>Dashboard (5001): <span class="status-ok">✅ Running</span></div>
<div>Monitoring (5003): <span class="status-ok">✅ Running</span></div>
<div>Command (5002): <span class="status-warning">⚠️ Optional</span></div>
</div>
</div>
<div class="card">
<h2>📈 RPA Vision V3 Status</h2>
<p>✅ Fiche #1 & #2 Corrections Applied</p>
<p>🎯 BBOX Precision: ~95% (improved from ~60%)</p>
<p>🔧 All contrats de données unified</p>
<p>🚀 Full ecosystem running!</p>
</div>
</div>
</body>
</html>
''',
cpu=psutil.cpu_percent(),
memory=psutil.virtual_memory().percent,
disk=psutil.disk_usage('/').percent,
uptime=str(datetime.now() - datetime.fromtimestamp(psutil.boot_time())).split('.')[0]
)
if __name__ == '__main__':
app.run(host='0.0.0.0', port=5003, debug=False)
EOF
MONITORING_PID=$(start_service "Monitoring" "$VENV_DIR/bin/python3 monitoring_server.py" "5003" "monitoring.log")
# Start Visual Workflow Builder v4 (in background)
@@ -662,7 +543,7 @@ EOF
echo ""
echo -e "${CYAN}⚡ Running quick tests...${NC}"
echo ""
./test_quick.sh
$VENV_DIR/bin/python3 -m pytest tests/ -m "not slow" -q --tb=short
;;
test-bbox)
@@ -696,7 +577,8 @@ EOF
echo ""
check_service_status "API Server" "8000"
check_service_status "Dashboard" "5001"
check_service_status "Agent Chat" "5002"
check_service_status "Agent Chat" "5004"
check_service_status "Streaming API" "5005"
check_service_status "Monitoring" "5003"
check_service_status "Workflow Builder" "3002"
echo ""
@@ -705,12 +587,9 @@ EOF
stop)
echo ""
echo -e "${RED}🛑 Stopping all services...${NC}"
pkill -f "port 8000" 2>/dev/null || true
pkill -f "port 5001" 2>/dev/null || true
pkill -f "port 5002" 2>/dev/null || true
pkill -f "port 5003" 2>/dev/null || true
pkill -f "port 3002" 2>/dev/null || true
pkill -f "vite.*3002" 2>/dev/null || true
for p in 8000 5001 5002 5003 5004 5005 3002; do
fuser -k "${p}/tcp" 2>/dev/null || true
done
echo -e "${GREEN}✓${NC} All services stopped"
;;
esac