Pipeline replay visuel : - VLM-first : l'agent appelle Ollama directement pour trouver les éléments - Template matching en fallback (seuil strict 0.90) - Stop immédiat si élément non trouvé (pas de clic blind) - Replay depuis session brute (/replay-session) sans attendre le VLM - Vérification post-action (screenshot hash avant/après) - Gestion des popups (Enter/Escape/Tab+Enter) Worker VLM séparé : - run_worker.py : process distinct du serveur HTTP - Communication par fichiers (_worker_queue.txt + _replay_active.lock) - Le serveur HTTP ne fait plus jamais de VLM → toujours réactif - Service systemd rpa-worker.service Capture clavier : - raw_keys (vk + press/release) pour replay exact indépendant du layout - Fix AZERTY : ToUnicodeEx + AltGr detection - Enter capturé comme \n, Tab comme \t - Filtrage modificateurs seuls (Ctrl/Alt/Shift parasites) - Fusion text_input consécutifs, dédup key_combo Sécurité & Internet : - HTTPS Let's Encrypt (lea.labs + vwb.labs.laurinebazin.design) - Token API fixe dans .env.local - HTTP Basic Auth sur VWB - Security headers (HSTS, CSP, nosniff) - CORS domaines publics, plus de wildcard Infrastructure : - DPI awareness (SetProcessDpiAwareness) Python + Rust - Métadonnées système (dpi_scale, window_bounds, monitors, os_theme) - Template matching multi-scale [0.5, 2.0] - Résolution dynamique (plus de hardcode 1920x1080) - VLM prefill fix (47x speedup, 3.5s au lieu de 180s) Modules : - core/auth/ : credential vault (Fernet AES), TOTP (RFC 6238), auth handler - core/federation/ : LearningPack export/import anonymisé, FAISS global - deploy/ : package Léa (config.txt, Lea.bat, install.bat, LISEZMOI.txt) UX : - Filtrage OS (VWB + Chat montrent que les workflows de l'OS courant) - Bibliothèque persistante (cache local + SQLite) - Clustering hybride (titre fenêtre + DBSCAN) - EdgeConstraints + PostConditions peuplés - GraphBuilder compound actions (toutes les frappes) Agent Rust : - Token Bearer auth (network.rs) - sysinfo.rs (DPI, résolution, window bounds via Win32 API) - config.txt lu automatiquement - Support Chrome/Brave/Firefox (pas que Edge) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
431 lines
15 KiB
Rust
431 lines
15 KiB
Rust
//! Agent RPA Vision — Phases 1-5 (parite complete)
|
|
//!
|
|
//! Point d'entree principal. Architecture multi-threads :
|
|
//!
|
|
//! - Thread principal : boucle d'evenements systray (Windows) ou attente console (Linux)
|
|
//! - Thread heartbeat : capture + envoi toutes les 5s (avec dedup par hash)
|
|
//! - Thread replay : poll toutes les 1s, execute les actions
|
|
//! - Thread serveur : HTTP port 5006 pour les captures a la demande
|
|
//! - Thread recorder : capture evenements souris/clavier (quand enregistrement actif)
|
|
//! - Thread chat : fenetre WebView2 (Windows, a la demande)
|
|
//! - Thread health : verification connexion serveur (toutes les 30s)
|
|
//!
|
|
//! Le thread principal gere le systray sur Windows via winit.
|
|
//! Sur Linux, le thread principal attend Ctrl+C (mode console).
|
|
//!
|
|
//! Configuration via variables d'environnement ou valeurs par defaut.
|
|
//! Compatible avec le serveur streaming existant (api_stream.py, port 5005).
|
|
|
|
#[allow(dead_code)]
|
|
mod blur;
|
|
mod capture;
|
|
mod chat;
|
|
mod config;
|
|
mod executor;
|
|
mod network;
|
|
#[allow(dead_code)]
|
|
mod notifications;
|
|
mod recorder;
|
|
mod replay;
|
|
mod server;
|
|
#[allow(dead_code)]
|
|
mod state;
|
|
mod sysinfo;
|
|
mod tray;
|
|
mod visual;
|
|
|
|
use config::Config;
|
|
use reqwest::blocking::Client;
|
|
use state::AgentState;
|
|
use std::sync::Arc;
|
|
use std::thread;
|
|
use std::time::Duration;
|
|
|
|
/// Trouve un navigateur compatible sur Windows (Edge, Chrome, Brave, Firefox)
|
|
#[cfg(target_os = "windows")]
|
|
fn find_browser() -> Option<String> {
|
|
let paths = [
|
|
// Edge
|
|
r"C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe",
|
|
r"C:\Program Files\Microsoft\Edge\Application\msedge.exe",
|
|
// Chrome
|
|
r"C:\Program Files\Google\Chrome\Application\chrome.exe",
|
|
r"C:\Program Files (x86)\Google\Chrome\Application\chrome.exe",
|
|
// Brave
|
|
r"C:\Program Files\BraveSoftware\Brave-Browser\Application\brave.exe",
|
|
// Firefox (supporte --kiosk mais pas --app)
|
|
r"C:\Program Files\Mozilla Firefox\firefox.exe",
|
|
];
|
|
for p in &paths {
|
|
if std::path::Path::new(p).exists() {
|
|
return Some(p.to_string());
|
|
}
|
|
}
|
|
None
|
|
}
|
|
|
|
fn main() {
|
|
// --- DPI awareness (DOIT etre appele avant toute operation graphique) ---
|
|
// Rend le process DPI-aware sur Windows pour que les API (enigo, xcap,
|
|
// GetSystemMetrics, etc.) travaillent en coordonnees physiques (pixels reels)
|
|
// au lieu de coordonnees logiques (virtualisees par le DPI scaling).
|
|
// Sans cet appel, un ecran 2560x1600 a 150% DPI apparait comme 1707x1067
|
|
// pour enigo et GetSystemMetrics, ce qui cause des erreurs de positionnement
|
|
// pendant le replay.
|
|
// PROCESS_PER_MONITOR_DPI_AWARE = 2 : le niveau le plus precis.
|
|
#[cfg(target_os = "windows")]
|
|
{
|
|
// SetProcessDpiAwareness (shcore.dll) et SetProcessDPIAware (user32.dll)
|
|
// ne sont pas toujours exposes par windows-sys selon les features.
|
|
// On utilise des appels FFI raw pour eviter d'ajouter des features.
|
|
#[link(name = "shcore")]
|
|
extern "system" {
|
|
fn SetProcessDpiAwareness(value: i32) -> i32;
|
|
}
|
|
#[link(name = "user32")]
|
|
extern "system" {
|
|
fn SetProcessDPIAware() -> i32;
|
|
}
|
|
unsafe {
|
|
// Tenter SetProcessDpiAwareness(2) = PROCESS_PER_MONITOR_DPI_AWARE
|
|
let hr = SetProcessDpiAwareness(2);
|
|
if hr != 0 {
|
|
// Fallback pour Windows < 8.1 : SetProcessDPIAware()
|
|
SetProcessDPIAware();
|
|
}
|
|
}
|
|
}
|
|
|
|
// Initialiser le logging
|
|
env_logger::Builder::from_env(
|
|
env_logger::Env::default().default_filter_or("info"),
|
|
)
|
|
.format_timestamp_secs()
|
|
.init();
|
|
|
|
let config = Config::from_env();
|
|
let config = Arc::new(config);
|
|
|
|
// Etat partage thread-safe
|
|
let state = AgentState::new();
|
|
|
|
// Banniere de demarrage
|
|
print_banner(&config);
|
|
|
|
// Handler Ctrl+C pour arret propre
|
|
install_ctrlc_handler(state.clone());
|
|
|
|
// Verifier que la capture d'ecran fonctionne
|
|
print!("[MAIN] Test de capture d'ecran... ");
|
|
match capture::screen_dimensions() {
|
|
Some((w, h)) => println!("OK ({}x{})", w, h),
|
|
None => {
|
|
println!("ECHEC");
|
|
eprintln!("[MAIN] ATTENTION : Capture d'ecran non disponible.");
|
|
eprintln!("[MAIN] Sur Linux sans display, les heartbeats seront desactives.");
|
|
}
|
|
}
|
|
|
|
// Thread 1 : Heartbeat loop
|
|
let hb_config = config.clone();
|
|
let hb_state = state.clone();
|
|
let _heartbeat_thread = thread::Builder::new()
|
|
.name("heartbeat".to_string())
|
|
.spawn(move || {
|
|
heartbeat_loop(&hb_config, &hb_state);
|
|
})
|
|
.expect("Impossible de demarrer le thread heartbeat");
|
|
|
|
// Thread 2 : Replay poll loop
|
|
let rp_config = config.clone();
|
|
let rp_state = state.clone();
|
|
let _replay_thread = thread::Builder::new()
|
|
.name("replay".to_string())
|
|
.spawn(move || {
|
|
replay::replay_poll_loop(&rp_config, &rp_state);
|
|
})
|
|
.expect("Impossible de demarrer le thread replay");
|
|
|
|
// Thread 3 : Capture HTTP server
|
|
let srv_port = config.capture_port;
|
|
let _server_thread = thread::Builder::new()
|
|
.name("capture-server".to_string())
|
|
.spawn(move || {
|
|
server::start_capture_server(srv_port);
|
|
})
|
|
.expect("Impossible de demarrer le thread serveur");
|
|
|
|
// Thread 4 : Health check (verification connexion serveur)
|
|
let hc_config = config.clone();
|
|
let hc_state = state.clone();
|
|
let _health_thread = thread::Builder::new()
|
|
.name("health-check".to_string())
|
|
.spawn(move || {
|
|
health_check_loop(&hc_config, &hc_state);
|
|
})
|
|
.expect("Impossible de demarrer le thread health check");
|
|
|
|
// Thread 5 : Recorder (capture evenements — inactif jusqu'a enregistrement)
|
|
let rec_config = config.clone();
|
|
let rec_state = state.clone();
|
|
let _recorder_rx = recorder::start_recorder(rec_config, rec_state);
|
|
|
|
// Thread 6 : Chat window (WebView2, a la demande)
|
|
let chat_config = config.clone();
|
|
let chat_state = state.clone();
|
|
chat::run_chat_thread(&chat_config, chat_state);
|
|
|
|
// Synchroniser les workflows disponibles depuis le serveur
|
|
let sync_config = config.clone();
|
|
let workflows = {
|
|
let client = Client::new();
|
|
network::fetch_workflows(&client, &sync_config)
|
|
};
|
|
if workflows.is_empty() {
|
|
println!("[MAIN] Aucun workflow disponible pour cette machine.");
|
|
} else {
|
|
println!(
|
|
"[MAIN] {} workflow(s) disponible(s) :",
|
|
workflows.len()
|
|
);
|
|
for wf in &workflows {
|
|
println!(
|
|
" - {} ({} noeuds, {} transitions)",
|
|
wf.name, wf.nodes, wf.edges
|
|
);
|
|
}
|
|
}
|
|
|
|
println!("\n[MAIN] Agent operationnel — tous les threads demarres.\n");
|
|
|
|
// Ouvrir Léa dans le navigateur disponible (mode app) au démarrage
|
|
#[cfg(target_os = "windows")]
|
|
{
|
|
let chat_url = config.chat_url();
|
|
if let Some(browser) = find_browser() {
|
|
let browser_name = if browser.contains("chrome") { "Chrome" }
|
|
else if browser.contains("edge") || browser.contains("Edge") { "Edge" }
|
|
else if browser.contains("brave") || browser.contains("Brave") { "Brave" }
|
|
else if browser.contains("firefox") || browser.contains("Firefox") { "Firefox" }
|
|
else { "navigateur" };
|
|
println!("[MAIN] Ouverture de Léa dans {}...", browser_name);
|
|
let _ = std::process::Command::new(&browser)
|
|
.args(&[
|
|
&format!("--app={}", chat_url),
|
|
"--window-size=600,800",
|
|
"--disable-extensions",
|
|
"--no-first-run",
|
|
])
|
|
.spawn();
|
|
} else {
|
|
println!("[MAIN] Aucun navigateur trouvé — ouvrez manuellement : {}", chat_url);
|
|
}
|
|
}
|
|
|
|
// Attente principale : Ctrl+C pour arrêter
|
|
println!("[MAIN] Appuyez sur Ctrl+C pour quitter.\n");
|
|
loop {
|
|
if !state.is_running() {
|
|
break;
|
|
}
|
|
thread::sleep(Duration::from_millis(500));
|
|
}
|
|
|
|
// Si on arrive ici, l'agent doit s'arreter
|
|
println!("\n[MAIN] Arret en cours...");
|
|
state.request_shutdown();
|
|
|
|
// Laisser le temps aux threads de se terminer
|
|
thread::sleep(Duration::from_millis(500));
|
|
|
|
println!("[MAIN] Agent arrete.");
|
|
}
|
|
|
|
/// Installe un handler Ctrl+C qui met l'etat a "arret demande".
|
|
fn install_ctrlc_handler(state: Arc<AgentState>) {
|
|
#[cfg(unix)]
|
|
{
|
|
let mut fds = [0i32; 2];
|
|
unsafe {
|
|
if libc::pipe(fds.as_mut_ptr()) != 0 {
|
|
eprintln!("[MAIN] Impossible de creer le pipe pour Ctrl+C");
|
|
return;
|
|
}
|
|
|
|
static mut WRITE_FD: i32 = -1;
|
|
WRITE_FD = fds[1];
|
|
|
|
// Sauvegarder un pointeur vers l'etat dans une static
|
|
// pour pouvoir y acceder depuis le handler
|
|
static mut STATE_PTR: *const AgentState = std::ptr::null();
|
|
STATE_PTR = Arc::as_ptr(&state);
|
|
|
|
extern "C" fn sigint_handler(_sig: i32) {
|
|
unsafe {
|
|
if !STATE_PTR.is_null() {
|
|
(*STATE_PTR)
|
|
.running
|
|
.store(false, std::sync::atomic::Ordering::SeqCst);
|
|
}
|
|
let buf = [1u8];
|
|
let _ = libc::write(WRITE_FD, buf.as_ptr() as *const _, 1);
|
|
}
|
|
}
|
|
|
|
libc::signal(libc::SIGINT, sigint_handler as *const () as libc::sighandler_t);
|
|
}
|
|
}
|
|
|
|
#[cfg(not(unix))]
|
|
{
|
|
// Sur Windows, le systray gere l'arret via le menu "Quitter"
|
|
// Le handler console est un bonus pour le mode headless
|
|
let _ = state;
|
|
}
|
|
}
|
|
|
|
/// Boucle de heartbeat : capture un screenshot toutes les N secondes
|
|
/// et l'envoie au serveur si l'ecran a change.
|
|
/// Applique le floutage des zones sensibles si active dans la config.
|
|
fn heartbeat_loop(config: &Config, state: &AgentState) {
|
|
let client = Client::new();
|
|
let session_id = config.bg_session_id();
|
|
let mut last_hash: u64 = 0;
|
|
let mut consecutive_errors: u32 = 0;
|
|
|
|
println!(
|
|
"[HEARTBEAT] Boucle permanente demarree (session={}, intervalle={}s)",
|
|
session_id, config.heartbeat_interval_s
|
|
);
|
|
|
|
while state.is_running() {
|
|
// Verifier l'arret d'urgence
|
|
if state
|
|
.emergency_stop
|
|
.load(std::sync::atomic::Ordering::SeqCst)
|
|
{
|
|
thread::sleep(Duration::from_secs(1));
|
|
continue;
|
|
}
|
|
|
|
// Capturer l'ecran
|
|
match capture::capture_screenshot() {
|
|
Some(img) => {
|
|
// Deduplication par hash perceptuel
|
|
let current_hash = capture::image_hash(&img);
|
|
if current_hash == last_hash {
|
|
thread::sleep(Duration::from_secs(config.heartbeat_interval_s));
|
|
continue;
|
|
}
|
|
last_hash = current_hash;
|
|
|
|
// Appliquer le floutage des zones sensibles si active
|
|
let final_img = if config.blur_sensitive {
|
|
blur::blur_sensitive_fields(&img)
|
|
} else {
|
|
img
|
|
};
|
|
|
|
// Encoder en JPEG
|
|
let jpeg_bytes =
|
|
capture::screenshot_to_jpeg_bytes(&final_img, config.jpeg_quality);
|
|
if jpeg_bytes.is_empty() {
|
|
thread::sleep(Duration::from_secs(config.heartbeat_interval_s));
|
|
continue;
|
|
}
|
|
|
|
// Envoyer au serveur
|
|
let success =
|
|
network::send_heartbeat(&client, config, &jpeg_bytes, &session_id);
|
|
if success {
|
|
consecutive_errors = 0;
|
|
} else {
|
|
consecutive_errors += 1;
|
|
if consecutive_errors == 1 || consecutive_errors % 12 == 0 {
|
|
eprintln!(
|
|
"[HEARTBEAT] {} erreur(s) consecutives",
|
|
consecutive_errors
|
|
);
|
|
}
|
|
}
|
|
}
|
|
None => {
|
|
thread::sleep(Duration::from_secs(config.heartbeat_interval_s * 2));
|
|
continue;
|
|
}
|
|
}
|
|
|
|
thread::sleep(Duration::from_secs(config.heartbeat_interval_s));
|
|
}
|
|
|
|
println!("[HEARTBEAT] Boucle arretee.");
|
|
}
|
|
|
|
/// Boucle de health check : verifie la connexion au serveur toutes les 30s.
|
|
/// Met a jour l'etat de connexion dans AgentState.
|
|
fn health_check_loop(config: &Config, state: &AgentState) {
|
|
let client = Client::new();
|
|
let check_interval = Duration::from_secs(30);
|
|
let timeout = Duration::from_secs(5);
|
|
|
|
println!("[HEALTH] Boucle health check demarree (intervalle=30s)");
|
|
|
|
while state.is_running() {
|
|
let url = format!("{}/stats", config.server_url);
|
|
let request = client.get(&url).timeout(timeout);
|
|
let connected = network::with_auth(request, config)
|
|
.send()
|
|
.map(|r| r.status().is_success())
|
|
.unwrap_or(false);
|
|
|
|
let was_connected = state.connected.load(std::sync::atomic::Ordering::SeqCst);
|
|
state.set_connected(connected);
|
|
|
|
// Notifier si le statut a change
|
|
if connected != was_connected {
|
|
notifications::connection_changed(connected);
|
|
}
|
|
|
|
thread::sleep(check_interval);
|
|
}
|
|
|
|
println!("[HEALTH] Boucle arretee.");
|
|
}
|
|
|
|
/// Affiche la banniere de demarrage.
|
|
fn print_banner(config: &Config) {
|
|
let meta = sysinfo::get_screen_metadata();
|
|
|
|
println!("======================================================");
|
|
println!(
|
|
" RPA Vision Agent v{} (Rust)",
|
|
config::AGENT_VERSION
|
|
);
|
|
println!(" Phases 1-5 — Parite complete");
|
|
println!("------------------------------------------------------");
|
|
println!(" Machine : {}", config.machine_id);
|
|
println!(" Serveur : {}", config.server_url);
|
|
println!(" Capture : port {}", config.capture_port);
|
|
println!(" Chat : port {}", config.chat_port);
|
|
println!(" Heartbeat : toutes les {}s", config.heartbeat_interval_s);
|
|
println!(" JPEG : qualite {}", config.jpeg_quality);
|
|
println!(" Floutage : {}", if config.blur_sensitive { "actif" } else { "inactif" });
|
|
println!(" Logs : retention {} jours", config.log_retention_days);
|
|
println!(" Auth : {}", if config.api_token.is_empty() { "aucune" } else { "Bearer token" });
|
|
println!(" Workflows : synchronisation au demarrage");
|
|
println!(
|
|
" Ecran : {}x{} @ {}% DPI",
|
|
meta.screen_resolution[0], meta.screen_resolution[1], meta.dpi_scale
|
|
);
|
|
println!(
|
|
" Moniteur : #{} ({})",
|
|
meta.monitor_index,
|
|
if meta.monitor_index == 0 { "principal" } else { "secondaire" }
|
|
);
|
|
println!("======================================================");
|
|
println!();
|
|
println!(" [IA] Cet agent utilise l'intelligence artificielle.");
|
|
println!(" Article 50 du Reglement europeen sur l'IA.");
|
|
println!();
|
|
}
|