757432ee19
1527 lignes Rust, compile sans warnings, testé sur Linux. - Capture d'écran (xcap) + JPEG base64 + hash dedup - Heartbeat toutes les 5s vers streaming server - Poll replay + exécution actions (clic, frappe, combos) - Serveur HTTP port 5006 (capture, health, file-action) - Compatible avec le streaming server Python existant Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
131 lines
4.2 KiB
Markdown
131 lines
4.2 KiB
Markdown
# RPA Vision Agent (Rust) — Phase 1
|
|
|
|
Agent headless pour RPA Vision V3, ecrit en Rust.
|
|
Capture des screenshots, les envoie au serveur streaming, poll les actions de replay et les execute.
|
|
|
|
Equivalent fonctionnel de `agent_v0/agent_v1/` (Python) mais en un seul executable sans dependance.
|
|
|
|
## Fonctionnalites (Phase 1)
|
|
|
|
- **Heartbeat** : capture l'ecran toutes les 5s, encode en JPEG, envoie au serveur avec deduplication par hash perceptuel
|
|
- **Replay** : poll GET /replay/next toutes les secondes, execute les actions (click, type, key_combo, scroll, wait), rapporte le resultat avec screenshot post-action
|
|
- **Serveur de capture** : mini serveur HTTP sur port 5006 pour screenshots a la demande (GET /capture, GET /health, POST /file-action)
|
|
- **Configuration** : via variables d'environnement ou valeurs par defaut
|
|
|
|
## Build
|
|
|
|
### Linux (pour tests)
|
|
|
|
```bash
|
|
# Pre-requis systeme (Ubuntu/Debian)
|
|
sudo apt install libpipewire-0.3-dev libclang-dev libgbm-dev libxdo-dev
|
|
|
|
# Build debug
|
|
cargo build
|
|
|
|
# Build release optimise
|
|
cargo build --release
|
|
```
|
|
|
|
### Cross-compilation vers Windows
|
|
|
|
```bash
|
|
# Option A : cargo-xwin (recommande, produit un .exe MSVC)
|
|
cargo install cargo-xwin
|
|
rustup target add x86_64-pc-windows-msvc
|
|
cargo xwin build --target x86_64-pc-windows-msvc --release
|
|
|
|
# Option B : MinGW (plus simple)
|
|
rustup target add x86_64-pc-windows-gnu
|
|
sudo apt install gcc-mingw-w64-x86-64
|
|
cargo build --release --target x86_64-pc-windows-gnu
|
|
|
|
# Option C : cross (Docker)
|
|
cargo install cross
|
|
cross build --release --target x86_64-pc-windows-msvc
|
|
```
|
|
|
|
Le binaire release se trouve dans `target/release/rpa-agent` (Linux) ou
|
|
`target/x86_64-pc-windows-msvc/release/rpa-agent.exe` (Windows).
|
|
|
|
## Configuration
|
|
|
|
| Variable | Defaut | Description |
|
|
|---|---|---|
|
|
| `RPA_SERVER_URL` | `http://localhost:5005/api/v1` | URL du serveur streaming |
|
|
| `RPA_MACHINE_ID` | `{hostname}_{os}` | Identifiant de la machine |
|
|
| `RPA_CAPTURE_PORT` | `5006` | Port du serveur de capture |
|
|
| `RPA_HEARTBEAT_INTERVAL` | `5` | Intervalle heartbeat (secondes) |
|
|
| `RPA_JPEG_QUALITY` | `85` | Qualite JPEG (1-100) |
|
|
|
|
## Execution
|
|
|
|
```bash
|
|
# Avec les defauts (serveur local)
|
|
./target/release/rpa-agent
|
|
|
|
# Vers un serveur distant
|
|
RPA_SERVER_URL=http://192.168.1.10:5005/api/v1 ./target/release/rpa-agent
|
|
|
|
# Avec un identifiant machine specifique
|
|
RPA_MACHINE_ID=pc_bureau_windows ./target/release/rpa-agent
|
|
```
|
|
|
|
## API du serveur de capture (port 5006)
|
|
|
|
### GET /capture
|
|
Retourne un screenshot frais en JSON :
|
|
```json
|
|
{
|
|
"image": "<base64 JPEG>",
|
|
"width": 1920,
|
|
"height": 1080,
|
|
"format": "jpeg",
|
|
"source": "rust_agent",
|
|
"capture_ms": 42
|
|
}
|
|
```
|
|
|
|
### GET /health
|
|
```json
|
|
{"status": "ok", "agent": "rust", "version": "0.1.0-rust"}
|
|
```
|
|
|
|
### POST /file-action
|
|
Actions fichiers sur la machine locale :
|
|
```json
|
|
{"action": "file_list_dir", "params": {"path": "C:\\Users\\dom\\Documents"}}
|
|
{"action": "file_create_dir", "params": {"path": "C:\\Users\\dom\\Documents\\tri"}}
|
|
{"action": "file_move", "params": {"source": "...", "destination": "..."}}
|
|
{"action": "file_copy", "params": {"source": "...", "destination": "..."}}
|
|
{"action": "file_sort_by_ext", "params": {"source_dir": "C:\\Users\\dom\\Downloads"}}
|
|
```
|
|
|
|
## Architecture
|
|
|
|
```
|
|
src/
|
|
├── main.rs — Point d'entree, 3 threads (heartbeat, replay, serveur)
|
|
├── config.rs — Configuration (env vars + defauts)
|
|
├── capture.rs — Capture ecran (xcap), encodage JPEG, hash perceptuel
|
|
├── network.rs — Client HTTP (heartbeat, poll replay, rapport resultat)
|
|
├── replay.rs — Boucle de polling replay avec backoff
|
|
├── executor.rs — Execution actions (click, type, key_combo, scroll, wait)
|
|
└── server.rs — Mini serveur HTTP port 5006 (/capture, /health, /file-action)
|
|
```
|
|
|
|
## Compatibilite serveur
|
|
|
|
Cet agent est compatible avec le serveur streaming existant (`agent_v0/server_v1/api_stream.py`, port 5005).
|
|
|
|
Endpoints utilises :
|
|
- `POST /api/v1/traces/stream/image` — envoi heartbeat screenshot
|
|
- `GET /api/v1/traces/stream/replay/next` — poll action replay
|
|
- `POST /api/v1/traces/stream/replay/result` — rapport resultat replay
|
|
|
|
## Phases suivantes
|
|
|
|
- **Phase 2** : Systray + notifications (tray-icon, winrt-notification)
|
|
- **Phase 3** : Fenetre de chat (wry/WebView2)
|
|
- **Phase 4** : Parite complete (floutage, capture evenements rdev)
|