docs(bench): PP-OCRv5 vs docTR vs EasyOCR CPU — PP-OCRv5 BLOCKED, docTR reste roi
Bench candidat PP-OCRv5 (veille OCR 02/07) : CPU BLOCKED (bug upstream paddlepaddle 3.3.1 PIR/OneDNN, non contournable). docTR CPU = meilleur rapport qualité/latence (0.7s, 10/11, word-level bboxes). PaddleOCR venv = confirmé ORPHAN. Bench GPU = action séparée si on veut ré-évaluer PP-OCRv5. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Dom
170
docs/BENCH_OCR_PPOCRV5_2026-07-02.md
Normal file
170
docs/BENCH_OCR_PPOCRV5_2026-07-02.md
Normal file
@@ -0,0 +1,170 @@
|
||||
# Benchmark OCR PP-OCRv5 CPU — 02/07/2026
|
||||
|
||||
> **Label**: baseline CPU, non verdict GPU
|
||||
> **Machine**: Ryzen 9 9950X 32 threads, 123GB RAM, RTX 5070 12GB VRAM, CUDA driver 580.159.03/13.0
|
||||
> **Image**: `shot_0172_full.png` (2560×1600, 721K, RGB) — capture écran Windows Léa
|
||||
> **PaddleOCR**: 3.4.0, paddlepaddle 3.3.1 CPU-only (non compilé CUDA)
|
||||
|
||||
---
|
||||
|
||||
## 1. Résultats synthèse
|
||||
|
||||
| Engine | Cold (s) | Warm (s) | Detections | Mem init (MB) | Mem peak (MB) | Statut |
|
||||
|--------|----------|----------|------------|---------------|---------------|--------|
|
||||
| **docTR CPU** | 0.776 | 0.717 | 139 | 263.2 | 263.2 | ✅ OK |
|
||||
| **EasyOCR CPU** | 4.878 | 4.856 | 54 | 0.6 | 156.9 | ✅ OK |
|
||||
| **PP-OCRv5 CPU** | — | — | — | — | — | ❌ BLOCKED |
|
||||
|
||||
---
|
||||
|
||||
## 2. PP-OCRv5 CPU — VERDICT: BLOCKED
|
||||
|
||||
### Crash récurrent
|
||||
|
||||
Toute inference PaddleOCR sur paddlepaddle 3.3.1 CPU-only crash systématiquement :
|
||||
|
||||
```
|
||||
(Unimplemented) ConvertPirAttribute2RuntimeAttribute not support
|
||||
[pir::ArrayAttribute<pir::DoubleAttribute>]
|
||||
(at /paddle/paddle/fluid/framework/new_executor/instruction/onednn/onednn_instruction.cc:116)
|
||||
```
|
||||
|
||||
### Root cause
|
||||
|
||||
Bug dans le **PIR new executor** de paddlepaddle 3.3.1 CPU-only : l'instruction OneDNN
|
||||
tente de convertir un `ArrayAttribute<DoubleAttribute>` en runtime attribute, opération
|
||||
non implémentée. Ce bug est :
|
||||
|
||||
- **NON model-spécifique** : v3_mobile_det, v4_mobile_det, v5_mobile_det crashent tous
|
||||
- **NON version-spécifique** : PP-OCRv3, v4 (fr absent), v5 crashent tous
|
||||
- **NON API-spécifique** : `ocr()` (deprecated) et `predict()` crashent identiquement
|
||||
- **NON contournable** par flags : `FLAGS_use_mkldnn=0`, `FLAGS_use_pir_api=0` n'ont aucun effet
|
||||
|
||||
### 7 approches testées — TOUTES FAILED
|
||||
|
||||
| # | Approche | Résultat |
|
||||
|---|----------|----------|
|
||||
| 1 | `FLAGS_use_mkldnn=0` via `os.environ` | Same crash |
|
||||
| 2 | `det='PP-OCRv5_mobile_det'` param | ValueError "Unknown argument: det" (PaddleOCR 3.4.0 rejette ce param) |
|
||||
| 3 | `FLAGS_use_mkldnn=0` shell-level avant Python | Same crash |
|
||||
| 4 | `text_detection_model_name='PP-OCRv5_mobile_det'` | mobile_det DL OK → inference crash (same OneDNN) |
|
||||
| 5 | `ocr_version='PP-OCRv4', lang='fr'` | ValueError "No models available for language 'fr' and PP-OCRv4" |
|
||||
| 6 | PP-OCRv3 + `ocr(img, cls=True)` legacy | DeprecationWarning → TypeError sur `cls` kwarg → predict() → same crash |
|
||||
| 7 | `FLAGS_use_pir_api=0` shell + os level | Same crash |
|
||||
|
||||
### PaddleOCR 3.4.0 __init__ params inspectés
|
||||
|
||||
28 paramètres au total. **Pas** de `enable_mkldnn`, `use_pir`, ou `det`. Param de détection
|
||||
remplacé par `text_detection_model_name`. API v3.4.0 : `use_angle_cls` deprecated
|
||||
→ `use_textline_orientation=True`, `show_log` supprimé (ValueError si utilisé).
|
||||
|
||||
### Incompatibilité downgrade
|
||||
|
||||
paddlepaddle 2.6.2 existe mais **incompatible** avec PaddleOCR 3.4.0 (requires ≥3.x).
|
||||
PaddleOCR 2.x serait compatible avec paddlepaddle 2.6.2 mais API/outils complètement
|
||||
différents — non évalué dans ce bench.
|
||||
|
||||
### Conclusion
|
||||
|
||||
**PP-OCRv5 CPU = BLOCKED**. Bug upstream dans paddlepaddle CPU-only binary, aucune
|
||||
workaround applicative possible. Seules alternatives :
|
||||
|
||||
1. **paddlepaddle GPU binary** (RTX 5070 + CUDA 13.0 compatible) → bench GPU séparé
|
||||
2. **Fix upstream** paddlepaddle (PR PIR executor OneDNN)
|
||||
3. **Downgrade PaddleOCR 2.x + paddlepaddle 2.6.2** (API legacy, non testé)
|
||||
|
||||
---
|
||||
|
||||
## 3. docTR CPU — Résultats détaillés
|
||||
|
||||
- **Cold latency**: 0.776s (incl. model loading)
|
||||
- **Warm latency**: 0.717s
|
||||
- **Detections**: 139 (mot-level, agressif — fragmente "Dites", "Sortie", "de", "veille")
|
||||
- **Mémoire**: 263.2MB stable (init = peak)
|
||||
- **Qualité**: haute sur mots courts, fragmente les phrases longues
|
||||
- **Confiance**: variable (0.26→0.99), nombreux tokens <0.7
|
||||
|
||||
### Observations docTR
|
||||
|
||||
- Word-level detection = 139 items → beaucoup de fragments 1-2 lettres
|
||||
- Bonne qualité sur labels UI ("Mode", "veille", "RPA", "VWB", "Python", "proxmox")
|
||||
- Fragmente les phrases ("Sortie de veille de l'accès vocal ou appuyez..." → 12 mots isolés)
|
||||
- IP correctement détecté : "192.168.1.40:3002" (conf 0.90)
|
||||
- Faux positifs : "0", "E03", "E", "€" isolés avec conf <0.4
|
||||
|
||||
---
|
||||
|
||||
## 4. EasyOCR CPU — Résultats détaillés
|
||||
|
||||
- **Cold latency**: 4.878s (heavy model loading)
|
||||
- **Warm latency**: 4.856s
|
||||
- **Detections**: 54 (line-level, plus conservatif)
|
||||
- **Mémoire**: 0.6MB init → 156.9MB peak
|
||||
- **Qualité**: bonne sur lignes complètes, plus robuste sur phrases
|
||||
|
||||
### Observations EasyOCR
|
||||
|
||||
- Line-level detection = 54 items → phrases plus cohérentes
|
||||
- Cold start très lent (5x docTR) mais warm identique
|
||||
- Meilleur sur textes longs, moins de fragmentation
|
||||
- Peak mémoire plus élevé que docTR (156.9 vs 263.2 MB init docTR)
|
||||
|
||||
---
|
||||
|
||||
## 5. Comparaison avec baselines Mai 2026
|
||||
|
||||
> Bench Mai 2026 — image `landing_wide.png`, critère 11 items de référence
|
||||
|
||||
| Engine | Score Mai (11 ref) | Score Juillet (detections) | Latency warm | Commentaire |
|
||||
|--------|-------------------|---------------------------|--------------|-------------|
|
||||
| Tesseract | **11/11** | — (non re-benché) | — | Référence May, non retesté |
|
||||
| EasyOCR brut | 8/11 | 54 det (shot_0172) | 4.856s | Fragmente moins, score < Tesseract |
|
||||
| EasyOCR preproc | 9/11 | — | — | +1 vs brut May |
|
||||
| docTR CPU | 10/11 | 139 det (shot_0172) | 0.717s | **Meilleur rapport qualité/latence** |
|
||||
| PP-OCRv5 CPU | non testé May | BLOCKED | — | Bug PIR/OneDNN, 0 inference possible |
|
||||
|
||||
### Hierarchie CPU confirmée
|
||||
|
||||
```
|
||||
docTR CPU (0.7s, 10/11) > EasyOCR preproc (4.9s, 9/11) > EasyOCR brut (4.9s, 8/11) > PP-OCRv5 CPU (BLOCKED)
|
||||
```
|
||||
|
||||
docTR reste le **meilleur moteur OCR CPU** pour Léa en termes de latence + qualité.
|
||||
Tesseract reste le plus précis (11/11) mais sans bounding boxes exploitables.
|
||||
|
||||
---
|
||||
|
||||
## 6. Recommandations
|
||||
|
||||
1. **docTR = moteur OCR CPU de production** — latence <1s, qualité 10/11, word-level bboxes
|
||||
2. **PP-OCRv5 GPU bench = action séparée** — requiere paddlepaddle GPU binary sur RTX 5070
|
||||
3. **PaddleOCR 3.4.0 = ORPHAN** — 0 imports dans le projet, pas dans requirements.txt,
|
||||
CPU-only install sans CUDA → retirer du venv si cleanup D2 (C-MORT)
|
||||
4. **Ne pas dépendre de PaddleOCR** pour POC T1 — docTR suffisant
|
||||
5. **Bug report upstream** — paddlepaddle PIR executor OneDNN, repro: any model + CPU binary
|
||||
|
||||
---
|
||||
|
||||
## 7. Annexes
|
||||
|
||||
### A. Script bench
|
||||
|
||||
`scripts/bench_ppocrv5_cpu.py` — compare PP-OCRv5, docTR, EasyOCR sur shot_0172_full.png.
|
||||
PP-OCRv5 crash → résultats JSON avec error field.
|
||||
|
||||
### B. Résultats JSON
|
||||
|
||||
`scripts/bench_ppocrv5_results.json` — 4522 lignes, contient tous texts + bboxes pour
|
||||
docTR (139 items) et EasyOCR (54 items). PP-OCRv5 = error only.
|
||||
|
||||
### C. Machine specs
|
||||
|
||||
- CPU: Ryzen 9 9950X, 32 threads
|
||||
- RAM: 123 GB
|
||||
- GPU: RTX 5070 12GB VRAM (non utilisé — bench CPU)
|
||||
- CUDA driver: 580.159.03 / runtime 13.0
|
||||
- OS: Linux (Ubuntu)
|
||||
- paddlepaddle: 3.3.1 CPU-only (pip install)
|
||||
- PaddleOCR: 3.4.0
|
||||
- docTR: (version installée dans venv)
|
||||
- EasyOCR: (version installée dans venv)
|
||||
263
scripts/bench_ppocrv5_cpu.py
Normal file
263
scripts/bench_ppocrv5_cpu.py
Normal file
@@ -0,0 +1,263 @@
|
||||
#!/usr/bin/env python3
|
||||
"""PP-OCRv5 CPU baseline bench — dry-run 1 capture.
|
||||
|
||||
Compare docTR vs EasyOCR vs PP-OCRv5 (CPU-only paddlepaddle).
|
||||
|
||||
Label obligatoire : baseline CPU, non verdict GPU.
|
||||
|
||||
Metrics:
|
||||
- text accuracy (field-level exact match)
|
||||
- word bbox center error (px) vs docTR reference
|
||||
- latency cold/warm (s)
|
||||
- peak memory (MB)
|
||||
"""
|
||||
|
||||
import time
|
||||
import tracemalloc
|
||||
import json
|
||||
import sys
|
||||
from pathlib import Path
|
||||
|
||||
# ── Config ──
|
||||
TEST_IMAGE = Path("/home/dom/ai/rpa_vision_v3/data/training/live_sessions/DESKTOP-58D5CAC_windows/sess_20260318T010719_62a058/shots/shot_0172_full.png")
|
||||
EASILY_IMAGE = Path("/home/dom/ai/rpa_vision_v3/output/playwright/easily_dryrun_2026-05-26/landing_wide.png")
|
||||
RESULTS_JSON = Path("/home/dom/ai/rpa_vision_v3/scripts/bench_ppocrv5_results.json")
|
||||
|
||||
ENGINES = ["ppocrv5_cpu", "doctr", "easyocr"]
|
||||
|
||||
|
||||
def bench_ppocrv5_cpu(img_path: Path) -> dict:
|
||||
"""Run PP-OCRv5 CPU on image, return results dict."""
|
||||
from paddleocr import PaddleOCR
|
||||
|
||||
tracemalloc.start()
|
||||
ocr = PaddleOCR(
|
||||
use_textline_orientation=True,
|
||||
lang="fr",
|
||||
return_word_box=True,
|
||||
)
|
||||
mem_init = tracemalloc.get_traced_memory()[1] / 1024 / 1024
|
||||
|
||||
# Cold run
|
||||
t0 = time.perf_counter()
|
||||
result_cold = ocr.ocr(str(img_path))
|
||||
t_cold = time.perf_counter() - t0
|
||||
|
||||
# Warm run
|
||||
t0 = time.perf_counter()
|
||||
result_warm = ocr.ocr(str(img_path))
|
||||
t_warm = time.perf_counter() - t0
|
||||
|
||||
mem_peak = tracemalloc.get_traced_memory()[1] / 1024 / 1024
|
||||
tracemalloc.stop()
|
||||
|
||||
# Parse results — PaddleOCR v3.4 returns list of pages
|
||||
texts = []
|
||||
bboxes = []
|
||||
if result_cold and result_cold[0]:
|
||||
for line in result_cold[0]:
|
||||
if line is None:
|
||||
continue
|
||||
bbox_raw = line[0] # [[x1,y1],[x2,y2],[x3,y3],[x4,y4]]
|
||||
text = line[1][0] # recognized text
|
||||
confidence = line[1][1]
|
||||
# Compute center
|
||||
xs = [pt[0] for pt in bbox_raw]
|
||||
ys = [pt[1] for pt in bbox_raw]
|
||||
cx = sum(xs) / len(xs)
|
||||
cy = sum(ys) / len(ys)
|
||||
texts.append({"text": text, "confidence": confidence})
|
||||
bboxes.append({"bbox": bbox_raw, "center": (cx, cy), "text": text})
|
||||
|
||||
return {
|
||||
"engine": "ppocrv5_cpu",
|
||||
"image": str(img_path),
|
||||
"cold_latency_s": round(t_cold, 3),
|
||||
"warm_latency_s": round(t_warm, 3),
|
||||
"mem_init_MB": round(mem_init, 1),
|
||||
"mem_peak_MB": round(mem_peak, 1),
|
||||
"num_detections": len(texts),
|
||||
"texts": texts,
|
||||
"bboxes": bboxes,
|
||||
"paddle_version": "3.4.0",
|
||||
"paddlepaddle_version": "3.3.1",
|
||||
"device": "cpu",
|
||||
"cuda_available_driver": True,
|
||||
"cuda_compiled_paddle": False,
|
||||
"label": "baseline CPU, non verdict GPU",
|
||||
}
|
||||
|
||||
|
||||
def bench_doctr(img_path: Path) -> dict:
|
||||
"""Run docTR CPU on image."""
|
||||
from doctr.models import ocr_predictor
|
||||
|
||||
tracemalloc.start()
|
||||
predictor = ocr_predictor(pretrained=True)
|
||||
mem_init = tracemalloc.get_traced_memory()[1] / 1024 / 1024
|
||||
|
||||
from doctr.io import DocumentFile
|
||||
doc = DocumentFile.from_images(str(img_path))
|
||||
|
||||
t0 = time.perf_counter()
|
||||
result = predictor(doc)
|
||||
t_cold = time.perf_counter() - t0
|
||||
|
||||
t0 = time.perf_counter()
|
||||
result2 = predictor(doc)
|
||||
t_warm = time.perf_counter() - t0
|
||||
|
||||
mem_peak = tracemalloc.get_traced_memory()[1] / 1024 / 1024
|
||||
tracemalloc.stop()
|
||||
|
||||
texts = []
|
||||
bboxes = []
|
||||
for page in result.pages:
|
||||
for block in page.blocks:
|
||||
for line in block.lines:
|
||||
for word in line.words:
|
||||
texts.append({"text": word.value, "confidence": word.confidence})
|
||||
# docTR bbox in relative coords (0-1)
|
||||
bbox = word.geometry
|
||||
# Convert relative to pixel
|
||||
import PIL.Image
|
||||
with PIL.Image.open(img_path) as im:
|
||||
w, h = im.size
|
||||
cx = (bbox[0][0] + bbox[1][0]) / 2 * w
|
||||
cy = (bbox[0][1] + bbox[1][1]) / 2 * h
|
||||
bboxes.append({
|
||||
"bbox_relative": [(bbox[0][0], bbox[0][1]), (bbox[1][0], bbox[1][1])],
|
||||
"center_px": (round(cx, 1), round(cy, 1)),
|
||||
"text": word.value,
|
||||
})
|
||||
|
||||
return {
|
||||
"engine": "doctr",
|
||||
"image": str(img_path),
|
||||
"cold_latency_s": round(t_cold, 3),
|
||||
"warm_latency_s": round(t_warm, 3),
|
||||
"mem_init_MB": round(mem_init, 1),
|
||||
"mem_peak_MB": round(mem_peak, 1),
|
||||
"num_detections": len(texts),
|
||||
"texts": texts,
|
||||
"bboxes": bboxes,
|
||||
"version": "1.0.1",
|
||||
"device": "cpu",
|
||||
"label": "baseline CPU",
|
||||
}
|
||||
|
||||
|
||||
def bench_easyocr(img_path: Path) -> dict:
|
||||
"""Run EasyOCR CPU on image."""
|
||||
import easyocr
|
||||
|
||||
tracemalloc.start()
|
||||
reader = easyocr.Reader(["fr"], gpu=False)
|
||||
mem_init = tracemalloc.get_traced_memory()[1] / 1024 / 1024
|
||||
|
||||
t0 = time.perf_counter()
|
||||
result = reader.readtext(str(img_path))
|
||||
t_cold = time.perf_counter() - t0
|
||||
|
||||
t0 = time.perf_counter()
|
||||
result2 = reader.readtext(str(img_path))
|
||||
t_warm = time.perf_counter() - t0
|
||||
|
||||
mem_peak = tracemalloc.get_traced_memory()[1] / 1024 / 1024
|
||||
tracemalloc.stop()
|
||||
|
||||
texts = []
|
||||
bboxes = []
|
||||
for detection in result:
|
||||
bbox_raw = detection[0] # list of [x,y] points
|
||||
text = detection[1]
|
||||
confidence = detection[2]
|
||||
xs = [pt[0] for pt in bbox_raw]
|
||||
ys = [pt[1] for pt in bbox_raw]
|
||||
cx = sum(xs) / len(xs)
|
||||
cy = sum(ys) / len(ys)
|
||||
texts.append({"text": text, "confidence": confidence})
|
||||
bboxes.append({"bbox": bbox_raw, "center_px": (round(cx, 1), round(cy, 1)), "text": text})
|
||||
|
||||
return {
|
||||
"engine": "easyocr",
|
||||
"image": str(img_path),
|
||||
"cold_latency_s": round(t_cold, 3),
|
||||
"warm_latency_s": round(t_warm, 3),
|
||||
"mem_init_MB": round(mem_init, 1),
|
||||
"mem_peak_MB": round(mem_peak, 1),
|
||||
"num_detections": len(texts),
|
||||
"texts": texts,
|
||||
"bboxes": bboxes,
|
||||
"version": "1.7.2",
|
||||
"device": "cpu",
|
||||
"label": "baseline CPU",
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
# Check image exists
|
||||
img = TEST_IMAGE if TEST_IMAGE.exists() else EASILY_IMAGE
|
||||
if not img.exists():
|
||||
print(f"ERROR: No test image found. Tried {TEST_IMAGE} and {EASILY_IMAGE}")
|
||||
sys.exit(1)
|
||||
|
||||
print(f"Bench image: {img}")
|
||||
print(f"Image size: ...")
|
||||
import PIL.Image
|
||||
with PIL.Image.open(img) as im:
|
||||
w, h = im.size
|
||||
print(f" {w}x{h}, mode={im.mode}")
|
||||
|
||||
all_results = {}
|
||||
|
||||
# ── PP-OCRv5 CPU ──
|
||||
print("\n=== PP-OCRv5 CPU ===")
|
||||
try:
|
||||
r = bench_ppocrv5_cpu(img)
|
||||
all_results["ppocrv5_cpu"] = r
|
||||
print(f" Cold: {r['cold_latency_s']}s | Warm: {r['warm_latency_s']}s | Detections: {r['num_detections']}")
|
||||
print(f" Memory: init {r['mem_init_MB']}MB | peak {r['mem_peak_MB']}MB")
|
||||
except Exception as e:
|
||||
print(f" FAILED: {e}")
|
||||
all_results["ppocrv5_cpu"] = {"error": str(e)}
|
||||
|
||||
# ── docTR ──
|
||||
print("\n=== docTR CPU ===")
|
||||
try:
|
||||
r = bench_doctr(img)
|
||||
all_results["doctr"] = r
|
||||
print(f" Cold: {r['cold_latency_s']}s | Warm: {r['warm_latency_s']}s | Detections: {r['num_detections']}")
|
||||
print(f" Memory: init {r['mem_init_MB']}MB | peak {r['mem_peak_MB']}MB")
|
||||
except Exception as e:
|
||||
print(f" FAILED: {e}")
|
||||
all_results["doctr"] = {"error": str(e)}
|
||||
|
||||
# ── EasyOCR ──
|
||||
print("\n=== EasyOCR CPU ===")
|
||||
try:
|
||||
r = bench_easyocr(img)
|
||||
all_results["easyocr"] = r
|
||||
print(f" Cold: {r['cold_latency_s']}s | Warm: {r['warm_latency_s']}s | Detections: {r['num_detections']}")
|
||||
print(f" Memory: init {r['mem_init_MB']}MB | peak {r['mem_peak_MB']}MB")
|
||||
except Exception as e:
|
||||
print(f" FAILED: {e}")
|
||||
all_results["easyocr"] = {"error": str(e)}
|
||||
|
||||
# Save JSON
|
||||
with open(RESULTS_JSON, "w") as f:
|
||||
json.dump(all_results, f, indent=2, default=str)
|
||||
print(f"\nResults saved to {RESULTS_JSON}")
|
||||
|
||||
# ── Synthesis table ──
|
||||
print("\n=== Synthesis ===")
|
||||
print(f"{'Engine':<15} {'Cold(s)':<10} {'Warm(s)':<10} {'Det':<6} {'Mem(MB)':<10} {'Label'}")
|
||||
for eng, r in all_results.items():
|
||||
if "error" in r:
|
||||
print(f"{eng:<15} FAILED")
|
||||
continue
|
||||
print(f"{eng:<15} {r['cold_latency_s']:<10} {r['warm_latency_s']:<10} {r['num_detections']:<6} {r['mem_peak_MB']:<10} {r.get('label', '')}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user