docs: add POC specs, handoffs, and research notes
This commit is contained in:
Dom
313
docs/POC/REQUIREMENTS_DGX_AARCH64_DRAFT_2026-06-01.md
Normal file
313
docs/POC/REQUIREMENTS_DGX_AARCH64_DRAFT_2026-06-01.md
Normal file
@@ -0,0 +1,313 @@
|
||||
# requirements-dgx-aarch64.txt — DRAFT pour relecture Codex/Dom
|
||||
|
||||
**Statut** : DRAFT — pas encore appliqué. Aucun changement au `requirements.txt` réel.
|
||||
**Date** : 2026-06-01
|
||||
**Cible** : DGX Spark, Ubuntu 24.04 ARM64, CUDA 13 système, GB10 (sm_121), container Docker headless serveur.
|
||||
**Source de vérité** : `docs/POC/PORTAGE_DGX_SPARK_2026-05-28.md` §5.
|
||||
**Base** : `requirements.txt` actuel = 180 paquets — bench x86 `venv_v3/`.
|
||||
|
||||
---
|
||||
|
||||
## Règles appliquées
|
||||
|
||||
- PyTorch : `torch`, `torchvision` désépinglés, installés via `--index-url https://download.pytorch.org/whl/cu128` (binaire sm_120 compat sm_121).
|
||||
- Wheels GPU NVIDIA (`nvidia-*-cu12`, 15 paquets) : **supprimés**, tirés transitivement par torch cu128 ARM.
|
||||
- Capture / GUI / X11 : **supprimés** — container serveur headless ; l'agent capture tourne sur poste Linux client séparé.
|
||||
- `onnx==1.20.1` : **supprimé** (audit dépendances : fantôme transitif).
|
||||
- Reste : wheels neutres ARM/CPU **gardés tels quels** ; wheels torch-dependent **désépinglés** pour laisser pip résoudre.
|
||||
|
||||
---
|
||||
|
||||
## 1. GARDÉ TEL QUEL (paquets neutres ARM/CPU)
|
||||
|
||||
Wheels pur Python ou wheels manylinux2014_aarch64 / musllinux_aarch64 publiés sur PyPI. §5 RAS pour les binaires explicitement listés.
|
||||
|
||||
- **Web/API** : `fastapi`, `Flask`, `Flask-Caching`, `Flask-Cors`, `Flask-Migrate`, `Flask-SocketIO`, `Flask-SQLAlchemy`, `starlette`, `uvicorn`, `uvloop` (§5), `httptools`, `httpcore`, `httpx`, `h11`, `websockets`, `wsproto`, `simple-websocket`, `python-engineio`, `python-socketio`, `python-multipart`, `Werkzeug`, `Jinja2`, `itsdangerous`, `blinker`, `click`, `MarkupSafe`, `cachelib`, `watchfiles`, `anyio`, `bidict`
|
||||
- **DB/migrations** : `SQLAlchemy`, `alembic`, `Mako`, `greenlet`, `redis`
|
||||
- **ML/vision CPU** : `faiss-cpu` (§5), `opencv-python` (§5), `pillow`, `numpy`, `scipy`, `scikit-learn`, `matplotlib`, `contourpy`, `fonttools`, `kiwisolver`, `cycler`, `pyparsing`, `ml_dtypes`, `networkx`, `sympy`, `mpmath`, `joblib`, `threadpoolctl`, `RapidFuzz`, `shapely` (§5), `pyclipper` (§5), `h5py` (§5)
|
||||
- **PDF/docs** : `pypdfium2` (§5), `lxml` (§5), `python-docx`, `openpyxl`, `et_xmlfile`, `defusedxml`, `anyascii`, `langdetect`, `ftfy`, `wcwidth`, `regex`
|
||||
- **Ollama** : `ollama` (client HTTP pur Python)
|
||||
- **Validation/config** : `pydantic`, `pydantic_core`, `annotated-doc`, `annotated-types`, `typing-inspection`, `typing_extensions`, `validators`, `marshmallow`, `jsonschema`, `jsonschema-specifications`, `referencing`, `rpds-py`, `attrs`, `PyYAML`, `python-dotenv`
|
||||
- **HTTP/réseau** : `aiohttp`, `aiohappyeyeballs`, `aiosignal`, `frozenlist`, `multidict`, `propcache`, `yarl`, `requests`, `urllib3`, `charset-normalizer`, `idna`, `certifi`
|
||||
- **Crypto/utils** : `cryptography`, `cffi`, `pycparser`, `filelock`, `fsspec`, `packaging`, `platformdirs`, `pathspec`, `six`, `python-dateutil`, `sortedcontainers`, `tqdm`, `Pygments`
|
||||
- **Monitoring** : `prometheus_client`, `psutil`, `pynvml`, `nvidia-ml-py` (bindings NVML pur Python, pas un wheel CUDA toolkit)
|
||||
- **Protobuf** : `protobuf` (§5)
|
||||
- **Dev/tests** : `pytest`, `pytest-asyncio`, `pytest-cov`, `pytest-flask`, `pytest-mock`, `hypothesis`, `coverage`, `iniconfig`, `pluggy`, `black`, `flake8`, `mypy`, `mypy_extensions`, `mccabe`, `pycodestyle`, `pyflakes`
|
||||
|
||||
## 2. GARDÉ MAIS DÉSÉPINGLÉ (laisser pip résoudre avec torch cu128 ARM)
|
||||
|
||||
| Paquet | Version actuelle x86 | Raison |
|
||||
|--------|----------------------|--------|
|
||||
| `torch` | `2.9.1` | installé via index PyTorch cu128 ARM (étape 1 ci-dessous) |
|
||||
| `torchvision` | `0.24.1` | idem, doit matcher torch cu128 ARM |
|
||||
| `transformers` | `4.57.3` | dépend de torch — laisser résoudre |
|
||||
| `accelerate` | `1.13.0` | dépend de torch |
|
||||
| `timm` | `1.0.24` | dépend de torch |
|
||||
| `open_clip_torch` | `3.2.0` | dépend de torch |
|
||||
| `python-doctr` | `1.0.1` | dépend de torch + torchvision (§5 RAS) |
|
||||
| `safetensors` | `0.7.0` | binding Rust, couplé à transformers/torch |
|
||||
| `tokenizers` | `0.22.2` | binding Rust, couplé à transformers |
|
||||
| `huggingface-hub` | `0.36.0` | couplé à transformers |
|
||||
| `hf-xet` | `1.2.0` | binding Rust, wheel aarch64 à vérifier (cf. §4) |
|
||||
|
||||
## 3. SUPPRIMÉ
|
||||
|
||||
### 3a. Capture / GUI / X11 (serveur DGX headless)
|
||||
|
||||
`PyQt5`, `PyQt5-Qt5`, `PyQt5_sip`, `mss`, `PyAutoGUI`, `pynput`, `evdev`, `python-xlib`, `python3-xlib`, `pystray`, `PyGetWindow`, `PyMsgBox`, `PyScreeze`, `MouseInfo`, `pyperclip`, `pytweening`, `PyRect` — tous hors scope container serveur (§5 ligne 81 : non bloquant si non chargés). Restent sur le poste Linux client de l'agent capture.
|
||||
|
||||
### 3b. Wheels GPU NVIDIA épinglées x86 (15 paquets)
|
||||
|
||||
`nvidia-cublas-cu12`, `nvidia-cuda-cupti-cu12`, `nvidia-cuda-nvrtc-cu12`, `nvidia-cuda-runtime-cu12`, `nvidia-cudnn-cu12`, `nvidia-cufft-cu12`, `nvidia-cufile-cu12`, `nvidia-curand-cu12`, `nvidia-cusolver-cu12`, `nvidia-cusparse-cu12`, `nvidia-cusparselt-cu12`, `nvidia-nccl-cu12`, `nvidia-nvjitlink-cu12`, `nvidia-nvshmem-cu12`, `nvidia-nvtx-cu12` — §5 ligne 78 : désépinglés, retirés du requirements, tirés par torch cu128 ARM.
|
||||
|
||||
### 3c. Fantôme
|
||||
|
||||
`onnx==1.20.1` — audit runtime : non importé par le code server. `onnxruntime` déjà confirmé non utilisé (cf. brief Dom). Suppression nette.
|
||||
|
||||
## 4. À VÉRIFIER (validation Codex/Dom requise)
|
||||
|
||||
| Paquet | Question | Action proposée |
|
||||
|--------|----------|-----------------|
|
||||
| `triton` | §5 risque ligne 79 : PTXAS embarqué peut crasher avec CUDA 13 | Non listé (tiré par torch cu128) ; si KO au runtime : `export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas` |
|
||||
| `hf-xet` | Wheel aarch64 publié au moment du build ? | Désépingler ; si non dispo, retirer (dep optionnelle de `huggingface-hub`) |
|
||||
| `accelerate` | Version compatible torch cu128 ARM ? | Désépinglé (cat. 2) — flag pour test post-install |
|
||||
| `onnx` (suppression) | Confirmé sur scope server/ + core/grounding/ + core/detection/ ? | `grep -r "^import onnx\|^from onnx" --include='*.py' server/ core/` avant build |
|
||||
| `nvidia-ml-py` vs `pynvml` | Doublon (officiel vs fork tiers) | Probable retrait de `pynvml`, à arbitrer |
|
||||
|
||||
---
|
||||
|
||||
## 5. Bloc `requirements-dgx-aarch64.txt` proposé
|
||||
|
||||
Installation en 2 étapes (cu128 ARM **avant** le reste, sinon pip tire les wheels x86 du cache) :
|
||||
|
||||
```bash
|
||||
# Étape 1 : PyTorch ARM cu128
|
||||
pip install --no-cache-dir torch torchvision \
|
||||
--index-url https://download.pytorch.org/whl/cu128
|
||||
|
||||
# Étape 2 : reste
|
||||
pip install --no-cache-dir -r requirements-dgx-aarch64.txt
|
||||
```
|
||||
|
||||
```txt
|
||||
# requirements-dgx-aarch64.txt
|
||||
# Cible : DGX Spark / Ubuntu 24.04 ARM64 / CUDA 13 / GB10 sm_121
|
||||
# Pré-requis : torch+torchvision installés via index cu128 ARM AVANT ce fichier.
|
||||
|
||||
# --- Stack torch-dependent (désépinglé) ---
|
||||
transformers
|
||||
accelerate
|
||||
timm
|
||||
open_clip_torch
|
||||
python-doctr
|
||||
safetensors
|
||||
tokenizers
|
||||
huggingface-hub
|
||||
hf-xet
|
||||
|
||||
# --- Web / API ---
|
||||
fastapi==0.128.0
|
||||
Flask==3.0.0
|
||||
Flask-Caching==2.1.0
|
||||
Flask-Cors==4.0.0
|
||||
Flask-Migrate==4.1.0
|
||||
Flask-SocketIO==5.3.5
|
||||
Flask-SQLAlchemy==3.1.1
|
||||
starlette==0.50.0
|
||||
uvicorn==0.40.0
|
||||
uvloop==0.22.1
|
||||
httptools==0.7.1
|
||||
httpcore==1.0.9
|
||||
httpx==0.28.1
|
||||
h11==0.16.0
|
||||
websockets==16.0
|
||||
wsproto==1.3.2
|
||||
simple-websocket==1.1.0
|
||||
python-engineio==4.8.0
|
||||
python-socketio==5.10.0
|
||||
python-multipart==0.0.21
|
||||
Werkzeug==3.1.5
|
||||
Jinja2==3.1.6
|
||||
itsdangerous==2.2.0
|
||||
blinker==1.9.0
|
||||
click==8.3.1
|
||||
MarkupSafe==3.0.3
|
||||
cachelib==0.9.0
|
||||
watchfiles==1.1.1
|
||||
anyio==4.12.1
|
||||
bidict==0.23.1
|
||||
|
||||
# --- DB / migrations ---
|
||||
SQLAlchemy==2.0.23
|
||||
alembic==1.18.4
|
||||
Mako==1.3.10
|
||||
greenlet==3.3.0
|
||||
redis==5.0.1
|
||||
|
||||
# --- ML / vision CPU ---
|
||||
faiss-cpu==1.13.2
|
||||
opencv-python==4.12.0.88
|
||||
pillow==12.1.0
|
||||
numpy==2.2.6
|
||||
scipy==1.17.0
|
||||
scikit-learn==1.8.0
|
||||
matplotlib==3.10.8
|
||||
contourpy==1.3.3
|
||||
fonttools==4.62.1
|
||||
kiwisolver==1.5.0
|
||||
cycler==0.12.1
|
||||
pyparsing==3.3.2
|
||||
ml_dtypes==0.5.4
|
||||
networkx==3.6.1
|
||||
sympy==1.14.0
|
||||
mpmath==1.3.0
|
||||
joblib==1.5.3
|
||||
threadpoolctl==3.6.0
|
||||
RapidFuzz==3.14.3
|
||||
shapely==2.1.2
|
||||
pyclipper==1.4.0
|
||||
h5py==3.16.0
|
||||
|
||||
# --- PDF / docs ---
|
||||
pypdfium2==5.6.0
|
||||
lxml==6.0.2
|
||||
python-docx==1.2.0
|
||||
openpyxl==3.1.5
|
||||
et_xmlfile==2.0.0
|
||||
defusedxml==0.7.1
|
||||
anyascii==0.3.3
|
||||
langdetect==1.0.9
|
||||
ftfy==6.3.1
|
||||
wcwidth==0.2.14
|
||||
regex==2025.11.3
|
||||
|
||||
# --- Ollama ---
|
||||
ollama==0.6.1
|
||||
|
||||
# --- Validation / config ---
|
||||
pydantic==2.12.5
|
||||
pydantic_core==2.41.5
|
||||
annotated-doc==0.0.4
|
||||
annotated-types==0.7.0
|
||||
typing-inspection==0.4.2
|
||||
typing_extensions==4.15.0
|
||||
validators==0.35.0
|
||||
marshmallow==3.20.1
|
||||
jsonschema==4.20.0
|
||||
jsonschema-specifications==2025.9.1
|
||||
referencing==0.37.0
|
||||
rpds-py==0.30.0
|
||||
attrs==25.4.0
|
||||
PyYAML==6.0.1
|
||||
python-dotenv==1.0.0
|
||||
|
||||
# --- HTTP / réseau ---
|
||||
aiohttp==3.13.3
|
||||
aiohappyeyeballs==2.6.1
|
||||
aiosignal==1.4.0
|
||||
frozenlist==1.8.0
|
||||
multidict==6.7.0
|
||||
propcache==0.4.1
|
||||
yarl==1.22.0
|
||||
requests==2.32.5
|
||||
urllib3==2.6.3
|
||||
charset-normalizer==3.4.4
|
||||
idna==3.11
|
||||
certifi==2026.1.4
|
||||
|
||||
# --- Crypto / utils ---
|
||||
cryptography==46.0.3
|
||||
cffi==2.0.0
|
||||
pycparser==2.23
|
||||
filelock==3.20.3
|
||||
fsspec==2026.1.0
|
||||
packaging==25.0
|
||||
platformdirs==4.5.1
|
||||
pathspec==1.0.3
|
||||
six==1.17.0
|
||||
python-dateutil==2.8.2
|
||||
sortedcontainers==2.4.0
|
||||
tqdm==4.67.1
|
||||
Pygments==2.19.2
|
||||
|
||||
# --- Monitoring ---
|
||||
prometheus_client==0.23.1
|
||||
psutil==7.2.1
|
||||
pynvml==13.0.1
|
||||
nvidia-ml-py==13.590.48
|
||||
|
||||
# --- Protobuf ---
|
||||
protobuf==7.34.0
|
||||
|
||||
# --- Dev / tests (à isoler dans requirements-dev.txt à terme) ---
|
||||
pytest==9.0.2
|
||||
pytest-asyncio==1.3.0
|
||||
pytest-cov==4.1.0
|
||||
pytest-flask==1.3.0
|
||||
pytest-mock==3.12.0
|
||||
hypothesis==6.92.1
|
||||
coverage==7.13.1
|
||||
iniconfig==2.3.0
|
||||
pluggy==1.6.0
|
||||
black==23.12.1
|
||||
flake8==6.1.0
|
||||
mypy==1.7.1
|
||||
mypy_extensions==1.1.0
|
||||
mccabe==0.7.0
|
||||
pycodestyle==2.11.1
|
||||
pyflakes==3.1.0
|
||||
```
|
||||
|
||||
**Bilan** : 180 → ~125 lignes. Retraits = 17 capture/GUI + 15 nvidia-cu12 + 1 onnx + 11 remontés cat. 2 désépinglée (torch/torchvision/transformers/accelerate/timm/open_clip_torch/python-doctr/safetensors/tokenizers/huggingface-hub/hf-xet) = 44 lignes en moins, 9 désépinglées remontées en tête.
|
||||
|
||||
---
|
||||
|
||||
## 6. Vérifications post-install sur DGX
|
||||
|
||||
```bash
|
||||
# 1. PyTorch + GPU sm_121
|
||||
python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_device_capability(0))"
|
||||
# attendu : (True, (12, 1))
|
||||
|
||||
python -c "import torch; print(torch.__version__, torch.version.cuda)"
|
||||
# attendu : 2.9.x+ / 12.8
|
||||
|
||||
# 2. FAISS
|
||||
python -c "import faiss; print(faiss.__version__)"
|
||||
# attendu : 1.13.x
|
||||
|
||||
# 3. Transformers stack
|
||||
python -c "import transformers; print(transformers.__version__)"
|
||||
python -c "import accelerate, timm, open_clip, doctr; print(accelerate.__version__, timm.__version__, open_clip.__version__, doctr.__version__)"
|
||||
|
||||
# 4. Bindings Rust aarch64
|
||||
python -c "import tokenizers, safetensors; print(tokenizers.__version__, safetensors.__version__)"
|
||||
python -c "import hf_xet; print(hf_xet.__version__)" # point de vérif cat. 4
|
||||
|
||||
# 5. Confirmation onnx absent et runtime OK
|
||||
python -c "import onnx" 2>&1 | grep -q ModuleNotFoundError && echo "OK : onnx absent"
|
||||
|
||||
# 6. Points d'entrée applicatifs
|
||||
python -c "from web_dashboard.app import app; print('dashboard OK')"
|
||||
python -c "from visual_workflow_builder.backend.app import app; print('VWB backend OK')"
|
||||
# adapter le module server selon le wiring réel
|
||||
|
||||
# 7. Sanity GPU
|
||||
python -c "import torch; x=torch.randn(8,8,device='cuda'); print('matmul GPU OK', (x@x).device)"
|
||||
|
||||
# 8. Ollama client → daemon
|
||||
python -c "import ollama; print(ollama.list())"
|
||||
```
|
||||
|
||||
**Garde-fous §5** :
|
||||
- Si kernels Triton crashent : `export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas`.
|
||||
- Si pip tire encore des `nvidia-*-cu12` x86 : confirmer que l'étape 1 (torch cu128 ARM) a été exécutée **avant**, et que `--no-cache-dir` est bien passé.
|
||||
|
||||
---
|
||||
|
||||
## 7. Points ouverts pour Codex/Dom
|
||||
|
||||
1. Confirmer suppression `onnx` : `grep -r "^import onnx\|^from onnx" --include='*.py' server/ core/` avant build image Docker.
|
||||
2. Splitter dès le POC `requirements-dev.txt` (pytest/black/mypy/flake8/coverage/hypothesis) pour ne pas polluer l'image server ? À trancher.
|
||||
3. `hf-xet` : si wheel aarch64 indispo au build, retirer (dep optionnelle `huggingface-hub`).
|
||||
4. `nvidia-ml-py` vs `pynvml` : doublon. Garder uniquement `nvidia-ml-py` (officiel) ?
|
||||
Reference in New Issue
Block a user