19 Mar, 2026
2 commits
-
The instability is very likely real overload, but `lsof -i :6005 | wc -l = 75` alone does not prove it. What does matter is the live shape of the service: it is a single `uvicorn` worker on port `6005`, and the code had one shared process handling both text and image requests, with image work serialized behind a single lock. Under bursty image traffic, requests could pile up and sit blocked with almost no useful tracing, which matches the “only blocking observed” symptom. now adds persistent log files, request IDs, per-request request/response/failure logs, text microbatch dispatch logs, health stats with active/rejected counts, and explicit overload admission control. New knobs are `TEXT_MAX_INFLIGHT`, `IMAGE_MAX_INFLIGHT`, and `EMBEDDING_OVERLOAD_STATUS_CODE`. Startup output now shows those limits and log paths in [scripts/start_embedding_service.sh](/data/saas-search/scripts/start_embedding_service.sh#L80). I also added focused tests in [tests/test_embedding_service_limits.py](/data/saas-search/tests/test_embedding_service_limits.py#L1). What this means operationally: - Text and image are still in one process, so this is not the final architecture. - But image spikes will now be rejected quickly once the image lane is full instead of sitting around and consuming the worker pool. - Logs will now show each request, each rejection, each microbatch dispatch, backend time, response time, and request ID. Verification: - Passed: `.venv/bin/python -m pytest -q tests/test_embedding_service_limits.py` - I also ran a wider test command, but 3 failures came from pre-existing drift in [tests/test_embedding_pipeline.py](/data/saas-search/tests/test_embedding_pipeline.py#L95), where the tests still monkeypatch `embeddings.text_encoder.redis.Redis` even though [embeddings/text_encoder.py](/data/saas-search/embeddings/text_encoder.py#L1) no longer imports `redis` that way. 已把 CLIP_AS_SERVICE 的默认模型切到 ViT-L-14，并把这套配置收口成可变更的统一入口了。现在默认值在 embeddings/config.py (line 29) 的 CLIP_AS_SERVICE_MODEL_NAME，当前为 CN-CLIP/ViT-L-14；scripts/start_cnclip_service.sh (line 37) 会自动读取这个配置，不再把默认模型写死在脚本里，同时支持 CNCLIP_MODEL_NAME 和 --model-name 临时覆盖。scripts/start_embedding_service.sh (line 29) 和 embeddings/server.py (line 425) 也补了模型信息输出，方便排查实际连接的配置。 文档也一起更新了，重点在 docs/CNCLIP_SERVICE说明文档.md (line 62) 和 embeddings/README.md (line 58)：现在说明的是“以配置为准、可覆盖”的机制，而不是写死某个模型名；相关总结文档和内部说明也同步改成了配置驱动表述。
-
中采用了最优T4配置：ct2_inter_threads=2、ct2_max_queued_batches=16、ct2_batch_type=examples。该设置使NLLB获得了显著更优的在线式性能，同时大致保持了大批次吞吐量不变。我没有将相同配置应用于两个Marian模型，因为聚焦式报告显示了复杂的权衡：opus-mt-zh-en 在保守默认配置下更为均衡，而 opus-mt-en-zh 虽然获得了吞吐量提升，但在 c=8 时尾延迟波动较大。 我还将部署/配置经验记录在 /data/saas-search/translation/README.md 中，并在 /data/saas-search/docs/TODO.txt 中标记了优化结果。关键实践要点现已记录如下：使用CT2 + float16，保持单worker，将NLLB的 inter_threads 设为2、max_queued_batches 设为16，在T4上避免使用 inter_threads=4（因为这会损害高批次吞吐量），除非区分在线/离线配置，否则保持Marian模型的默认配置保守。
18 Mar, 2026
1 commit
-
batch×并发矩阵”彻底分开展示。 改动在这几处： scripts/benchmark_translation_local_models.py：新增 --suite extended，支持 batch_size=1,4,8,16,32,64、concurrency=1,2,4,8,16,64、以及 batch_size * concurrency <= 128 的组合矩阵；并且单场景模式现在只加载目标模型，load_seconds 更干净，也支持 --disable-cache。 translation/README.md：把性能章节拆成了 batch_sweep、concurrency_sweep、batch x concurrency matrix 三块，补了这次复测的参数、复现命令和摘要表。 perf_reports/20260318/translation_local_models/README.md：新增本轮补测摘要。 完整结果在 translation_local_models_extended_221846.md 和 translation_local_models_extended_221846.json。 这次补测的核心结论很明确： 在线单条请求应该看 concurrency_sweep，也就是固定 batch_size=1 的表。 离线批量吞吐应该看 batch_sweep，4 个方向的最高 raw throughput 都出现在 batch_size=64，但更均衡的默认值仍更像 batch_size=16。 当前本地 seq2seq backend 有单模型锁，提升客户端并发几乎不涨吞吐，主要是把排队时间变成更高的 p95；所以并发更像“延迟预算”问题，不是“扩容吞吐”手段。 本轮在线单条里最快的是 opus-mt-zh-en；最慢、且并发放大最明显的是 nllb-200-distilled-600m en->zh。
17 Mar, 2026
3 commits
12 Mar, 2026
1 commit
11 Mar, 2026
1 commit